From: ICLR.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Spotlight)
comment: This paper introduces a new technique for discovering closed-form functional forms (ordinary differential equations) that explain noisy observed trajectories x(t) where the "label" x'(t) = f(x(t), t) is not observed, but without trying to approximate it. The method first tries to approximate a smoother trajector x^hat(t), then relies on a variational formulation using a loss function over functionals {C_j}_j, defined in terms of an orthonormal basis {g_1, …, g_S} of sampling functions such that the sum of squares of all the C_j approximates the theoretical distance between f(x) and the solution f*(x). These sampling functions are typically chosen to be a basis of sine functions. The method is evaluated on several canonical ODEs (growth model, glycolitic oscillator, Lorenz chaotic attractor) and compared to gaussian processes-based differentiation, to spline-based differentiation, regularised differentiation, and applied to model the temporal effect of chemotherapy on tumor volume. 

Reviewers found that the paper was well-motivated and easy to follow (EBvJ), well evaluated (EBvJ), offering new perspectives to symbolic regression (79Ft). Reviewer vaG3 had their concerns addressed. Reviewer ZddY had concerns about the running time (a misunderstanding that was clarified) and the lack of comparison to a simple baseline consisting in double optimisation over f and x^hat(0) using Neural ODEs (the authors have added a Neural ODE baseline but were in disagreement with ZddY and 79Ft about their limitations).

Reviewers engaged in a discussion with the authors, and the scores are 6, 6, 8, 8. I believe that the paper definitely meets the conference acceptance bar and would advocate for its inclusion as a spotlight in the conference.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_ZddY
title: Thank you for the clarification
comment: Dear authors,

Thank you for the clarification about the learning time. I updated my score back to 6.

I'm still not convinced by the fact that DO would be that prohibitive. Don't you need to also learn the parameters of the ODE in your approach ? That is, for every type of functional form that the genetic algorithm proposes, you need to learn what is the best set of parameters the parametrize that functional form. Please let me know if this is incorrect but this seems to be what you are doing in Algorithm 1 in the appendix.

If this is indeed the case, then I see no big issue in having to optimize for the initial parameters as well (on top of the parameters of the functional form).

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_vaG3
title: Reply to Paper1109 Authors
comment: Thank you very much for your response. 

About (3) Multivariate data, yes, I did not catch this. Thank you for pointing this out. Does the method scale to high dimensions? 
About (1) High noise level and sparse observation. This is very interesting, and I believe it is a great advantage of the method, well illustrated by Appendix G.6. This convinces me to upgrade my score from 6 to 8. I would recommend that the authors emphasize this point in the main paper. 

About (2) (and (4)), many papers on approximating closed-form ODEs show results on dynamical system with unknown latent ODE but with specific, interpretable properties such as cancer relapse. I do think it would be possible to show results and dynamics on more than one real dataset. 

I apologize for my late response.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Reply to Reviewer 79Ft
comment: Thank you for the additional question. Yes, your understanding of the Neural ODE experiment is totally correct. To train NODE, we estimate the network weights as well as the initial conditions by backpropagation. 

However, NODE training has a key difference from the Double Optimization (DO) method raised by Reviwer ZddY. In DO, the outer optimization searches for the functional form of $f$ explicitly -- it is a combinatorial optimization problem (solved by genetic programming). In contrast, the network weights in NODE are continuous and they can be trained *jointly* with the initial conditions. 

Since NODE learns all the unknowns (network weights + initial conditions) *simultaneously* via gradient updates, it is indeed much faster than DO. However, unlike DO, NODE does not directly produce closed-form equations (and it needs to be combined with post-hoc distillation, which we have evaluated in the additional experiments).

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_79Ft
title: Initial conditions and compute time of Double Optimization
comment: Regarding the discussion on the review from Reviewer ZddY, I have some questions. Do you need to estimate the initial condition because of the high noise level you assume? If I interpret your experiments correctly you assume fully observed dynamical systems without hidden variables, so your first observation is a noisy sample of your initial condition.
In this case training a NeuralODE is simple, as you can learn the initial condition with backpopagation. I do not expect this would result in  very large training time. In this case using NODE in a Double Optimization loop is maybe preferred to the other methods you mention and compare to.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Clarifications on computation time and applicability
comment: Thank you for your follow-up comments. 

First of all, we would like to clarify that D-CODE takes days to complete *all the simulations in the study*. 
Note that the study involves 9 different ODEs, and for each ODE we perform 100 simulation runs to estimate the success probability (900 runs in total). 

The average time for each run (i.e time to extract an ODE from *a given dataset*) ranges from 8 minutes (Gompertz ODE) to 30 minutes (Glycolytic), as shown in **Figure 11**. We believe D-CODE's computation time is acceptable for discovering ODEs, which is not a time-critical application (note that the task is traditionally carried out by human experts manually). 

Secondly, we would like to point out that Double Optimization requires estimating the initial conditions for *all* trajectories. Given a dataset with $N$ trajectories and $J$ variables, there will be $N\times J$ additional parameters (i.e. unknown initial conditions) to  optimize in the inner step. For a sizable dataset, DO could easily introduce hundreds of new parameters.  As we have shown, this significantly slows down the procedure (30 - 50 times slower than D-CODE).

Finally, we stress that discovering complex equations is a challenging problem *in general* --- for D-CODE as well as for the alternative methods (possibly even for human experts). Note that the success probability for equation 8 also dropped significantly for other benchmark methods (which are outperformed by D-CODE). 

We hope that our response clarifies the concerns about computation time and applicability, and we would much appreciate it if you could update your score based on this response.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_ZddY
title: Concern about applicability of the method
comment: Thank you for your answers. However, the answers now make me doubt about the relevance and the applicability of the method. I therefore updated my score accordingly.

As for the double optimization, I think that depending on how you frame your optimization problem, learning the initial value should not be so problematic. If you iterate on the functional form and for each of those try to learn the best set of parameters, you can just include the initial value as another parameter : 

$min_{f} min_{\theta,\hat{x}_0} \mid \mid y(t)-\hat{x}(t) \mid \mid$

Where $f$ is here the functional form.

What concerns me now is the fact that your method takes days to complete, on simple, dimensional datasets, which in my opinion limits the applicability of the approach. 

Furthermore, the fact that equation 8 might result in much lower success probability because 2 parameters are involved in the equation is quite worrisome for more realistic functional forms where more elaborate functional forms are very likely.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Dear Reviewer 79Ft
comment: Thank you very much for your response. We would much appreciate if you would update your score on the basis of this response. Thank you.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Dear Reviewer vaG3 
comment: Once again, thank you for your invaluable feedback. We were wondering whether our response and revision have addressed your concerns. If you have any additional comments, please let us know, we would be eager to address them.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Dear Reviewer ZddY
comment: Once again, thank you for your invaluable feedback. We were wondering whether our response and revision have addressed your concerns. If you have any additional comments, please let us know, we would be eager to address them.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_79Ft
title: Valuable experiments
comment: Thank you for your detailed answers to my questions. In my opinion the new experiments with NODEs helps to pinpoint the limitations of the NODE based gradient estimator better, and is a valuable part of the paper.
Indeed using the Lyapunov time is just a theoretical possibility and it is not practical.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Reply to Reviewer 79Ft
comment: Thank you for your thoughtful comments and suggestions. We’ve provided a response below and included additional experimental results and discussions in the revised paper (**Appendix H.2**). Please let us know if anything needs further clarification. 

---

### (1) Neural ODE with post-hoc distillation

We agree that the method proposed (training a Neural ODE and then distilling a closed-form expression) is applicable to our setting.
We have evaluated its performance in the updated **Appendix H.2** and found it to perform worse compared to D-CODE. 

We would like to highlight that this approach is an instance of *two-step regression*, where the derivative is estimated by NODE instead of numerical differentiation (Figure 1 B). 
In fact, we found that this approach behaves very similarly with other two-step regression methods, i.e. sharp decline in success probability with the increased noise level. 

Furthermore, although NODE is known to be a universal function approximator, its estimation efficiency (e.g. $\sqrt{n}$ consistency), finite sample performance, and robustness to noise are still largely under investigation. 
Therefore, we would remind the reviewer that NODE is not guaranteed to outperform numerical differentiation in estimating the derivatives (and producing the dataset for symbolic regression). 

### (2) Neural ODE training

We agree with the reviewer that NODE may be able to capture chaotic dynamics when trained with a short prediction horizon. We have modified the wording in Section 3.2 to avoid confusion. However, using a short prediction horizon may lead to a myopic model that fails to capture long-term patterns [1,2].

In the experiments presented in **Appendix H.2**, we found that NODE trained with long or short prediction horizons achieve very different results (see Appendix H.2 for a detailed analysis).
Therefore, the prediction horizon is a *hyperparameter* that plays an important role in the training. 
The strong dependency on this hyperparameter may introduce additional difficulty for using NODE in practical applications --- note that the Lyapunov time of the true dynamics is unknown and cannot inform the choice of the prediction horizon.
In contrast, D-CODE does not suffer from this issue as it is not trained to minimize prediction loss. 

---

### References 


[1] Schmidt, Dominik, et al. "Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies." International Conference on Learning Representations. 2020.

[2] Holger Kantz and Thomas Schreiber.Nonlinear time series analysis, volume 7. Cambridge university press, 2004.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Reply to Reviewer vaG3
comment: Thank you for your thoughtful comments and suggestions. We’ve provided a point-by-point response below. Please let us know if anything needs further clarification. 

---

### (1) High noise level and sparse observation

Thanks for raising the interesting question about whether D-CODE can still discover the true ODE when the smoother is not working perfectly. 
We’d like to clarify that we have included results under a wide range of signal-to-noise ratios $1/\sigma_R$ and sampling sparsities $\Delta t$ (see Table 3 for the parameter range, and Figure 2, 4, 7 and Table 2 for the results). 

We’d like to point out that having Gaussian noise does not guarantee that the true trajectories can be recovered accurately in all circumstances. 
In fact, in many experiments, we observe that the smoothed trajectory deviates significantly from the true one; yet D-CODE can still recover the true equation with high probability. 
To illustrate this point, we have included visualizations of the smoothed and the true trajectories in **Appendix G.6** annotated with the success probabilities of D-CODE.


### (2) Real data experiment

The key challenge to performing additional real data experiments is that we do not know the ground truth equation for real datasets. Hence, it is difficult to quantify whether the discovered ODE is correct or not. In the paper, we included an illustrative use case on a tumor growth dataset and showed qualitatively that the learned ODE was able to capture a key aspect of the dynamics (cancer relapse). 

### (3) Multivariate data

We’d like to clarify that D-CODE is not restricted to one-dimensional data. In fact, the glycolytic oscillator and the Lorenz system in the experiments involve more than one dimension (Section 5). 
For a system with $J$ variables, D-CODE can iteratively discover $J$ ODEs, one for each variable (see the pseudocode in Appendix B). 

### (4) Latent ODE

We consider discovering *latent* ODEs as an open challenge for future works. The main difficulty is the identifiability issue, which is commonly encountered in dynamical system modeling [1]. 

### (5) Minor points
Thank you for pointing that out. We have tried to improve the clarity of some of the figures by better utilizing the whitespaces around them. 

---

### References 

[1] Diop, Sette, and Michel Fliess. "Nonlinear observability, identifiability, and persistent trajectories." [1991] Proceedings of the 30th IEEE Conference on Decision and Control. IEEE, 1991.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Reply to Reviewer ZddY
comment: Thank you for your thoughtful comments and suggestions. We’ve provided a point-by-point response below. Please let us know if anything needs further clarification. 

---

### (1) Double optimization

Thank you for bringing up the double optimization method (DO). This method is indeed relevant and (in theory) applicable to the problem. We have included additional discussions and results in the updated **Appendix H.1** and summarized below. 

DO solves the following problem (some subscripts are dropped because OpenReview fails to render complex equations; please refer to Equation 39 in the paper for a more precise definition): 

$
\min_f \Big[ \min_{\hat{x}(0)}  || y(t) - \hat{x}(t) ||^2_2 \Big]
$

subject to the constraint

$
\hat{x}(t) = \hat{x}(0) + \int_0^t f(\hat{x}(s)) ds
$

This is a nested optimization problem where the outer part searches for the unknown function $f$ and the inner part searches for the unknown initial conditions $x(0)$ given the current guess of $f$. Note that the inner and outer parts are *coupled* because the optimal $\hat{x}(0)$ depends on $f$, e.g. the best-fitting exponential curve would have a different $x(0)$ from the best linear fit.

The DO method was first proposed in the literature of ODE parametric estimation, where the functional form of $f$ is known but it contains unknown parameters $\theta$ [1]. 
As known in the literature, the main shortcoming of DO is its computational complexity [2]. The inner optimization over $x(0)$ is typically non-convex and may take many steps to converge. Furthermore, each inner optimization step requires solving the IVP numerically. 

The computational challenge becomes even more pronounced in our setting, where the functional form of $f$ is also unknown. Genetic programming needs to evaluate many candidate functions $f$ but DO requires solving the inner optimization for each evaluation. In contrast, D-CODE (Equation 5) does not involve any nested optimization. 

We have implemented DO and tested its computation time to evaluate a candidate ODE $f$. The results are shown in **Appendix H.1**. We found that DO is, on average, 30-50 times slower than D-CODE (for reference, the full simulation on D-CODE takes a few days to finish).
DO’s computational cost is too high to be considered as a feasible method for discovering closed-form ODEs.
 

### (2) Difference in performance

Although Equation 8 and 9 have similar functional forms, Equation 8 involves two numerical constants $\theta_1, \theta_2$ while Equation 9 involves only one. This may explain the difference in performance because Equation 8 requires estimating an additional continuous parameter. 

### (3) Computation time

We discussed and visualized the computation time in **Appendix G.4**. We expect D-CODE to have a longer computation time than two-step regression due to the numerical integration step in evaluating the functional $C$ (Eq. 3). We stress that this integration is *always* one-dimensional (i.e. over time) no matter how many variables are in the system. 
The 1D numerical integration can be efficiently implemented as a vectorized operation, which is highly optimized for modern CPUs. We found that, in the worst case, a simulation run of D-CODE takes twice as long as two-step regression.


### (4) Interpolation for multivariate data

For multivariate data, we smoothed each dimension separately in the experiments. Using multi-task GP to smooth all dimensions jointly may even strengthen the performance of D-CODE. We highlight that an advantage of D-CODE is that it allows the user to choose any smoothing algorithm suitable for the application -- more advanced algorithms such as multi-task GP can be used in the framework. 

### (5) Typo
As Equation 3 is a part of Definition 1, all symbols used are defined at the beginning of the definition. There $f: \mathbb{R}^J \rightarrow \mathbb{R}$ is in fact a scalar field. To differentiate between vector and scalar quantities, we denote vectors using boldface, e.g. $\mathbb{v}$. As the functional $C_j$ can be defined for any single function $f$, we decided not to include the subscript $j$ as it would play no role in the definition. 



---

### References 

[1] Ramsay, Jim O., et al. "Parameter estimation for differential equations: a generalized smoothing approach." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69.5 (2007): 741-796.

[2] Gugushvili, Shota, and Chris AJ Klaassen. "$\sqrt n $-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing." Bernoulli 18.3 (2012): 1061-1098.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Reply to Reviewer EBvJ
comment: Thank you for your thoughtful comments and suggestions. We’ve provided a point-by-point response below. Please let us know if anything needs further clarification. 

---

### (1) When to consider high order or non-autonomous systems?
A key property of a first-order autonomous system is that the trajectories cannot intersect (This is a well-known result, e.g. Proposition C.6 in [1]). Formally speaking, let $\mathbf{x}_1$ and $\mathbf{x}_2$ be two solutions of the ODE (Equation 1). If their initial conditions are different, i.e. $\mathbf{x}_1(0) \ne \mathbf{x}_2(0)$, then it holds that $\forall t>0,$  $\mathbf{x}_1(t) \ne \mathbf{x}_2(t)$. 

Hence, If intersections occur, we should consider modeling the data with a high order or a non-autonomous system. This is the rationale to include time $t$ in the tumor growth models (Section 6). 

### (2) Representation power of closed-form equations. 

It is possible that the true ODE $f$ does not have a closed form  (i.e. cannot be expressed as a finite sequence of basic operations and input variables). For example, it may contain the Gauss error function (erf)  [2]. 
When this is the case, the algorithm finds a compact closed-form equation to approximate the true equation. We emphasize that making an approximation to achieve compact mathematical form is a common practice in sciences (e.g. Classical Mechanics is an approximation as it ignores relativity and quantum effects). 

We further discuss the failure modes and options for iterative modeling in Section 7 and Appendix C.

### (3) Neural ODE.

We have included additional experiments and discussions on Neural ODEs in the updated **Appendix H.2**. Note that the original Neural ODE cannot directly produce a *closed-form* ODE, which is the purpose of this work. 

Our results also suggest that the performance of Neural ODE *strongly* depends on the prediction horizon used for training (a hyperparameter). However, in practice, the Lyapunov exponent of the true dynamics is unknown and cannot inform the choice of the prediction horizon. The strong dependency on the hyper-parameter may introduce practical difficulty in Neural ODE modeling. 


---

### References 


[1] Laurent Younes. Shapes and Diffeomorphisms, volume 171. Springer Science & Business Media, 2010.

[2] Andrews, Larry C. (1998). Special functions of mathematics for engineers. SPIE Press. p. 110. ISBN 9780819426161.

From: ICLR.cc/2022/Conference/Paper1109/Authors
title: Summary of revision
comment: We thank all reviewers for the thoughtful comments and suggestions. We have revised the manuscript based on the reviews. All revisions are in blue text; they include

1. **Section 3.2 and 5.1**: revision on the discussion about Neural ODE.

2. **Figure 4**: showing a Neural ODE trained with a small prediction horizon. 

3. **Appendix G.6**: discussion on D-CODE’s performance when the smoother does not accurately recover the true trajectory. 

4. **Appendix H.1**: discussion and additional results on the double optimization method.

5. **Appendix H.2**: discussion and additional results on Neural ODE with post-hoc distillation. 

6. Minor change on the size of graphics.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_EBvJ
summary_of_the_paper: This paper suggests a variational formulation for discovering ODE systems in a closed-form. The main advantage of such a formulation is that it can find the system based on the observed trajectories (and some analytical testing functions) rather than estimating (possibly noisy) derivatives of them. The variational objective is equal to the distance between estimated and true velocity vector fields on the manifold when the estimated trajectory converges to the true one in L2 sense and there are infinitely many testing functions (while a finite number of functions are okay empirically). The proposed method outperforms its counterparts for various ODE systems and tumor volume dataset (whose true dynamics is unknown).
main_review: The paper is well-motivated and easy to follow. The proposed variational approach is sound for by-passing time-derivatives of noisy observations. It is also a model-agnostic framework, thus might be widely applicable regardless own’s specific setting.

Empirical validation on the proposed framework (D-CODE) shows that D-CODE is the right way to recover ODEs under some undesirable but unavoidable measurement errors.

The tumor growth experiment is very interesting because its closed-form ODE is indeed unknown. D-CODE can capture an interesting behavior (tumor volume increase due to drug resistance?) while the baseline cannot.

I have not found any major drawbacks to this paper. For these reasons, I think this paper is worthy of publication for ICLR. The followings are some (relatively minor) questions:

* Q1. Treating second-order non-autonomous systems as a set of first-order autonomous ODEs requires non-trivial effort since the number of variables increases to 2J + 1. Is there any suggestion about when one should consider such a situation (without prior knowledge of the target system)? It seems that the authors have found non-autonomous ODE for the tumor growth task, thus it will be nice if they can explain their modeling procedure briefly.

* Q2. Is the search space for mathematical operations used in this paper enough for arbitrary systems? If not, and at the same time if the variational objective does not converge to 0 well for an unknown target system, then what would be the best option for the next behavior, e.g., adding some additional operations, or considering higher-order non-autonomous systems such as Q1? It may depend on the target problem, but it will be nice if the authors can discuss about it.

* Q3. The authors state that neural ODEs should estimate the unknown initial condition (thus they are not very effective for chaotic systems), but it seems that one can rather directly use y(0) = x(0) + eps(0) as an initial condition for neural ODEs. Are neural ODEs still bad for such a case, though it might depend on the noise level and Lyapunov exponent of the target dynamics? The current evaluation of neural ODEs in Figure 4 seems to be extremely poor (and I felt the evaluation is slightly unfair), thus I wonder it is truly a limitation of neural ODEs or due to a bad estimation of the initial state.

summary_of_the_review: As I mentioned above, I think the paper is interesting and thus recommend the acceptance of this paper.

***
Post-rebuttal: I appreciate the authors’ clarifications on my questions. I'd like to keep my score.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_ZddY
summary_of_the_paper: This paper proposes a new methodology to infer symbolic ODE representation from observed time series. In contrast to previous methods, they bypass the inference of the time derivative and rather proceed by first estimating a continuous approximation of the trajectory and then optmizing a novel objective function. The first step can be performed with the interpolation method of choice (GP or splines). The second step can use any optimization scheme that does not require derivative (such as genetic algorithm). The authors then evaluate their approach. on a series of dynamical system and show improved performance compared to the baselines under consideration.
main_review: The paper is well written and addressing an important problem for scientific discovery and interpretability of ODE methods. 


- My main comment is that I think another baseline to investigate the importance of the proposed objective function would be the following : you optimize over an initial value and the function f and you minimize the reconstruction error. Is this a baseline you could compare against ? As it does not involve computing the derivative either, it could also be competitive and really assess the added value of the proposed objective function. I also believe this setup would fit Theorem 1. If this is not a good idea, can authors explain why ? As I don't see a clear advantage of the proposed method over the one I just described, it currently undermines my evaluation of the significance/novelty of this work. However, I'd be very happy to change my mind on this topic.

- In Table 2 it seems that the performance differ massively between both dimensions (8) and (9). Yet, at a glance, their functional form look really similar. It would be nice to provide some extra information or insights in that regard in the text.

- Because all these symbolic regression approaches make use of non-differentiable optimizers, the computational cost can be quite high. I would appreciate some discussion of the computational complexity of the different methods and to have an order of magnitude of the computation time.

- It's also not exactly clear to me how the interpolation steps work in the multi-dimensional setup. In practice do you use single dimension GPs for each dimension or do you use multi-task GP to model the correlations ? 

- In Equation 3, the first integral should have $f_j$, as $f$ is not a scalar function.
summary_of_the_review: I think this is a nice paper but to fully motivate the approach, it should compare against another obvious baseline. I would like the authors to motivate why the setup I described above is not considered in the experiments. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_vaG3
summary_of_the_paper: Authors propose D-CODE, a framework to approximate closed-form ODEs using symbolic regression. The method first estimate the trajectory by smoothing the observations (using any convenient smoother) before inferring a closed form (finite sequence of operations) transition function. This allows for better interpretability of the system. 

Authors test D-Code on 5 simulated datasets, where they show high performance of the method, and on one real dataset where they use D-Code to model the temporal effect of chemotherapy on the tumor volume, and compare it to Total variation regularized differentiation (SR-T), the closest method in the literature. 
main_review: Using symbolic regression to approximate closed-form ODEs is novel and very interesting. The results presented in the paper are compelling but I would want to see more comparisons to other methods, as well as results on more datasets. 

The method section is very well described, with useful and precise details found in the appendix.

The weakness of this paper lies in the experiments and results section. 

Authors have selected five dynamical systems governed by a wide range of nine closed-form ODEs. However, all these system have independent Gaussian noise (with different noise level). I assume that if the noise is properly handled by the smoother, this will not impact the model, but I would like to see experiments on datasets where the smoother does not necessary perform very well (low signal to noise ratio). In that case, will D-CODE be robust enough to recover correct parameters of the ODE? 
If this is not the case, I don't really see the advantage of D-CODE in addition to "converting" an already inferred trajectory into an ODE for the sake of interpretability. 

Similarly, how will it handle sparse observations? One advantage of methods inferring ODE/SDE parameters for dynamical systems are the ability to approximate trajectories in regions with few observations. Here, won't the method suffer from the smoother if not able to capture the trajectory correctly? 

To understand better the advantages of D-CODE over other methods, I think that showing results on more than one real dataset (I understand the dataset presented here is very challenging), with comparisons to more than one method (for example, Neural ODE, discussed here in the simulated data experiments). 

Also, is D-CODE restricted to one dimensional observations? (would it be possible to perform symbolic regression for higher dimensional latent variables?) If it is possible, then an extension of this framework to higher-dimensional observations and latent processes would be very compelling and allow for comparisons with many models from the Dynamical systems literature that infer latent ODEs.

Minor comment: the figures (especially Figure 2) could be made bigger and clearer (by removing white space between the panels).

Post rebuttal : I increased my score from 6 to 8 given that D-CODE allows to approximate the latent ODE even when the smoother does not perform well. 
summary_of_the_review: I would lean towards acceptance of the paper. Using symbolic regression to approximate closed-form ODEs is novel and very interesting, and the results presented in the paper are compelling. However, I would want to see more comparisons to other methods, as well as results on more datasets to gain a better understanding of when this method will be useful and outperform existing methods. I am hoping that the authors will be able to to answer my concerns by presenting more results. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

From: ICLR.cc/2022/Conference/Paper1109/Reviewer_79Ft
summary_of_the_paper: The paper addresses the problem of learning closed form ODEs from observational data, when the observation can be noisy and not frequent enough that instantaneous derivatives can be estimated with low enough error. The work uses a variational criterion on the solution of the ODE, circumventing the need to evaluate instantaneous derivatives. It uses existing symbolic regression techniques but instead of the regression loss, it minimizes a loss term quantifying the violation of the variational constraints. It is a work combining existing results in a novel way, but offers new perspective and there are synergies between the different parts applied.
main_review: The method is clearly described, and the potential benefits are clearly understandable (see summary). However, I find the baseline comparison somewhat incomplete.

My main questions are about the NeuralODE comparison. I do not agree that learning a chaotic dynamics directly with a NODE is not possible. The author argue that we need to estimate the initial condition what makes it impossible due to the chaotic instability. Only thing we actually need to train a NODE is to minimize the error in a shorter time horizon. Specifically, if we have samples more often than the Lyapunov time we should be able to learn the mapping. We integrate the dynamics forward inside a smaller than Lyapunov time window, and use this error to learn. 
Please clarify this point, and train the NODE with a loss appropriate, and show that result on the rightmost plot of Figure 4. A simulation should stay on the attractor, of course the concrete trajectory is impossible to reproduce.

The valid argument still remains that NODE is not interpretable while closed form ODE is. I have no doubt on this point. However if NODE training is actually possible, it can be used similarly as GP and others in the two-step methods, and the gradient networks can be distilled to a closed form equation.

summary_of_the_review: The work is derivative in methodology, but provide enough insight to be interesting. I have some questions on the Neural ODE comparison what would need some clarification. If this concern is addressed it would make it easier to support acceptance.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.