Paper: Provably Accurate ODE Forecasting Through Explicit Trajectory Optimization (ICLR-2024)

Reviewer comment (Theory):
I find the paper presents an overly simple idea with an unrealistic assumption. When the space of parameters $\Theta$, the space for the inputs $\mathcal{U}$ are both finite-dimensional, finding the unknown dynamics is, of course, a finite-dimensional problem. However, most of the time,  the space of parameters $\Theta$/the input space $\mathcal{U}$ are not finite-dimensional. This is why forecasting dynamics is hard. For example, suppose the initial condition is not known exactly but on an open interval, which is no longer finite-dimensional. In that case, we do not know what type of chaotic behavior will occur in the long-time horizon. The same thing applies to the parameter space. General variable coefficients, even for the smoothest functions in $C^\infty$, are in an infinite-dimensional space before further assumptions.

Author response:
We suspect there may be some confusion in this concern regarding the meaning of finite-dimensional spaces.
The example provided, "the initial condition is not known exactly but on an open interval," is one-dimensional as it is a continuous subset of the real line, a one-dimensional vector space.
General variable coefficients are indeed allowed by the finite-dimensionality assumption.

The finite-dimensionality assumption is commonly satisfied by many real-world systems such as robotics or circuits.
These come from practical engineering constraints, where construction of arbitrary input functions is infeasible due to physical constraints such as damping in a system.

Reviewer comment (Theory):
Even if we settle for finite-dimensional spaces for the initial condition, the parameters, the inputs, etc. (i.e., all the places where one can change the dynamics), the overall dimension can be large. Indeed, $C_{f,I}$ is the range of the forward problem, containing all possible trajectories. It is not possible to characterize the space analytically, so it will have to be done through the finite number of parameters mentioned above. In that sense, (13) becomes the standard parameterized minimization problem, like those we see in regressions. If we know a priori that the dynamics are only subject to a finite number of parameters, no one will use (12), which does not utilize this prior/expert knowledge.

Author response:
It is true that the space of solutions cannot be characterized analytically, as even simple ODEs do not have analytical solutions, though the lack of existence of analytical solutions does not cause any issues with inference.
A lack of analytical solutions does not prohibit the usage of the smooth structure in optimization problems, allowing gradient-based methods to be applied.

Equation (13) is *not* the current approach today, and we hope that other researchers begin to adopt it after reading this manuscript.
We suspect this is because it was not previously known that the objective is well-behaved enough to solve.
This guarantee comes from the core contribution of this work.

Neural ODEs and related work (Chen et al. 2018; Dupont et al., 2019; etc.) fit a trajectory to the observations, similar to a classical regression method.
In Bayesian statistics, Equation (12) represents the minimum mean squared error estimator, and is one of the main objectives of MCMC sampling.

Reviewer comment (Theory):
The paper titled "**Provably accurate** ODE forecasting through explicit trajectory optimization". I don't see where the "provably accurate" is based. Both Theorem 1 and Proposition 1 are results with little to do with ODE forecasting. Theorem 1 is a consequence of the "finite dimension" assumption, and Proposition 1 is a property from differential manifold.

Author response:
Theorem 1 and Proposition 1 enable provably optimal forecasting within a model family.
This contribution is distinct from other approaches for which the objective function is dependent explicitly on the parameters rather than on the future predictions.

Reviewer comment (Theory):
The metric tensor K is diagonal, in the sense that it is zero if the time points don't match. This is very special and does not accommodate correlations in time. Why do the authors want to choose this?

Author response:
The choice of a diagonal metric tensor in Equation (6) does not represent a limitation to this work, and the more general form in Equation (5) is applicable.
As stated in the manuscript, the diagonal structure in Equation (6) was chosen to enable a re-weighting of the importance of different time horizons, a common requirement in forecasting problems.
It represents a useful example which later has a fundamental implication for Section 4.3, where it implies point-wise optimality for the MMSE estimation in the ambient space.

Reviewer comment (Experiments):
Experiments: They work pretty well, but two things are not clear to me (a) assumptions for computation of $\det D\psi$ and (b) what is the input to the training pipeline. (a) From the appendix, it seems you compute $\det D\psi$ assuming access to the exact solver. From what I understood in the paper, it instead seems you have access only to noisy measurements. (b) this is related. seems from the paper you assume access to a single noisy trajectory, but I guess you actually use more than one. Can you make this more clear to me?

Author response:
> "Experiments: They work pretty well, but two things are not clear to me (a) assumptions for computation of and (b) what is the input to the training pipeline. (a) From the appendix, it seems you compute assuming access to the exact solver. From what I understood in the paper, it instead seems you have access only to noisy measurements. (b) this is related. seems from the paper you assume access to a single noisy trajectory, but I guess you actually use more than one. Can you make this more clear to me?"

Part (a):
We assume knowledge that true model exists in some known parameterized family of ODEs.
We then use a numerical solver to evaluate the associated transformations in this model family.
These transformations are deterministic functions of the model family, and are not dependent on the observations.

Part (b):
We observe a single noisy trajectory.
This noisy trajectory induces the probability density on which we operate with the transformations from part (a).

Reviewer comment (Experiments):
Comparisons with other methods: This is totally missing. Only variants of the proposed methods are discussed. While the discussion is still interesting, I would like the authors to show comparisons with other alternatives for solving the problem. In terms of Accuracy, Speed, Assumptions. I am totally happy to revise this score if you are able to show this.

Author response:
> "Comparisons with other methods: This is totally missing. Only variants of the proposed methods are discussed. While the discussion is still interesting, I would like the authors to show comparisons with other alternatives for solving the problem. In terms of Accuracy, Speed, Assumptions. I am totally happy to revise this score if you are able to show this."

We included simulations for the methods that are most relevant to this work.
The core contribution in this work is the recognition that statistical cost functions are well-behaved on the space of solutions of a differential equation.
It is for this reason that we compared the optimal solutions to these costs to their counterparts in the state estimation problem.

Methods with weaker assumptions are unable to capture long-term structure of the trajectory (as seen in Appendix A.2).
Furthermore, other methods emphasizing speed with similar assumptions typically aim to construct approximate solutions to the state estimation problem.
Thus, we did not include them as we felt a more thorough characterization of the difference between the optimal solutions would be more salient.

Reviewer comment (Theory):
In the main paper assumptions, you have that $\mathcal{U}$ is a manifold. However, in App. B, you assume the inputs have a finite-sum structure. Hence, it seems to me the assumption on the input set is a bit stronger than what you claim. Am I right?

Author response:
> "In the main paper assumptions, you have that is a manifold. However, in App. B, you assume the inputs have a finite-sum structure. Hence, it seems to me the assumption on the input set is a bit stronger than what you claim. Am I right?"

We appreciate your careful reading of the appendices.
You are correct regarding the mismatch between the paragraph at the beginning of Appendix B and the claim.

We will revise the paragraph to state that the parameter space can be augmented as the product manifold of $\mathcal{U}$ and $\Theta$, rather than attempting to illustrate the process concretely through a linear subspace.

Reviewer comment (Theory):
When the parameter space $\Theta$ is derived from a neural network, it is unclear whether it is possible to verify if $f$ is a smooth function, satisfying assumption 1.

Author response:
> "When the parameter space $\Theta$s derived from a neural network, is it possible to verify whether $f$ is a smooth function, satisfying assumption 1?"  

Yes, the smoothness property requires the usage of smooth activation functions.  
The smoothness assumption is common, and is additionally required in Neural ODEs and related work (Chen et al. 2018; Dupont et al., 2019; etc.).

Reviewer comment (Writing):
The meaning of the bracketed term $[D\varphi^{\gamma_i}]$ in equation (14) is unclear.

Author response:
> "What is the meaning of the bracketed term in the Equation (14)?"  

Similar to Equation (7), the $D$ notation represents the matrix of partial derivatives in local coordinates.  
Thus,  
$$D\varphi^{\tau_i} = \begin{bmatrix}  
\rule[.5ex]{2.5ex}{0.5pt} & D\_{\mathbf{v}_1}|\_{\mathbf{x}_0} \varphi^{\tau_i}& \rule[.5ex]{2.5ex}{0.5pt} \\\\  
\rule[.5ex]{2.5ex}{0.5pt} & D\_{\mathbf{v}_2}|\_{\mathbf{x}_0} \varphi^{\tau_i} & \rule[.5ex]{2.5ex}{0.5pt} \\\\  
 & \vdots &  
\end{bmatrix},$$  
where $\{\mathbf{v}_i\}$ is a basis of the local coordinates on the state space around $\mathbf{x}_0$.

Reviewer comment (Evaluation):
A comparison of the computational complexity between the proposed methods for computing trajectory estimations and the estimations of the initial condition is not provided.

Author response:
> "Could you offer a comparison of the computational complexity between the proposed methods for computing trajectory estimations and the estimations of the initial condition?"  

We believe that such a comparison would only be informative for the particular pair of differential equation and ODE solver being investigated.  
The primary source of the difference is in requiring the solutions of the differential equation to be computed over the entire forecasting horizon rather than the observation horizon.  
The relative amount of time required is dependent on the dynamically chosen step sizes in the ODE solver in the observation regime and the forecasting regime.  
These step sizes are dependent on the ODE solver, the differential equation, and the initial conditions, making it difficult to generalize statements on the computational cost.

Reviewer comment (Presentation):
Regarding Figure 2, it appears to be somewhat confusing. Is the caption of each subfigure indicating the names of methods? How do they correspond to the six different forecasting objectives used in simulations? Could you provide further insights into which surfaces of objective functions yield better results and which do not? Are these outcomes affected by changes in the initialization point? For the statement 'the trajectory MSE illustrates a valley of initializations ... structure not captured by any competing technique', it seems the State MSE also exhibits a valley structure. Could you elaborate on this observation?

Author response:
> "Is the caption of each subfigure indicating the names of methods? How do they correspond to the six different forecasting objectives used in simulations?"  

The title of each subfigure is the function in the plot, e.g., "Likelihood" represents the likelihood function of the parameters, "State Posterior" represents the posterior distribution of the state variable, etc.  
We will rework the caption to improve clarity.  

> "Could you provide further insights into which surfaces of objective functions yield better results and which do not?"  

The objective function relating directly to prediction performance should yield better results in prediction performance.  
Figure 2 was intended to illustrate the qualitative differences between costs in the space of solutions and costs in the space of initial conditions, demonstrating that the "valley of initialization which lead to similar trajectories" is not captured by other objectives.  

> "Are these outcomes affected by changes in the initialization point?"  

This work does not propose a specific optimization algorithm, but rather a new cost function dependent directly on the forecasting performance.  
As such, there is no initialization point in our proposed method.  

> "For the statement 'the trajectory MSE illustrates a valley of initializations ... structure not captured by any competing technique', it seems the State MSE also exhibits a valley structure. Could you elaborate on this observation?"  

The key part of the sentence was "which lead to similar trajectories along the interval."  
We are referring to the fact that the proposed cost function meaningfully associates similar forecasts, rather than similar initializations.  
The state MSE figure would suggest that a significant reduction in the initial prey population would be possible without sacrificing performance, whereas the trajectory MSE plot demonstrates this is not the case, and that the path is extremely sensitive to the initial prey population.

Reviewer comment (Novelty):
__Theoretical aspect.__ Main results of the paper seam, at best, based on well-known arguments, if not already present is the same form in the existing literature. How the paper is presented, my impression is that novelty is highly overstated. Giving proper references for all arguments based on classical ODE theory is necessary.

Author response:
> "Main results of the paper seam, at best, based on well-known arguments, if not already present is the same form in the existing literature. How the paper is presented, my impression is that novelty is highly overstated."

To our knowledge, the provided structural result is novel.
Other related results either require significantly stronger system constraints (e.g. Quotient manifold theorem (Lee, 2013)) or fail to provide sufficient structure for practical applications (e.g. non-Hausdorff manifold of solutions discussed by Souriau (1997)*. Without the Hausdorff structure, many common tools break down due to lack of uniqueness of limits).

The novelty is in identifying the appropriate set of constraints to induce sufficient structure for statistical analysis.
While remaining broadly applicable, the set of constraints enables the ability to directly optimize forecasts over the entire time horizon of interest.

*: J.-M. Souriau, _Structure of Dynamical Systems: a symplectic view of physics._ Springer Science+Business Media, New York, NY, 1997.


> "Giving proper references for all arguments based on classical ODE theory is necessary."

We believe that we have included proper references.
When appropriate throughout the manuscript, we include references to "Nonlinear Systems" by Hassan Khalil, a standard book on the analysis of nonlinear ODEs.
When discussing geometric approaches, we additionally reference "Introduction to Smooth Manifolds" by John Lee.
We are unclear as to which arguments you feel are missing references; we could additionally include a real analysis book for the epsilon-delta proofs.

Reviewer comment (Novelty):
__Methodological aspect.__ In related works many references study different problem - estimating dynamics when ODE is not known, which is not the problem that this paper studies. On the other hand, existing literature on the ML MAP and MMSE estimation of the initial conditions of a known ODE is not properly reviewed, and the novelty of the considered methodology lacks perspective.

Author response:
> "In related works many references study different problem - estimating dynamics when ODE is not known, which is not the problem that this paper studies."

We are unclear as to which references you feel study fundamentally different problems, as our work relies on the same assumptions as neural ODEs and related work (Chen et al. 2018; Dupont et al., 2019; etc.), is most comparable to physics-informed ML (Raissi & Karniadakis, 2018;
Raissi et al., 2019), and represents a change in cost function similar to regularization methods.
Furthermore, we believe that hierarchical forecasting  (Rangapuram et al., 2021; 2023) is implicitly operating under a similar structural assumption.

The problem of estimating dynamics and making predictions are intrinsically linked.
The benefits of algorithms such as neural ODEs are in their predictive power, not in capturing a interpretable representation of some fundamental physical principle.
Furthermore, we do not assume a fully known ODE, but rather that the ODE exists in some parameterized family of candidate models, the same underlying assumption in such a work.

> "On the other hand, existing literature on the ML MAP and MMSE estimation of the initial conditions of a known ODE is not properly reviewed, and the novelty of the considered methodology lacks perspective."

We do not believe that existing literature on ML, MAP, and MMSE estimation in dynamic systems addresses the problem of interest in this work.
Existing literature on specific objective functions largely focus on computationally efficient algorithms and approximations.

Reviewer comment (Experiments):
__Experimental aspect.__ One toy ODE model is by far bellow ICLR standard. No broader context presented. Such discussion is insufficient to draw any kind of reliable conclusions. Additional material in the Appendix is minor and in part (Section A.2) obvious.

Author response:
> "One toy ODE model is by far bellow ICLR standard. No broader context presented. Such discussion is insufficient to draw any kind of reliable conclusions."

The reliable conclusion in the manuscript is in the theoretical foundations of the work, not the numerical aspect.
The proposed analysis guarantees optimality of the forecast within the model family.
Simulations are included largely to characterize situations in which the performance gap may be meaningful (e.g., large time horizons in Figure 3).

> "Additional material in the Appendix is minor"

We believe that we have included extensive additional material throughout the appendix.
Appendix A includes simulations for three additional systems and two competing techniques.
Appendix B and Appendix D include full proofs for the main results.
Appendix C includes a review of key properties in Riemannian geometry.
Appendix E provides some analysis on tolerance selection for the ODE solvers used in the work.

> "in part (Section A.2) obvious."

While we agree that Section A.2 is well-known to many in this field, in past publications comparisons with these models have been specifically requested by reviewers.
Thus, we have included the additional figures for those for whom it is not obvious.

Reviewer comment (Theory):
Aspects of the statistical learning theory of kernel methods are not addressed properly, in particular the fact that the regression function in Eq. (12) may or may not belong to the RKHS defined by the kernel, and the minimisation is typically not done in $L^2$ but in RKSH.

Author response:
> "When introducing kernel for the trajectories, due to the change of geometries between RKHS and $L^2(I)$ spaces, the existence and the properties of the diffeomorphism should be at least commented.  Also, can you please clarify when the optimization is done in the RKSH norm and when in $L^2$ norm. To me it seams that aspects of the statistical learning theory of kernel methods are not addressed properly. In particular, regression function in Eq. (12) may or may not belong to the RKHS defined by the kernel, and the minimisation is typically not done in $L^2$ but in RKSH."

While it can be shown that the solutions of the ODE exist in an RKHS directly based on continuity of the evaluation functional, we do not change from $L^2(I)$ into an RKHS by introducing a linear operator.
It is true that positive definite integral operators induce an RKHS by Mercer's theorem, but we do not require the RKHS structure and thus feel it would serve as a distraction from the main contributions.
The core contribution in this work is to show that the space of solutions is a Riemannian manifold, not an arbitrary structure in an RKHS, nor $L^2(I)$.
The embedding in $L^2(I)$ is used only to enable naturally occurring cost functions through a restriction onto the embedded manifold and to inherit a metric.