Original: D-CODE: D ISCOVERING C LOSED - FORM ODE S FROM O BSERVED T RAJECTORIES
Revised: D-CODE: D ISCOVERING C LOSED - FORM ODE S FROM O BSERVED T RAJECTORIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For centuries, scientists have manually designed closed-form ordinary differential equations (ODEs) to model dynamical systems.
Revised: For centuries, scientists have manually designed closed-form ordinary differential equations (ODEs) to model dynamical systems.

Original: An automated tool to distill closedform ODEs from observed trajectories would accelerate the modeling process.
Revised: An automated tool to distill closedform ODEs from observed trajectories would accelerate the modeling process.

Original: Traditionally, symbolic regression is used to uncover a closed-form prediction function a = f ( b ) with label-feature pairs ( a i , b i ) as training examples.
Revised: Traditionally, symbolic regression is used to uncover a closed-form prediction function a = f ( b ) with label-feature pairs ( a i , b i ) as training examples.

Original: However, an ODE models the time derivative ˙ x ( t ) of a dynamical system, e.g. ˙ x ( t ) =
Revised: However, an ODE models the time derivative ˙ x ( t ) of a dynamical system, e.g. ˙ x ( t ) =

Original: f
Revised: f

Original: ( x ( t ) , t ) , and the “label” ˙ x ( t ) is usually not observed.
Revised: ( x ( t ) , t ) , and the “label” ˙ x ( t ) is usually not observed.

Original: The existing ways to bridge this gap only perform well for a narrow range of settings with low measurement noise and frequent sampling.
Revised: The existing ways to bridge this gap only perform well for a narrow range of settings with low measurement noise and frequent sampling.

Original: In this work, we propose the Discovery of Closedform ODE framework (D-CODE), which advances symbolic regression beyond the paradigm of supervised learning.
Revised: In this work, we propose the Discovery of Closedform ODE framework (D-CODE), which advances symbolic regression beyond the paradigm of supervised learning.

Original: D-CODE uses a novel objective function based on the variational formulation of ODEs to bypass the unobserved time derivative.
Revised: D-CODE uses a novel objective function based on the variational formulation of ODEs to bypass the unobserved time derivative.

Original: For formal justiﬁcation, we prove that this objective is a valid proxy for the estimation error of the true (but unknown) ODE.
Revised: For formal justiﬁcation, we prove that this objective is a valid proxy for the estimation error of the true (but unknown) ODE.

Original: In the experiments, D-CODE successfully discovered the governing equations of a diverse range of dynamical systems under challenging measurement settings with high noise and infrequent sampling.
Revised: In the experiments, D-CODE successfully discovered the governing equations of a diverse range of dynamical systems under challenging measurement settings with high noise and infrequent sampling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: An ordinary differential equation (ODE) links the state of a continuous-time dynamical system x ( t ) to its time derivative ˙ x ( t ) via a function f , e.g. ˙ x ( t ) =
Revised: An ordinary differential equation (ODE) links the state of a continuous-time dynamical system x ( t ) to its time derivative ˙ x ( t ) via a function f , e.g. ˙ x ( t ) =

Original: f ( x ( t ) , t ) .
Revised: f ( x ( t ) , t ) .

Original: The ODE is closed-form when f has a concise and analytical expression (e.g. f ( x, t )
Revised: The ODE is closed-form when f has a concise and analytical expression (e.g. f ( x, t )

Original: = − t log( x ) ).
Revised: = − t log( x ) ).

Original: Closed-form ODEs are ubiquitous in science and engineering.
Revised: Closed-form ODEs are ubiquitous in science and engineering.

Original: The popularity is partly due to their transparency and interpretability to human experts (Petersen et al., 2019).
Revised: The popularity is partly due to their transparency and interpretability to human experts (Petersen et al., 2019).

Original: Their concise functional form also facilitates the analysis of many key properties of the dynamical system, e.g. asymptotic stability (Lyapunov, 1992).
Revised: Their concise functional form also facilitates the analysis of many key properties of the dynamical system, e.g. asymptotic stability (Lyapunov, 1992).

Original: However, the discovery of closed-form ODEs has been laborious and time-consuming as it heavily relies on human experts (Simmons, 1972).
Revised: However, the discovery of closed-form ODEs has been laborious and time-consuming as it heavily relies on human experts (Simmons, 1972).

Original: The goal of this work is to discover closed-form ODEs in an automated and data-driven way.
Revised: The goal of this work is to discover closed-form ODEs in an automated and data-driven way.

Original: We envision that such a tool would accelerate the quantitative modeling of dynamical systems, with applications ranging from discovering the kinetics of biochemical reactions to modeling tumor growth (Schuster, 2019;
Revised: We envision that such a tool would accelerate the quantitative modeling of dynamical systems, with applications ranging from discovering the kinetics of biochemical reactions to modeling tumor growth (Schuster, 2019;

Original: Geng et al., 2017).
Revised: Geng et al., 2017).

Original: Symbolic regression is an established approach to discover a closed-form function a = f ( b ) for predicting the label a from the feature b .
Revised: Symbolic regression is an established approach to discover a closed-form function a = f ( b ) for predicting the label a from the feature b .

Original: It operates in the supervised learning setting and requires a dataset of label-feature pairs for training, i.e. D = { a i , b i } Ni =1 (Schmidt & Lipson, 2009).
Revised: It operates in the supervised learning setting and requires a dataset of label-feature pairs for training, i.e. D = { a i , b i } Ni =1 (Schmidt & Lipson, 2009).

Original: However, symbolic regression is not directly applicable to ODEs because we usually do not observe the time derivative ˙ x ( t ) , i.e. the labels for regression are not available in the data.
Revised: However, symbolic regression is not directly applicable to ODEs because we usually do not observe the time derivative ˙ x ( t ) , i.e. the labels for regression are not available in the data.

Original: In contrast, we only have access to the measurements of state y ( t ) =
Revised: In contrast, we only have access to the measurements of state y ( t ) =

Original: x ( t ) + (cid:15) at some discrete time points and with noise.
Revised: x ( t ) + (cid:15) at some discrete time points and with noise.

Original: For instance, the datasets for tumor growth modeling typically contain tumor volumes measured at different clinical visits with substantial noise (Wilkerson et al., 2017; Mazaheri et al., 2009).
Revised: For instance, the datasets for tumor growth modeling typically contain tumor volumes measured at different clinical visits with substantial noise (Wilkerson et al., 2017; Mazaheri et al., 2009).

Original: We refer to this problem as the equation-data mismatch .
Revised: We refer to this problem as the equation-data mismatch .

Original: A simple way to resolve this mismatch is to estimate ˙ x ( t ) from data (e.g. by numerical differentiation (Bickley, 1941)), and use the estimated derivatives as labels for regression (Gaucel et al., 2014).
Revised: A simple way to resolve this mismatch is to estimate ˙ x ( t ) from data (e.g. by numerical differentiation (Bickley, 1941)), and use the estimated derivatives as labels for regression (Gaucel et al., 2014).

Original: However, it is very challenging to recover the time derivative under high measurement noise or infrequent sampling (Cullum, 1971), and symbolic regression tends to perform poorly with inaccurate labels (ˇZegklitz & Poˇs´ık, 2021).
Revised: However, it is very challenging to recover the time derivative under high measurement noise or infrequent sampling (Cullum, 1971), and symbolic regression tends to perform poorly with inaccurate labels (ˇZegklitz & Poˇs´ık, 2021).

Original: This is an substantial drawback because many applications, such as healthcare, may involve noisy or infrequently sampled data (Jensen et al., 2014).
Revised: This is an substantial drawback because many applications, such as healthcare, may involve noisy or infrequently sampled data (Jensen et al., 2014).

Original: In this work, we develop the Discovery of Closed-form ODE framework (D-CODE), which extends symbolic regression beyond the supervised learning setting.
Revised: In this work, we develop the Discovery of Closed-form ODE framework (D-CODE), which extends symbolic regression beyond the supervised learning setting.

Original: The key insight behind D-CODE is the variational formulation of ODE (Hackbusch, 2017), which establishes a direct link between the trajectory x ( t ) and the ODE f while bypassing the unobservable time derivative ˙ x ( t ) .
Revised: The key insight behind D-CODE is the variational formulation of ODE (Hackbusch, 2017), which establishes a direct link between the trajectory x ( t ) and the ODE f while bypassing the unobservable time derivative ˙ x ( t ) .

Original: We develop a novel objective function based on this insight, and prove that it is a valid proxy for the estimation error of the true (but unknown) ODE.
Revised: We develop a novel objective function based on this insight, and prove that it is a valid proxy for the estimation error of the true (but unknown) ODE.

Original: We demonstrate via extensive experiments that D-CODE can uncover the governing equations for a diverse range of dynamical systems while being substantially more robust to measurement artifacts than the alternative methods.
Revised: We demonstrate via extensive experiments that D-CODE can uncover the governing equations for a diverse range of dynamical systems while being substantially more robust to measurement artifacts than the alternative methods.

Original: Finally, D-CODE is designed as a general framework, where some of its components can be ﬂexibly adapted based on the application.
Revised: Finally, D-CODE is designed as a general framework, where some of its components can be ﬂexibly adapted based on the application.

Original:  B ACKGROUND AND PROBLEM SETTING
Revised:  B ACKGROUND AND PROBLEM SETTING

Original: Curating a dataset of trajectories for ODE discovery involves many decisions, e.g. What variables to include?
Revised: Curating a dataset of trajectories for ODE discovery involves many decisions, e.g. What variables to include?

Original: When and for how long to take measurements?
Revised: When and for how long to take measurements?

Original: (we discuss dataset curation in Appendix
Revised: (we discuss dataset curation in Appendix

Original: C.)
Revised: C.)

Original: In this work, we assume the dataset is given and the variables can be modeled by a system of ﬁrst-order autonomous ODEs (Eq. 1).
Revised: In this work, we assume the dataset is given and the variables can be modeled by a system of ﬁrst-order autonomous ODEs (Eq. 1).

Original: We note that higher-order or time-dependent ODEs can be represented in this form by curating a dataset with additional variables (Simmons, 1972)
Revised: We note that higher-order or time-dependent ODEs can be represented in this form by curating a dataset with additional variables (Simmons, 1972)

Original: (Appendix
Revised: (Appendix

Original: B).
Revised: B).

Original: The system with J ∈ N + variables is deﬁned as
Revised: The system with J ∈ N + variables is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where we use Newton’s notation ˙ x j ( t ) for the time derivative.
Revised: ) where we use Newton’s notation ˙ x j ( t ) for the time derivative.

Original: The functions f j : R J → R will be sometimes referred to as the ODEs directly.
Revised: The functions f j : R J → R will be sometimes referred to as the ODEs directly.

Original: We denote T ∈ R + as the maximum time horizon we have data for.
Revised: We denote T ∈ R + as the maximum time horizon we have data for.

Original: We highlight the following distinction: the trajectory x j : [0 , T ] → R is a function of time, whereas the state x j ( t ) ∈ R , ∀ t ∈ [0 , T ] is a point on the trajectory 1 .
Revised: We highlight the following distinction: the trajectory x j : [0 , T ] → R is a function of time, whereas the state x j ( t ) ∈ R , ∀ t ∈ [0 , T ] is a point on the trajectory 1 .

Original: We denote the state vector x ( t ) :=
Revised: We denote the state vector x ( t ) :=

Original: [ x 1 ( t ) , . . .
Revised: [ x 1 ( t ) , . . .

Original: , x J ( t )] (cid:62) ∈ R J and the vector-valued trajectory function x :=
Revised: , x J ( t )] (cid:62) ∈ R J and the vector-valued trajectory function x :=

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: , x J ] .
Revised: , x J ] .

Original: Let f ∗ j ’s be the true but unknown ODEs to be uncovered, and x i : [0 , T ] → R J , i ≤ N , N ∈ N + be the true trajectories that satisfy f ∗ j ’s. In practice, we only measure the true trajectories at discrete times and with noise.
Revised: Let f ∗ j ’s be the true but unknown ODEs to be uncovered, and x i : [0 , T ] → R J , i ≤ N , N ∈ N + be the true trajectories that satisfy f ∗ j ’s. In practice, we only measure the true trajectories at discrete times and with noise.

Original: Denote the measurement of trajectory i at time t as y i ( t ) ∈
Revised: Denote the measurement of trajectory i at time t as y i ( t ) ∈

Original: R J ; we assume
Revised: R J ; we assume

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) i ( t ) ∈
Revised: where (cid:15) i ( t ) ∈

Original: R J is zero-mean noise with standard deviation σ .
Revised: R J is zero-mean noise with standard deviation σ .

Original: The measurements are made at time t ∈ T = { t 1 , t 2 , . .
Revised: The measurements are made at time t ∈ T = { t 1 , t 2 , . .

Original: , T } .
Revised: , T } .

Original: We denote the dataset as D =
Revised: We denote the dataset as D =

Original: { y i ( t ) | i ≤ N, t ∈ T } .
Revised: { y i ( t ) | i ≤ N, t ∈ T } .

Original: Closed form .
Revised: Closed form .

Original: The function f
Revised: The function f

Original: j : R J → R has a closed form if it can be expressed as a ﬁnite sequence of operations ( + , ÷ , log , . . . )
Revised: j : R J → R has a closed form if it can be expressed as a ﬁnite sequence of operations ( + , ÷ , log , . . . )

Original: , input variables ( x 1 , x 2 , . . . )
Revised: , input variables ( x 1 , x 2 , . . . )

Original: and numeric constants ( 1 . 5 , 0 . 8 , . . . )
Revised: and numeric constants ( 1 . 5 , 0 . 8 , . . . )

Original: (Borwein et al., 2013).
Revised: (Borwein et al., 2013).

Original: The functional form of f j is the expression with all the numeric constants replaced by placeholders θ k ’s, e.g. f j ( x ) =
Revised: The functional form of f j is the expression with all the numeric constants replaced by placeholders θ k ’s, e.g. f j ( x ) =

Original: θ 1 x · log( θ 2 x ) .
Revised: θ 1 x · log( θ 2 x ) .

Original: To fully uncover f ∗ j , we need to infer its functional form and estimate the unknown constants θ k ’s (if any).
Revised: To fully uncover f ∗ j , we need to infer its functional form and estimate the unknown constants θ k ’s (if any).

Original: Variational formulation .
Revised: Variational formulation .

Original: The variational formulation provides a direct link between the trajectory x and the ODE f j without involving ˙ x
Revised: The variational formulation provides a direct link between the trajectory x and the ODE f j without involving ˙ x

Original: (Hackbusch, 2017).
Revised: (Hackbusch, 2017).

Original: We start with the following deﬁnition.
Revised: We start with the following deﬁnition.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Consider J ∈ N + , T ∈ R + , continuous functions x : [0 , T ] → R J , f : R J → R , and g ∈ C 1 [0 , T ] , where C 1 is the set of continuously differentiable functions.
Revised: Consider J ∈ N + , T ∈ R + , continuous functions x : [0 , T ] → R J , f : R J → R , and g ∈ C 1 [0 , T ] , where C 1 is the set of continuously differentiable functions.

Original: We deﬁne the functionals
Revised: We deﬁne the functionals

Original: [Equation]
Revised: [Equation]

Original: Importantly, the functional C j depends on the testing function g ( t ) and its derivative ˙ g ( t ) but not ˙ x j .
Revised: Importantly, the functional C j depends on the testing function g ( t ) and its derivative ˙ g ( t ) but not ˙ x j .

Original: Proposition 1 below provides the variational formulation of ODE.
Revised: Proposition 1 below provides the variational formulation of ODE.

Original: Essentially it speciﬁes the inﬁnitely many constraints that x has to satisfy in order to be a solution to the ODE.
Revised: Essentially it speciﬁes the inﬁnitely many constraints that x has to satisfy in order to be a solution to the ODE.

Original: (see Appendix A).
Revised: (see Appendix A).

Revised: [Table]

Original: Proposition 1.
Revised: Proposition 1.

Original: (Hackbusch, 2017) Consider J ∈ N + , T ∈ R + , a continuously differentiable function x : [0 , T ] → R J , and continuous functions f j : R J → R for j = 1 , . . .
Revised: (Hackbusch, 2017) Consider J ∈ N + , T ∈ R + , a continuously differentiable function x : [0 , T ] → R J , and continuous functions f j : R J → R for j = 1 , . . .

Original: , J .
Revised: , J .

Original: Then x is the solution to the system of ODEs in Equation 

Revised: Then x is the solution to the system of ODEs in Equation

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original:  R ELATED WORK 3. S YMBOLIC REGRESSION
Revised:  R ELATED WORK 3. S YMBOLIC REGRESSION

Original: Symbolic regression attempts to uncover a closed-form prediction function a = f ( b ) using supervised learning (Table 1).
Revised: Symbolic regression attempts to uncover a closed-form prediction function a = f ( b ) using supervised learning (Table 1).

Original: The main challenge is optimization : searching for the optimal f is thought to be NP-hard because the space of closed-form functions is vast and complex—it is combinatorial in the functional form and continuous in the numeric constants (Lu et al., 2016).
Revised: The main challenge is optimization : searching for the optimal f is thought to be NP-hard because the space of closed-form functions is vast and complex—it is combinatorial in the functional form and continuous in the numeric constants (Lu et al., 2016).

Original: Therefore, most existing work focuses on developing optimization algorithms.
Revised: Therefore, most existing work focuses on developing optimization algorithms.

Original: Genetic programming has been one of the most successfully and widely used methods (Koza, 1994; Schmidt & Lipson, 2009).
Revised: Genetic programming has been one of the most successfully and widely used methods (Koza, 1994; Schmidt & Lipson, 2009).

Original: It represents f as a tree where the internal nodes are operations and the leaves are variables or constants (Figure 1 B).
Revised: It represents f as a tree where the internal nodes are operations and the leaves are variables or constants (Figure 1 B).

Original: It then applies genetic algorithm to search for the best tree representation (Forrest, 1993).
Revised: It then applies genetic algorithm to search for the best tree representation (Forrest, 1993).

Original: More recently, AI Feynman introduces a set of heuristics to directly prune the search space, thereby improving the optimization efﬁciency (Udrescu & Tegmark, 2020).
Revised: More recently, AI Feynman introduces a set of heuristics to directly prune the search space, thereby improving the optimization efﬁciency (Udrescu & Tegmark, 2020).

Original: Optimization methods based on reinforcement learning (Petersen et al., 2019), pre-trained neural networks (Biggio et al., 2021), and
Revised: Optimization methods based on reinforcement learning (Petersen et al., 2019), pre-trained neural networks (Biggio et al., 2021), and

Original: Meijer G-functions (Alaa & van der Schaar, 2019) have also been proposed.
Revised: Meijer G-functions (Alaa & van der Schaar, 2019) have also been proposed.

Original: All methods above use prediction error (e.g. RMSE) as the objective, which is not applicable to ODE discovery because the label (time derivative) is not observed.
Revised: All methods above use prediction error (e.g. RMSE) as the objective, which is not applicable to ODE discovery because the label (time derivative) is not observed.

Original: D-CODE advances symbolic regression beyond the supervised learning setting and addresses the unique challenges for ODE discovery: the equation-data mismatch and the sensitivity to noisy or infrequent observations.
Revised: D-CODE advances symbolic regression beyond the supervised learning setting and addresses the unique challenges for ODE discovery: the equation-data mismatch and the sensitivity to noisy or infrequent observations.

Original: 3. D ATA - DRIVEN DISCOVERY OF CLOSED - FORM ODE S
Revised: 3. D ATA - DRIVEN DISCOVERY OF CLOSED - FORM ODE S

Original: Two-step sparse regression performs sparse regression on the estimated time derivative (cid:98) ˙
Revised: Two-step sparse regression performs sparse regression on the estimated time derivative (cid:98) ˙

Original: x j (Brunton et al., 2016; Rudy et al., 2017).
Revised: x j (Brunton et al., 2016; Rudy et al., 2017).

Original: It assumes that the true ODE can be symbolically expressed in a linear form: ˙ x j ( t ) =
Revised: It assumes that the true ODE can be symbolically expressed in a linear form: ˙ x j ( t ) =

Original: (cid:80) K k
Revised: (cid:80) Kk

Original: [-=
-] θ k h k
Revised: [-=
-]=1 θ k h k

Original: ( x ( t )) where θ k ∈ R are unknown constants and h k : R D
Revised: ( x ( t )) where θ k ∈ R are unknown constants and h k : R D

Original: → R are pre-speciﬁed candidate functions, such as monomials x, x 2 , . . .
Revised: → R are pre-speciﬁed candidate functions, such as monomials x, x 2 , . . .

Original: (Table 1).
Revised: (Table 1).

Original: The method uses L regularization on θ k to ensure the learned function only contains a few terms.
Revised: The method uses L regularization on θ k to ensure the learned function only contains a few terms.

Original: We review the methods to estimate time derivative in Appendix E.
Revised: We review the methods to estimate time derivative in Appendix E.

Original: However, the assumed linear form is very restrictive (e.g. x 3 / 2 or 1 / ( x + 2) are not allowed).
Revised: However, the assumed linear form is very restrictive (e.g. x 3 / 2 or 1 / ( x + 2) are not allowed).

Original: In fact, it cannot symbolically represent many well-known ODEs (e.g. generalized logistic model).
Revised: In fact, it cannot symbolically represent many well-known ODEs (e.g. generalized logistic model).

Original: Furthermore, the choice of candidate functions heavily depends on the human experts, which deviates from our goal of automated and data-driven ODE discovery.
Revised: Furthermore, the choice of candidate functions heavily depends on the human experts, which deviates from our goal of automated and data-driven ODE discovery.

Original: Two-step symbolic regression also uses the estimated time derivatives (cid:98) ˙
Revised: Two-step symbolic regression also uses the estimated time derivatives (cid:98) ˙

Original: x j as the label but applies symbolic regression instead (Gaucel et al., 2014) (Table 1).
Revised: x j as the label but applies symbolic regression instead (Gaucel et al., 2014) (Table 1).

Original: It may use any optimization method discussed in Section 3.1 to search for the optimal function.
Revised: It may use any optimization method discussed in Section 3.1 to search for the optimal function.

Original: In this way, it removes the linear assumption on the functional form and can discover closed-form functions in general.
Revised: In this way, it removes the linear assumption on the functional form and can discover closed-form functions in general.

Original: The main difﬁculty is that the true label (the derivative) may not be accurately recoverable due to measurement and discretization error, but symbolic regression is sensitive to inaccurate labels because of the large search space (Agapitos et al., 2012).
Revised: The main difﬁculty is that the true label (the derivative) may not be accurately recoverable due to measurement and discretization error, but symbolic regression is sensitive to inaccurate labels because of the large search space (Agapitos et al., 2012).

Original: As we show experimentally in Section 5, even with the state-of-the-art methods for estimating derivatives, this approach still often fails when the noise is relatively low.
Revised: As we show experimentally in Section 5, even with the state-of-the-art methods for estimating derivatives, this approach still often fails when the noise is relatively low.

Original: We brieﬂy review the methods for estimating derivatives from y ( t ) in Appendix E and show why this is generally a very challenging problem in Section 4.2.
Revised: We brieﬂy review the methods for estimating derivatives from y ( t ) in Appendix E and show why this is generally a very challenging problem in Section 4.2.

Original: The function approximator approach learns the true ODE with a function approximator (cid:98) f , such as a neural network (NN) (Chen et al., 2018; Rubanova et al., 2019) or a Gaussian process (GP) (Heinonen et al., 2018).
Revised: The function approximator approach learns the true ODE with a function approximator (cid:98) f , such as a neural network (NN) (Chen et al., 2018; Rubanova et al., 2019) or a Gaussian process (GP) (Heinonen et al., 2018).

Original: It estimates the unknown initial condition (cid:98) x (0) and predicts the entire trajectory (cid:98) x by solving the initial value problem (IVP) of the approximated ODE (cid:98) f .
Revised: It estimates the unknown initial condition (cid:98) x (0) and predicts the entire trajectory (cid:98) x by solving the initial value problem (IVP) of the approximated ODE (cid:98) f .

Original: The model is trained by minimizing the prediction error between predicted and the observed trajectories (Table 1).
Revised: The model is trained by minimizing the prediction error between predicted and the observed trajectories (Table 1).

Original: Importantly, the function approximator approach does not give a concise closed-form expression to describe the dynamics, which is the main objective of this work.
Revised: Importantly, the function approximator approach does not give a concise closed-form expression to describe the dynamics, which is the main objective of this work.

Original: Furthermore, its performance strongly depends on the prediction horizon used for training.
Revised: Furthermore, its performance strongly depends on the prediction horizon used for training.

Original: Using an inappropriate horizon may fail with certain type of systems, e.g. chaotic systems (Ditto & Munakata, 1995).
Revised: Using an inappropriate horizon may fail with certain type of systems, e.g. chaotic systems (Ditto & Munakata, 1995).

Original: We further discuss this issue and provide experimental results in Appendix H.2.
Revised: We further discuss this issue and provide experimental results in Appendix H.2.

Original:  M ETHOD 4. T HE D-CODE A LGORITHM
Revised:  M ETHOD 4. T HE D-CODE A LGORITHM

Original: D-CODE consists of a preprocessing step and an optimization step.
Revised: D-CODE consists of a preprocessing step and an optimization step.

Original: Figure 1 provides a schematic illustration of D-CODE and Appendix B provides the pseudocode.
Revised: Figure 1 provides a schematic illustration of D-CODE and Appendix B provides the pseudocode.

Original: The two steps are detailed below.
Revised: The two steps are detailed below.

Original: Preprocessing .
Revised: Preprocessing .

Original: D-CODE starts by estimating the trajectories (cid:98) x
Revised: D-CODE starts by estimating the trajectories (cid:98) x

Original: i : [0 , T ] → R J , i ≤ N from the noisy and discretely-sampled data D as an approximation to the true trajectories x i .
Revised: i : [0 , T ] → R J , i ≤ N from the noisy and discretely-sampled data D as an approximation to the true trajectories x i .

Original: Denoising and interpolating signals is a well-studied problem in statistics and signal processing, with many proven solutions including Gaussian process and spline regression (Bernardo et al., 1998; Marsh & Cormier, 2001).
Revised: Denoising and interpolating signals is a well-studied problem in statistics and signal processing, with many proven solutions including Gaussian process and spline regression (Bernardo et al., 1998; Marsh & Cormier, 2001).

Original: The D-CODE framework is agnostic to the exact choice of the smoothing algorithm.
Revised: The D-CODE framework is agnostic to the exact choice of the smoothing algorithm.

Original: The user should choose a suitable algorithm based on the application and adopt best practices (e.g. Is the noise distribution Gaussian?
Revised: The user should choose a suitable algorithm based on the application and adopt best practices (e.g. Is the noise distribution Gaussian?

Original: Are the measurements made at regular intervals?
Revised: Are the measurements made at regular intervals?

Original: etc.).
Revised: etc.).

Original: Optimization .
Revised: Optimization .

Original: After estimating the trajectories (cid:98) x
Revised: After estimating the trajectories (cid:98) x

Original: i , we search for the function (cid:98) f j , ∀ j ≤ J that is consistent with (cid:98) x i .
Revised: i , we search for the function (cid:98) f j , ∀ j ≤ J that is consistent with (cid:98) x i .

Original: Speciﬁcally, we solve the following optimization problems for all j ≤ J :
Revised: Speciﬁcally, we solve the following optimization problems for all j ≤ J :

Original: [Equation]
Revised: [Equation]

Original: where C j is the functional deﬁned in Equation 3 and computed using numerical integration (Davis & Rabinowitz, 2007).
Revised: where C j is the functional deﬁned in Equation 3 and computed using numerical integration (Davis & Rabinowitz, 2007).

Original: The search space of f is the set of closed-form functions, and the user may instantiate D-CODE with any optimization algorithm proposed for symbolic regression (Section 3).
Revised: The search space of f is the set of closed-form functions, and the user may instantiate D-CODE with any optimization algorithm proposed for symbolic regression (Section 3).

Original: We use a set of pre-deﬁned testing functions g s ∈ C 1 [0 , T ] , s ≤ S (to be discussed in Section 4.2).
Revised: We use a set of pre-deﬁned testing functions g s ∈ C 1 [0 , T ] , s ≤ S (to be discussed in Section 4.2).

Original: Comparison with two-step symbolic regression .
Revised: Comparison with two-step symbolic regression .

Original: As depicted in Figure 1, D-CODE and two-step symbolic regression have two main differences.
Revised: As depicted in Figure 1, D-CODE and two-step symbolic regression have two main differences.

Original: (1) The estimands in the pre-processing steps are different: D-CODE estimates the true trajectory x while the two-step symbolic regression estimates the time derivative.
Revised: (1) The estimands in the pre-processing steps are different: D-CODE estimates the true trajectory x while the two-step symbolic regression estimates the time derivative.

Original: The optimization objectives are different: D-CODE optimizes the loss function in Equation 5 while the two-step symbolic regression optimizes the prediction loss, e.g. RMSE.
Revised: The optimization objectives are different: D-CODE optimizes the loss function in Equation 5 while the two-step symbolic regression optimizes the prediction loss, e.g. RMSE.

Original: 4. T HEORETICAL R ESULTS
Revised: 4. T HEORETICAL R ESULTS

Original: In this section we provide a formal justiﬁcation of the objective in Eq.
Revised: In this section we provide a formal justiﬁcation of the objective in Eq.

Original: 5 and describe the shortcomings of the objectives used in other methods (Table 1).
Revised: 5 and describe the shortcomings of the objectives used in other methods (Table 1).

Original: For ease of exposition, we assume that our dataset contains only one trajectory, i.e. N = 1 .
Revised: For ease of exposition, we assume that our dataset contains only one trajectory, i.e. N = 1 .

Original: The results can be easily extended to multiple trajectories.
Revised: The results can be easily extended to multiple trajectories.

Original: Distance between f and f ∗ .
Revised: Distance between f and f ∗ .

Original: A reasonable objective should measure the distance between a candidate function f and the target function f ∗ (i.e. the true ODE).
Revised: A reasonable objective should measure the distance between a candidate function f and the target function f ∗ (i.e. the true ODE).

Original: By minimizing the distance, the candidate f would better approximate f ∗ .
Revised: By minimizing the distance, the candidate f would better approximate f ∗ .

Original: A common way to measure the distance between functions is to use the metrics induced by L p norms of the function space.
Revised: A common way to measure the distance between functions is to use the metrics induced by L p norms of the function space.

Original: However, the L p norms consider the values in the entire domain R J .
Revised: However, the L p norms consider the values in the entire domain R J .

Original: This is a disadvantage because most dynamical systems in nature operate within speciﬁc range and scale, and R J contains areas where we do not have knowledge about or cannot collect data from.
Revised: This is a disadvantage because most dynamical systems in nature operate within speciﬁc range and scale, and R J contains areas where we do not have knowledge about or cannot collect data from.

Original: A natural solution is to restrict the comparison on the trajectory x , which can be achieved by composing f and f ∗ with x .
Revised: A natural solution is to restrict the comparison on the trajectory x , which can be achieved by composing f and f ∗ with x .

Original: We propose the following distance function:
Revised: We propose the following distance function:

Original: [Equation]
Revised: [Equation]

Original: (6) where ◦ denotes function composition and x is a true trajectory satisfying the ODE f ∗ .
Revised: where ◦ denotes function composition and x is a true trajectory satisfying the ODE f ∗ .

Original: In Appendix
Revised: In Appendix

Original: A, we further justify and discuss the properties of d x ( f, f ∗ ) .
Revised: A, we further justify and discuss the properties of d x ( f, f ∗ ) .

Original: However, in practice, the d x ( f, f ∗ ) cannot be computed from data because it depends on the unknown f ∗ and x .
Revised: However, in practice, the d x ( f, f ∗ ) cannot be computed from data because it depends on the unknown f ∗ and x .

Original: Next, we show that the objective in Eq. 5, which is computable from data, can be used to approximate this distance.
Revised: Next, we show that the objective in Eq. 5, which is computable from data, can be used to approximate this distance.

Original: Convergence to distance .
Revised: Convergence to distance .

Original: The ﬁrst step of our algorithm estimates a trajectory (cid:98) x from the measurements { y ( t ) | t ∈ T } as an approximation to the true trajectory x .
Revised: The ﬁrst step of our algorithm estimates a trajectory (cid:98) x from the measurements { y ( t ) | t ∈ T } as an approximation to the true trajectory x .

Original: For a suitable smoothing algorithm, if we increase the number of samples on the trajectory, the recovered trajectory should converge to the ground truth.
Revised: For a suitable smoothing algorithm, if we increase the number of samples on the trajectory, the recovered trajectory should converge to the ground truth.

Original: This is achieved, for instance, by spline regression (Stone, 1985; 1994) or Gaussian processes Choi & Schervish (2007).
Revised: This is achieved, for instance, by spline regression (Stone, 1985; 1994) or Gaussian processes Choi & Schervish (2007).

Original: If this is the case, Theorem 1 shows that given high enough sampling frequency and large enough number of testing functions our objective converges to the squared distance d x ( f, f ∗ ) 2 , which measures how different f is from f ∗ .
Revised: If this is the case, Theorem 1 shows that given high enough sampling frequency and large enough number of testing functions our objective converges to the squared distance d x ( f, f ∗ ) 2 , which measures how different f is from f ∗ .

Original: That justiﬁes our objective.
Revised: That justiﬁes our objective.

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider J ∈ N + , j ∈ { 1 , . .
Revised: Consider J ∈ N + , j ∈ { 1 , . .

Original: , J } , T ∈ R + .
Revised: , J } , T ∈ R + .

Original: Let f ∗ : R J
Revised: Let f ∗ : R J

Original: → R be a continuous function, and let x : [0 , T ] → R J be a continuously differentiable function satisfying ˙ x j ( t ) =
Revised: → R be a continuous function, and let x : [0 , T ] → R J be a continuously differentiable function satisfying ˙ x j ( t ) =

Original: f ∗ ( x ( t )) .
Revised: f ∗ ( x ( t )) .

Original: Consider a sequence of functions ( (cid:98) x k ) , where (cid:98) x k : [0 , T ] → R J is a continuously differentiable function.
Revised: Consider a sequence of functions ( (cid:98) x k ) , where (cid:98) x k : [0 , T ] → R J is a continuously differentiable function.

Original: If ( (cid:98) x k ) converges to x in L 2 norm.
Revised: If ( (cid:98) x k ) converges to x in L 2 norm.

Original: Then for any Lipschitz continuous function f
Revised: Then for any Lipschitz continuous function f

Original: [Equation]
Revised: [Equation]

Original: where { g 1 , g 2 , . . .
Revised: where { g 1 , g 2 , . . .

Original: } is a Hilbert (orthonormal) basis for L 2 [0 , T ] such that ∀ i , g i (0) =
Revised: } is a Hilbert (orthonormal) basis for L 2 [0 , T ] such that ∀ i , g i (0) =

Original: g i ( T ) =
Revised: g i ( T ) =

Original: and g i ∈ C 1 [0 , T ] .
Revised: and g i ∈ C 1 [0 , T ] .

Original: We provide the proof of Theorem 1 in the Appendix A.
Revised: We provide the proof of Theorem 1 in the Appendix A.

Original: Comparison with other methods .
Revised: Comparison with other methods .

Original: Theorem 1 does not place any additional constraints regarding the smoothing algorithm apart from the convergence of the regression function.
Revised: Theorem 1 does not place any additional constraints regarding the smoothing algorithm apart from the convergence of the regression function.

Original: In general, the convergence of functions does not imply the convergence of the derivatives (example in Appendix
Revised: In general, the convergence of functions does not imply the convergence of the derivatives (example in Appendix

Original: A).
Revised: A).

Original: That means that in contrast to D-CODE, other objectives in Table 1 place constraints on the type estimation algorithm, i.e., the estimated derivative ˆ˙ x needs to converge to the true derivative ˙ x .
Revised: That means that in contrast to D-CODE, other objectives in Table 1 place constraints on the type estimation algorithm, i.e., the estimated derivative ˆ˙ x needs to converge to the true derivative ˙ x .

Original: Even if the estimation algorithm satisﬁes this constraint (for instance, spline regression (Zhou & Wolfe, 2000)), the convergence rate for derivatives is slower than for the functions themselves (Stone, 1982).
Revised: Even if the estimation algorithm satisﬁes this constraint (for instance, spline regression (Zhou & Wolfe, 2000)), the convergence rate for derivatives is slower than for the functions themselves (Stone, 1982).

Original: Convergence rate depends on the smoothness (differentiability class) of the function and differentiation reduces its differentiability class.
Revised: Convergence rate depends on the smoothness (differentiability class) of the function and differentiation reduces its differentiability class.

Original: Namely, if a true trajectory has a maximum number of m continuous derivatives then its derivative ˙ x has only m − 1 continuous derivatives.
Revised: Namely, if a true trajectory has a maximum number of m continuous derivatives then its derivative ˙ x has only m − 1 continuous derivatives.

Original: Choice of testing functions .
Revised: Choice of testing functions .

Original: As stated in Theorem 1, we should use testing functions that are orthonormal basis for L 2 [0 , T ] and satisfy g (0)
Revised: As stated in Theorem 1, we should use testing functions that are orthonormal basis for L 2 [0 , T ] and satisfy g (0)

Original: =
Revised: =

Original: g ( T ) = 0 .
Revised: g ( T ) = 0 .

Original: Ideally, they should also have analytical time derivatives ˙ g ( t ) for efﬁcient computation of the functional C (Eq. 3).
Revised: Ideally, they should also have analytical time derivatives ˙ g ( t ) for efﬁcient computation of the functional C (Eq. 3).

Original: Hence, a natural choice is the sine functions g s ( t ) =
Revised: Hence, a natural choice is the sine functions g s ( t ) =

Original: (cid:112) 2 /T · sin( sπt/T ) , which we will use in all the main experiments.
Revised: (cid:112) 2 /T · sin( sπt/T ) , which we will use in all the main experiments.

Original: Another possibility is the cubic spline functions (investigated in Appendix G).
Revised: Another possibility is the cubic spline functions (investigated in Appendix G).

Original: Although Theoremuses inﬁnitely many testing functions to establish convergence, we observe in the experiments that typically 40-60 sine functions are sufﬁcient (a sensitivity analysis is conducted in Appendix G).
Revised: Although Theoremuses inﬁnitely many testing functions to establish convergence, we observe in the experiments that typically 40-60 sine functions are sufﬁcient (a sensitivity analysis is conducted in Appendix G).

Original:  E XPERIMENTS AND EVALUATION
Revised:  E XPERIMENTS AND EVALUATION

Original: In this section, we perform a series of simulations to evaluate whether the algorithms can discover the underlying closed-form ODEs that govern the observed trajectories.
Revised: In this section, we perform a series of simulations to evaluate whether the algorithms can discover the underlying closed-form ODEs that govern the observed trajectories.

Original: Choice of dynamical systems.
Revised: Choice of dynamical systems.

Original: In this study, we select ﬁve dynamical systems governed by nine closed-form ODEs (one of them is discussed and examined in Appendix G due to space limit).
Revised: In this study, we select ﬁve dynamical systems governed by nine closed-form ODEs (one of them is discussed and examined in Appendix G due to space limit).

Original: The selected systems have varying complexity and different properties of the temporal dynamics.
Revised: The selected systems have varying complexity and different properties of the temporal dynamics.

Original: We start with two common growth models, the Gompertz model and the generalized logistic model (Gompertz, 1825; Richards, 1959).
Revised: We start with two common growth models, the Gompertz model and the generalized logistic model (Gompertz, 1825; Richards, 1959).

Original: Both models involve one variable governed by a nonlinear
Revised: Both models involve one variable governed by a nonlinear

Original: ODE and converging to a global ﬁxed point.
Revised: ODE and converging to a global ﬁxed point.

Original: Next, we consider the glycolytic oscillator in biochemistry (Sel’Kov, 1968), which is a standard benchmark problem for dynamical system prediction and inference (Daniels & Nemenman, 2015a;b).
Revised: Next, we consider the glycolytic oscillator in biochemistry (Sel’Kov, 1968), which is a standard benchmark problem for dynamical system prediction and inference (Daniels & Nemenman, 2015a;b).

Original: The system involves two nonlinearly interacting variables that converge to a oscillatory limit cycle.
Revised: The system involves two nonlinearly interacting variables that converge to a oscillatory limit cycle.

Original: Finally, we consider the chaotic Lorenz system, which involves three variables forming a strange attractor (Lorenz, 1963).
Revised: Finally, we consider the chaotic Lorenz system, which involves three variables forming a strange attractor (Lorenz, 1963).

Original: Together, these systems represent a range of temporal dynamics and application scenarios.
Revised: Together, these systems represent a range of temporal dynamics and application scenarios.

Original: Measurement settings.
Revised: Measurement settings.

Original: For each dynamical system, we consider different measurement settings speciﬁed by (1) the measurement noise level σ R , (2) the sampling step size ∆ t , and (3) the number of trajectories N .
Revised: For each dynamical system, we consider different measurement settings speciﬁed by (1) the measurement noise level σ R , (2) the sampling step size ∆ t , and (3) the number of trajectories N .

Original: Since the dynamical systems have different scales, we will report the noise-tosignal ratio σ R =
Revised: Since the dynamical systems have different scales, we will report the noise-tosignal ratio σ R =

Original: σ/ std
Revised: σ/ std

Original: ( x ( t )) for ease of comparison.
Revised: ( x ( t )) for ease of comparison.

Original: We sample the system at regular intervals
Revised: We sample the system at regular intervals

Original: T = { ∆ t, 2∆ t, . . .
Revised: T = { ∆ t, 2∆ t, . . .

Original: T } to show the effect of changing sampling frequency.
Revised: T } to show the effect of changing sampling frequency.

Original: For each setting, we perform 100 independent simulation runs to compute the evaluation metric and its conﬁdence interval.
Revised: For each setting, we perform 100 independent simulation runs to compute the evaluation metric and its conﬁdence interval.

Original: Data generation.
Revised: Data generation.

Original: For each trajectory, we ﬁrst sample the initial condition x i (0) from a uniform distribution (speciﬁed in Appendix F).
Revised: For each trajectory, we ﬁrst sample the initial condition x i (0) from a uniform distribution (speciﬁed in Appendix F).

Original: Then we obtain the true x
Revised: Then we obtain the true x

Original: i by solving the IVP computationally.
Revised: i by solving the IVP computationally.

Original: Finally, we obtain y i ( t ) by adding independent Gaussian noise and sampling at discrete time steps.
Revised: Finally, we obtain y i ( t ) by adding independent Gaussian noise and sampling at discrete time steps.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use three metrics to study various aspects of the algorithms.
Revised: We use three metrics to study various aspects of the algorithms.

Original: (1) The probability of successfully recovering the functional form ( Success Prob. ).
Revised: (1) The probability of successfully recovering the functional form ( Success Prob. ).

Original: We use a computer algebra system to judge whether the true and discovered functional forms are equivalent (Meurer et al., 2017)
Revised: We use a computer algebra system to judge whether the true and discovered functional forms are equivalent (Meurer et al., 2017)

Original: (e.g. θ 1 + x 1 ≡ x 1 + θ 1 ).
Revised: (e.g. θ 1 + x 1 ≡ x 1 + θ 1 ).

Original: (2) When the functional form is correct, we evaluate the accuracy of the estimated constants ˆ θ using root mean square error ( RMSE ).
Revised: (2) When the functional form is correct, we evaluate the accuracy of the estimated constants ˆ θ using root mean square error ( RMSE ).

Original: (3) We evaluate how well the estimated (cid:98) f approximates f ∗ using the distance function in Equation 6 ( Dist. ).
Revised: (3) We evaluate how well the estimated (cid:98) f approximates f ∗ using the distance function in Equation 6 ( Dist. ).

Original: Note that it is possible that (cid:98) f has a wrong functional form but still approximates f ∗ well.
Revised: Note that it is possible that (cid:98) f has a wrong functional form but still approximates f ∗ well.

Original: Algorithm and Benchmarks.
Revised: Algorithm and Benchmarks.

Original: We instantiate D-CODE with Gaussian process for smoothing and genetic programming for optimization due to their success and popularity in the literature (Gramacy, 2020; Schmidt & Lipson, 2009).
Revised: We instantiate D-CODE with Gaussian process for smoothing and genetic programming for optimization due to their success and popularity in the literature (Gramacy, 2020; Schmidt & Lipson, 2009).

Original: Our main benchmark is the two-step symbolic regression method.
Revised: Our main benchmark is the two-step symbolic regression method.

Original: We consider three variants with different methods for estimating the derivative: Total variation regularized differentiation ( SR-T ) (Chartrand, 2011), Spline-smoothed differentiation ( SR-S )
Revised: We consider three variants with different methods for estimating the derivative: Total variation regularized differentiation ( SR-T ) (Chartrand, 2011), Spline-smoothed differentiation ( SR-S )

Original: (Ahnert & Abel, 2007), and Gaussian process smoothed differentiation ( SR-G ) (detailed in Appendix E).
Revised: (Ahnert & Abel, 2007), and Gaussian process smoothed differentiation ( SR-G ) (detailed in Appendix E).

Original: Note that SR-G and D-CODE both ﬁt Gaussian process in the pre-processing step but they use different objective functions.
Revised: Note that SR-G and D-CODE both ﬁt Gaussian process in the pre-processing step but they use different objective functions.

Original: For fairness of comparison, we use genetic programming for optimization in all these benchmarks.
Revised: For fairness of comparison, we use genetic programming for optimization in all these benchmarks.

Original: We also compare with Neural ODEs in some cases, although it does not give closed-form expressions.
Revised: We also compare with Neural ODEs in some cases, although it does not give closed-form expressions.

Original: Appendix F contains further implementation details.
Revised: Appendix F contains further implementation details.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Gompertz and generalized logistic growth models .
Revised: Gompertz and generalized logistic growth models .

Original: These two models are widely used to capture asymmetric growth with saturation.
Revised: These two models are widely used to capture asymmetric growth with saturation.

Original: Their governing equations are as follows:
Revised: Their governing equations are as follows:

Original: [Equation]
Revised: [Equation]

Original: Gompertz Model
Revised: Gompertz Model

Original: [Equation]
Revised: [Equation]

Original: Generalized Logistic Model where θ 1 , θ 2 > 0 are numeric constants.
Revised: where θ 1 , θ 2 > 0 are numeric constants.

Original: It is worth highlighting that the generalized logistic model involves a constant θ 2 ∈ R + in the power of x ( t ) .
Revised: It is worth highlighting that the generalized logistic model involves a constant θ 2 ∈ R + in the power of x ( t ) .

Original: Therefore, it cannot be symbolically expressed in a sparse linear form (cid:80) Kk
Revised: Therefore, it cannot be symbolically expressed in a sparse linear form (cid:80) Kk

Original: =1 θ k h k
Revised: =1 θ k h k

Original: ( x ( t )) for some candidate functions h k , and the two-step sparse regression method cannot discover the exact functional form of this equation (Section 3.2).
Revised: ( x ( t )) for some candidate functions h k , and the two-step sparse regression method cannot discover the exact functional form of this equation (Section 3.2).

Original: The main simulation results are presented in Figure 2 (the RMSE and distance metrics follow the same pattern and are shown in
Revised: The main simulation results are presented in Figure 2 (the RMSE and distance metrics follow the same pattern and are shown in

Original: Appendix G.
Revised: Appendix G.

Original: The discovered ODEs are shown in Appendix G
Revised: The discovered ODEs are shown in Appendix G

Original: Table 4).
Revised: Table 4).

Original: D-CODE achieves comparable performance as the benchmarks in the low noise, small step size settings ( σ R = 0 .
Revised: D-CODE achieves comparable performance as the benchmarks in the low noise, small step size settings ( σ R = 0 .

Original: 01 ), where the time derivative ˙ x can be accurately estimated from the observed trajectories.
Revised: 01 ), where the time derivative ˙ x can be accurately estimated from the observed trajectories.

Original: However, D-CODE consistently out-performs the benchmarks under the more challenging measurement settings.
Revised: However, D-CODE consistently out-performs the benchmarks under the more challenging measurement settings.

Original: With increased noise or step size, the success probabilities decrease much more rapidly in the benchmarks compared to D-CODE.
Revised: With increased noise or step size, the success probabilities decrease much more rapidly in the benchmarks compared to D-CODE.

Original: Note that although SR-G and
Revised: Note that although SR-G and

Original: D-CODE both use Gaussian process in the pre-processing step, D-CODE achieves much stronger performance.
Revised: D-CODE both use Gaussian process in the pre-processing step, D-CODE achieves much stronger performance.

Original: This suggests that D-CODE’s performance gain is mainly due to its objective function.
Revised: This suggests that D-CODE’s performance gain is mainly due to its objective function.

Original: Figure 3 contextualizes the reason for performance gain with an illustrative example.
Revised: Figure 3 contextualizes the reason for performance gain with an illustrative example.

Original: The panel
Revised: The panel

Original: A shows a typical observed trajectory with moderate measurement noise ( σ R = 0 . 1 ).
Revised: A shows a typical observed trajectory with moderate measurement noise ( σ R = 0 . 1 ).

Original: Here, the
Revised: Here, the

Original: Gaussian process can produce a good estimate (cid:98) x ( t ) by de-noising the measurements.
Revised: Gaussian process can produce a good estimate (cid:98) x ( t ) by de-noising the measurements.

Original: Its posterior interval covers the true trajectory x ( t ) .
Revised: Its posterior interval covers the true trajectory x ( t ) .

Original: Hence, (cid:98) x ( t ) is suitable to guide the search for the ODE, as is done in D-CODE.
Revised: Hence, (cid:98) x ( t ) is suitable to guide the search for the ODE, as is done in D-CODE.

Original: On the other hand, the moderate measurement noise on x ( t ) translates into very high noise on the time derivative ˙ x ( t ) (panel B).
Revised: On the other hand, the moderate measurement noise on x ( t ) translates into very high noise on the time derivative ˙ x ( t ) (panel B).

Original: Even with the state-of-the-art numerical differentiation methods used by SR-T and SR-S, the estimated derivative is still biased on certain  intervals (panel C).
Revised: Even with the state-of-the-art numerical differentiation methods used by SR-T and SR-S, the estimated derivative is still biased on certain [Table] intervals (panel C).

Original: Thus, applying symbolic regression on the biased estimates is unlikely to perform well, as is shown in the benchmark results.
Revised: Thus, applying symbolic regression on the biased estimates is unlikely to perform well, as is shown in the benchmark results.

Original: The same reasoning also applies to the scenario with large sampling step size ∆ t , another challenging situation for estimating the time derivatives.[Table]
Revised: The same reasoning also applies to the scenario with large sampling step size ∆ t , another challenging situation for estimating the time derivatives.

Original: Glycolytic oscillator .
Revised: Glycolytic oscillator .

Original: This biochemical oscillator consists of two ODEs (Sel’Kov, 1968):
Revised: This biochemical oscillator consists of two ODEs (Sel’Kov, 1968):

Original: [Equation]
Revised: [Equation]

Original: where θ 1 , θ 2 , θ 3 > 0 are numeric constants.
Revised: where θ 1 , θ 2 , θ 3 > 0 are numeric constants.

Original: Compared with the growth models above, the glycolytic oscillator involves more variables and has more complex expressions.
Revised: Compared with the growth models above, the glycolytic oscillator involves more variables and has more complex expressions.

Original: The two variables in the equation interact in a highly nonlinear way through the term x 1 ( t ) x 2 ( t ) 2 .
Revised: The two variables in the equation interact in a highly nonlinear way through the term x 1 ( t ) x 2 ( t ) 2 .

Original: The simulation results for the glycolytic oscillator are shown in Table 2.
Revised: The simulation results for the glycolytic oscillator are shown in Table 2.

Original: Here we report the success probability in addition to the RMSE ( × 10 − 2 ) on the constants θ 2 , θ 3 , whose true values are set to 0 .(other
Revised: Here we report the success probability in addition to the RMSE ( × 10 − 2 ) on the constants θ 2 , θ 3 , whose true values are set to 0 .(other

Original: metrics are reported in Appendix G).
Revised: metrics are reported in Appendix G).

Original: D-CODE performs better or equally well compared to the benchmarks in all metrics.
Revised: D-CODE performs better or equally well compared to the benchmarks in all metrics.

Original: Note that D-CODE is the only model that successfully recovered the functional form under σ R = 0 .
Revised: Note that D-CODE is the only model that successfully recovered the functional form under σ R = 0 .

Original: 2 (The benchmarks’ RMSE are NA in this setting because they never recovered the correct functional form).
Revised: 2 (The benchmarks’ RMSE are NA in this setting because they never recovered the correct functional form).

Original: Chaotic Lorenz system .
Revised: Chaotic Lorenz system .

Original: The Lorenz system is a model system for chaotic dynamics, deﬁned as:˙
Revised: The Lorenz system is a model system for chaotic dynamics, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We set the constants as θ 1 = 10 , θ 2 = 28 , θ 3 = 8 / 3 , which leads to chaotic behaviour.
Revised: We set the constants as θ 1 = 10 , θ 2 = 28 , θ 3 = 8 / 3 , which leads to chaotic behaviour.

Original: The simulation results for the chaotic Lorenz system are shown in Figure 4.
Revised: The simulation results for the chaotic Lorenz system are shown in Figure 4.

Original: D-CODE’s robustness to measurement noise is still evident when the underlying system is chaotic and non-periodic.
Revised: D-CODE’s robustness to measurement noise is still evident when the underlying system is chaotic and non-periodic.

Original: The benchmarks achieve high success probability only when the measurement noise is low.
Revised: The benchmarks achieve high success probability only when the measurement noise is low.

Original: The second row of Figure 4 shows a simulated trajectory in the phase space using the true or estimated ODEs (with σ R = 0 . 2 ).
Revised: The second row of Figure 4 shows a simulated trajectory in the phase space using the true or estimated ODEs (with σ R = 0 . 2 ).

Original: D-CODE generates a trajectory that is closest to the true one.
Revised: D-CODE generates a trajectory that is closest to the true one.

Original: SR-T fails because it is unable to recover the true ODE under this noise level.
Revised: SR-T fails because it is unable to recover the true ODE under this noise level.

Original: Although the Neural ODE is trained on a very short prediction horizon ( 0 . 04 ), it still fails to capture the non-periodic behavior of the system.
Revised: Although the Neural ODE is trained on a very short prediction horizon ( 0 . 04 ), it still fails to capture the non-periodic behavior of the system.

Original:  D-CODE IN A CTION
Revised:  D-CODE IN A CTION

Original: We apply D-CODE to model the temporal effect of chemotherapy on the tumor volume.
Revised: We apply D-CODE to model the temporal effect of chemotherapy on the tumor volume.

Original: For most patients, the tumor volume decreases initially in response to the treatment but it eventually grows again due to drug resistance.
Revised: For most patients, the tumor volume decreases initially in response to the treatment but it eventually grows again due to drug resistance.

Original: Although there exist many models for unintervened tumor growth, few models can capture the effect of chemotherapy or the possibility of relapsing—in many cases exponential decay (log cell-kill model) is used for a crude approximation (Geng et al., 2017).
Revised: Although there exist many models for unintervened tumor growth, few models can capture the effect of chemotherapy or the possibility of relapsing—in many cases exponential decay (log cell-kill model) is used for a crude approximation (Geng et al., 2017).

Original: We used the dataset collected by Wilkerson et al.
Revised: We used the dataset collected by Wilkerson et al.

Original: (2017) based on eight clinical trials on cancer patients.
Revised: (2017) based on eight clinical trials on cancer patients.

Original: The patients started chemotherapy at time t = 0 , and the trajectory of tumor volume are measured for up to one year T = 1 .
Revised: The patients started chemotherapy at time t = 0 , and the trajectory of tumor volume are measured for up to one year T = 1 .

Original: In practice, the tumor volumes are inferred from medical scans and are subject to signiﬁcant measurement noise (Mazaheri et al., 2009).
Revised: In practice, the tumor volumes are inferred from medical scans and are subject to signiﬁcant measurement noise (Mazaheri et al., 2009).

Original: After data cleaning, we obtained 310 trajectories, among which 200 is used for training and 110 for evaluation.
Revised: After data cleaning, we obtained 310 trajectories, among which 200 is used for training and 110 for evaluation.

Original: The following two ODEs are discovered by D-CODE and SR-T.
Revised: The following two ODEs are discovered by D-CODE and SR-T.

Original: [Equation]
Revised: [Equation]

Original: The equation discovered by SR-T is much more complex than the one found by D-CODE while still being a poor representation of the data.
Revised: The equation discovered by SR-T is much more complex than the one found by D-CODE while still being a poor representation of the data.

Original: Figure 5 shows an observed trajectory and the trajectories generated by the two ODEs.
Revised: Figure 5 shows an observed trajectory and the trajectories generated by the two ODEs.

Original: D-CODE successfully captures the relapse at t = 0 .
Revised: D-CODE successfully captures the relapse at t = 0 .

Original: 6 and predicts that, under no further intervention, the tumor volume would keep growing after t > 0 .
Revised: 6 and predicts that, under no further intervention, the tumor volume would keep growing after t > 0 .

Original: 6 until patient death.
Revised: 6 until patient death.

Original: In comparison, SR-T’s trajectory is almost constant.
Revised: In comparison, SR-T’s trajectory is almost constant.

Original: We also quantitatively evaluate the prediction errors on the testing set.
Revised: We also quantitatively evaluate the prediction errors on the testing set.

Original: Given the tumor size measured at t = 0 , we solve the IVP to obtain the full trajectory x and compute the RMSE between the estimated and the observed trajectory.
Revised: Given the tumor size measured at t = 0 , we solve the IVP to obtain the full trajectory x and compute the RMSE between the estimated and the observed trajectory.

Original: D-CODE achieves an error of 0 .
Revised: D-CODE achieves an error of 0 .

Original: 220 compared with 0 .
Revised: 220 compared with 0 .

Original: 637 of SR-T.
Revised: 637 of SR-T.

Original:  D ISCUSSION ON FAILURE MODES AND OPEN CHALLENGES
Revised:  D ISCUSSION ON FAILURE MODES AND OPEN CHALLENGES

Original: Scientiﬁc discovery is a challenging process with no success guarantee.
Revised: Scientiﬁc discovery is a challenging process with no success guarantee.

Original: The same holds for ODE discovery with D-CODE.
Revised: The same holds for ODE discovery with D-CODE.

Original: Here we discuss the failure modes and the challenges for future work.
Revised: Here we discuss the failure modes and the challenges for future work.

Original: Unobserved variables .
Revised: Unobserved variables .

Original: In some cases, certain external variables u ( t ) ∈ R L may also inﬂuence the temporal dynamics, i.e. ˙ x j ( t ) =
Revised: In some cases, certain external variables u ( t ) ∈ R L may also inﬂuence the temporal dynamics, i.e. ˙ x j ( t ) =

Original: f (cid:0) x
Revised: f (cid:0) x

Original: ( t ) , u ( t ) (cid:1) .
Revised: ( t ) , u ( t ) (cid:1) .

Original: When u ( t ) is observed or known, D-CODE can include these variables to discover the expanded equation.
Revised: When u ( t ) is observed or known, D-CODE can include these variables to discover the expanded equation.

Original: However, when they are unobserved , the identiﬁcation of system dynamics is challenging or even impossible without strong assumptions (Ljung, 1998).
Revised: However, when they are unobserved , the identiﬁcation of system dynamics is challenging or even impossible without strong assumptions (Ljung, 1998).

Original: In this case, D-CODE can still ﬁnd an ODE based on the observed variables—this
Revised: In this case, D-CODE can still ﬁnd an ODE based on the observed variables—this

Original: ODE may still be a useful approximation but it will leave some dynamics unexplained.
Revised: ODE may still be a useful approximation but it will leave some dynamics unexplained.

Original: Complex equations .
Revised: Complex equations .

Original: The discovery algorithm may fail to discover highly complex ODEs.
Revised: The discovery algorithm may fail to discover highly complex ODEs.

Original: The complexity can arise from two aspects: (1) x ( t ) may include many variables (high dimensionality) or (2) the equation f ∗ may include many mathematical operations (long expression).
Revised: The complexity can arise from two aspects: (1) x ( t ) may include many variables (high dimensionality) or (2) the equation f ∗ may include many mathematical operations (long expression).

Original: However, we highlight that these settings are difﬁcult in general, even for human experts with domain knowledge.
Revised: However, we highlight that these settings are difﬁcult in general, even for human experts with domain knowledge.

Original: Extreme measurement settings .
Revised: Extreme measurement settings .

Original: When the measurement noise or step size is very large, the preprocessing step of D-CODE may infer a trajectory ˆ x that is very different from the true trajectory x , leading to a failure case.
Revised: D-CODE may still fail when the measurement noise or step size is too large.

Original: However, as we show experimentally, D-CODE is more robust to measurement artifacts than the alternative discovery methods.
Revised: However, as we show experimentally, D-CODE is more robust to measurement artifacts than the alternative discovery methods.

Revised: A CKNOWLEDGMENTS

Revised: This work is supported by the US Ofﬁce of Naval Research (ONR), the National Science Foundation (NSF Grant number:1722516), and Roche.

Revised: We thank the anonymous reviewers, Professor William

Revised: Zame and the members of vanderschaar-lab for many insightful comments and suggestions.

Original: Ethics Statement .
Revised: Ethics Statement .

Original: In this work, we use a tumor growth dataset that is anonymized and licensed for research purpose (Section 6).
Revised: In this work, we use a tumor growth dataset that is anonymized and licensed for research purpose (Section 6).

Original: The dataset has also been used in prior research in the literature (Wilkerson et al., 2017).
Revised: The dataset has also been used in prior research in the literature (Wilkerson et al., 2017).

Original: We discuss the limitations and open challenges for D-CODE in Section 7.
Revised: We discuss the limitations and open challenges for D-CODE in Section 7.

Original: We stress that D-CODE is not intended to or capable of fully replacing human experts in modeling dynamical systems.
Revised: We stress that D-CODE is not intended to or capable of fully replacing human experts in modeling dynamical systems.

Original: We envision that D-CODE would facilitate the modeling process by automatically extracting closed-form ODEs from observed trajectories — allowing human experts to easily inspect, analyze and reﬁne the distilled ODEs.
Revised: We envision that D-CODE would facilitate the modeling process by automatically extracting closed-form ODEs from observed trajectories — allowing human experts to easily inspect, analyze and reﬁne the distilled ODEs.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: The proof of theoretical results is shown in Appendix A.
Revised: The proof of theoretical results is shown in Appendix A.

Original: The technical assumptions for the proof are listed in Proposition 1 and Theorem 1 in the main text and further discussed in Appendix A.
Revised: The technical assumptions for the proof are listed in Proposition 1 and Theorem 1 in the main text and further discussed in Appendix A.

Original: The experiment settings are discussed in Section 5 and 6 and further detailed in Appendix F. All experiment code will be released upon acceptance.
Revised: The experiment settings are discussed in Section 5 and 6 and further detailed in Appendix F. The experiment code is available at https://github.com/ZhaozhiQIAN/ D-CODE-ICLR-2022 .

Original: T ABLE OF S UPPLEMENTARY M ATERIALS
Revised: T ABLE OF S UPPLEMENTARY M ATERIALS

Original: Appendix A: proof and discussion on the theoretical results.
Revised: Appendix A: proof and discussion on the theoretical results.

Original: Appendix B: implementation details of D-CODE.
Revised: Appendix B: implementation details of D-CODE.

Original: Appendix C: discussion on dataset curation.
Revised: Appendix C: discussion on dataset curation.

Original: Appendix D: utilities of the discovered closed-form ODE. 5.
Revised: Appendix D: utilities of the discovered closed-form ODE. 5.

Original: Appendix E: background on numerical differentiation.
Revised: Appendix E: background on numerical differentiation.

Original: Appendix F: implementation details of the experiments.
Revised: Appendix F: implementation details of the experiments.

Original: Appendix G: additional simulation results and sensitivity analysis.
Revised: Appendix G: additional simulation results and sensitivity analysis.

Original: Appendix H: discussion on alternative methods.
Revised: Appendix H: discussion on alternative methods.

Original: All code will be released upon acceptance.

Original: A T HEORETICAL RESULTS : PROOF AND DISCUSSION A. V ARIATIONAL FORMULATION OF ODE
Revised: A T HEORETICAL RESULTS : PROOF AND DISCUSSION A. V ARIATIONAL FORMULATION OF ODE

Original: Before we prove Proposition 1, we need the following lemma which is a particular formulation of the
Revised: Before we prove Proposition 1, we need the following lemma which is a particular formulation of the

Original: Fundamental lemma of calculus of variations Elsgolc (1961).
Revised: Fundamental lemma of calculus of variations Elsgolc (1961).

Original: Lemma (Fundamental lemma of calculus of variations) .
Revised: Lemma (Fundamental lemma of calculus of variations) .

Original: Let h be a continuous function on a closed interval [0 , T ] .
Revised: Let h be a continuous function on a closed interval [0 , T ] .

Original: h is equal to 0 everywhere if and only if (cid:82) T 0 h ( t ) g ( t ) dt = 0 for all g ∈ C 1 [0 , T ] such that g (0) =
Revised: h is equal to 0 everywhere if and only if (cid:82) T 0 h ( t ) g ( t ) dt = 0 for all g ∈ C 1 [0 , T ] such that g (0) =

Original: g ( T ) = 0 .
Revised: g ( T ) = 0 .

Original: Proof.
Revised: Proof.

Original: The forward direction is trivial, so let us focus on the converse.
Revised: The forward direction is trivial, so let us focus on the converse.

Original: Assume for contradiction that there is a point t 0 ∈ [0 , T ] such that h ( t 0 ) (cid:54) = 0 .
Revised: Assume for contradiction that there is a point t 0 ∈ [0 , T ] such that h ( t 0 ) (cid:54) = 0 .

Original: Without loss of generality, assume that h ( t 0 ) > 0 .
Revised: Without loss of generality, assume that h ( t 0 ) > 0 .

Original: From the continuity of h , we know that there is actually a point t 1 ∈ (0 , T ) , such that h ( t 1 ) > 0 .
Revised: From the continuity of h , we know that there is actually a point t 1 ∈ (0 , T ) , such that h ( t 1 ) > 0 .

Original: Continuity of h implies that there exists δ > 0 and a small neighbourhood ( t 1 − δ, t 1 + δ ) ⊂ (0 , T ) such that h ( t ) > 0 ∀ t ∈
Revised: Continuity of h implies that there exists δ > 0 and a small neighbourhood ( t 1 − δ, t 1 + δ ) ⊂ (0 , T ) such that h ( t ) > 0 ∀ t ∈

Original: ( t 1 − δ, t 1 + δ ) .
Revised: ( t 1 − δ, t 1 + δ ) .

Original: We can now deﬁne a non-negative continuous function g : [0 , T ] → [0 , + ∞ ) such that g ( t ) =outside the neighbourhood ( t 1 − δ, t 1 + δ ) and g ( t ) > 0 for some interval inside ( t 1 − δ, t 1 + δ ) .
Revised: We can now deﬁne a non-negative continuous function g : [0 , T ] → [0 , + ∞ ) such that g ( t ) =outside the neighbourhood ( t 1 − δ, t 1 + δ ) and g ( t ) > 0 for some interval inside ( t 1 − δ, t 1 + δ ) .

Original: See
Revised: See

Original: Figure 6 for an illustrative example. 
Revised: Figure 6 for an illustrative example. Figure

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: That contradicts (cid:82) T 0 h ( t ) g ( t ) dt = 0 for all g ∈ C 1 [0 , T ] such that g (0) =
Revised: That contradicts (cid:82) T 0 h ( t ) g ( t ) dt = 0 for all g ∈ C 1 [0 , T ] such that g (0) =

Original: g ( T ) = 0 .
Revised: g ( T ) = 0 .

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: Now using Fundamental lemma of calculus of variations, we get that the following two statements are equivalent
Revised: Now using Fundamental lemma of calculus of variations, we get that the following two statements are equivalent

Original: [Equation]
Revised: [Equation]

Original: By linearity and integration by parts
Revised: By linearity and integration by parts

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where in the last equality we used the fact that g (0) =
Revised: where in the last equality we used the fact that g (0) =

Original: g ( T ) = 0 .
Revised: g ( T ) = 0 .

Original: That means that
Revised: That means that

Original: [Equation]
Revised: [Equation]

Original: This proves that
Revised: This proves that

Original: [Equation]
Revised: [Equation]

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: Discussion .
Revised: Discussion .

Original: As we show in the proof above, the key idea of the variational formulation is to bypass the derivative of the trajectory ˙
Revised: As we show in the proof above [Equation] the key idea of the variational formulation is to bypass the derivative of the trajectory ˙

Original: x j ( t ) with the derivative of a testing function ˙ g ( t ) — the swapping of derivatives is achieved by integration by parts (Equations 13).
Revised: x j ( t ) with the derivative of a testing function ˙ g ( t ) — the swapping of derivatives is achieved by integration by parts (Equations 13).

Original: An important consequence is that we no longer need to know or estimate ˙ x j ( t ) (which is unobserved and challenging to estimate), but only need to know ˙ g ( t ) for a given testing function g ( t ) .
Revised: An important consequence is that we no longer need to know or estimate ˙ x j ( t ) (which is unobserved and challenging to estimate), but only need to know ˙ g ( t ) for a given testing function g ( t ) .

Original: Indeed, if g ( t ) has an analytical derivative, we immediately have access to ˙ g ( t ) .
Revised: Indeed, if g ( t ) has an analytical derivative, we immediately have access to ˙ g ( t ) .

Original: Note that the variational formulation builds on the exact condition C j = 0 .
Revised: Note that the variational formulation builds on the exact condition C j = 0 .

Original: It does not consider the situation where C j (cid:54) = 0 , no matter how close C j is to 0 .
Revised: It does not consider the situation where C j (cid:54) = 0 , no matter how close C j is to 0 .

Original: Hence, the formulation itself is inadequate to guide the search of closed-form ODEs.
Revised: Hence, the formulation itself is inadequate to guide the search of closed-form ODEs.

Original: For instance, suppose the C j for two candidate functions ˆ f 1 and ˆ f 2 are 0 .
Revised: For instance, suppose the C j for two candidate functions ˆ f 1 and ˆ f 2 are 0 .

Original: 01 and 1000 respectively.
Revised: 01 and 1000 respectively.

Original: The variational formulation does not imply ˆ f 1 ﬁts the trajectory x better than ˆ f 2 ; rather, it implies that ˆ f 1 and ˆ f 2 are equally bad as C j (cid:54) = 0 for both.
Revised: The variational formulation does not imply ˆ f 1 ﬁts the trajectory x better than ˆ f 2 ; rather, it implies that ˆ f 1 and ˆ f 2 are equally bad as C j (cid:54) = 0 for both.

Original: In order to bridge this gap, we need to establish a notion of distance between functions and link the functional C j to this distance — they are addressed in Appendix A.2 and A.3 respectively.
Revised: In order to bridge this gap, we need to establish a notion of distance between functions and link the functional C j to this distance — they are addressed in Appendix A.2 and A.3 respectively.

Original: The formulation above is tailored for ODEs, which is the focus of the current work.
Revised: The formulation above is tailored for ODEs, which is the focus of the current work.

Original: However, the variational formulation can be generalized to other types of differential equations, e.g partial differential equations (PDE) and delay differential equations (DDE).
Revised: However, the variational formulation can be generalized to other types of differential equations, e.g partial differential equations (PDE) and delay differential equations (DDE).

Original: In fact, the variational formulation is the theoretical foundation of the Finite Element Method (FEM) — the industrial standard for solving
Revised: In fact, the variational formulation is the theoretical foundation of the Finite Element Method (FEM) — the industrial standard for solving

Original: PDEs in complex engineering systems and ﬂuid dynamics (Zienkiewicz et al., 1977; Gokhale, 2008).
Revised: PDEs in complex engineering systems and ﬂuid dynamics (Zienkiewicz et al., 1977; Gokhale, 2008).

Original: It has also been used to estimate the parameters of a known ODE or DDE (Brunel et al., 2014).
Revised: It has also been used to estimate the parameters of a known ODE or DDE (Brunel et al., 2014).

Original: We envision that future works may be built on the general variational formulation to discover closed-form
Revised: We envision that future works may be built on the general variational formulation to discover closed-form

Original: PDE or DDE from data.
Revised: PDE or DDE from data.

Original: A. D ISTANCE FUNCTION
Revised: A. D ISTANCE FUNCTION

Original: In this section, we justify that d x ( f, f ∗ ) is an appropriate metric for our problem.
Revised: In this section, we justify that d x ( f, f ∗ ) is an appropriate metric for our problem.

Original: There are two main ways of comparing functions.
Revised: There are two main ways of comparing functions.

Original: We can either compare their symbolic representations or measure the differences in values they take for a set of arguments.
Revised: We can either compare their symbolic representations or measure the differences in values they take for a set of arguments.

Original: As we want this metric to be a limit of some objective computable from the data, we focus on the latter.
Revised: As we want this metric to be a limit of some objective computable from the data, we focus on the latter.

Original: Commonly used distance functions are metrics induced by L p norms.
Revised: Commonly used distance functions are metrics induced by L p norms.

Original: L p norm is deﬁned in the following way:
Revised: L p norm is deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Original: In practice, L 1 or L 2 norm is used.
Revised: In practice, L 1 or L 2 norm is used.

Original: The induced metric is then deﬁned as the norm of the difference between the functions, namely d ( f, g ) =
Revised: The induced metric is then deﬁned as the norm of the difference between the functions, namely d ( f, g ) =

Original: ||
Revised: ||

Original: f − g || p .
Revised: f − g || p .

Original: Now the question remains which difference we should take the norm of.
Revised: Now the question remains which difference we should take the norm of.

Original: We have three reasonable choices:
Revised: We have three reasonable choices:

Original: f − f ∗ 2.
Revised: f − f ∗ 2.

Original: ˜ x − x , where ˜ x is trajectory prescribed by f computed from an estimated initial condition 3.
Revised: ˜ x − x , where ˜ x is trajectory prescribed by f computed from an estimated initial condition 3.

Original: f − f ∗ composed with some other function
Revised: f − f ∗ composed with some other function

Original: The ﬁrst choice poses the challenge of choosing the range [ a, b ] over which we integrate f − f ∗ .
Revised: The ﬁrst choice poses the challenge of choosing the range [ a, b ] over which we integrate f − f ∗ .

Original: In general we do not make any assumptions on the domain of the functions.
Revised: In general we do not make any assumptions on the domain of the functions.

Original: It also seems unreasonable to compare the functions far from trajectory, where we have no information about their behaviour.
Revised: It also seems unreasonable to compare the functions far from trajectory, where we have no information about their behaviour.

Original: The second choice is similar to what is done in the Neural ODE approach.
Revised: The second choice is similar to what is done in the Neural ODE approach.

Original: As we mentioned before, this involves solving the Initial Value Problem (IVP).
Revised: As we mentioned before, this involves solving the Initial Value Problem (IVP).

Original: Even if the IVP has a unique solution, it might be very sensitive to the initial condition, making it unﬁt for chaotic systems.
Revised: Even if the IVP has a unique solution, it might be very sensitive to the initial condition, making it unﬁt for chaotic systems.

Original: We verify this claim in the experiments (Figure 4)
Revised: We verify this claim in the experiments (Figure 4)

Original: The third proposition can solve the main issue of the ﬁrst option.
Revised: The third proposition can solve the main issue of the ﬁrst option.

Original: If we compose f − f ∗ with the ground truth trajectory, x , not only we have a well-deﬁned interval over which to integrate ([0 , T ]) but also we focus our attention on values of f ∗ where we have a chance to recover it, and where it matters for us the most.
Revised: If we compose f − f ∗ with the ground truth trajectory, x , not only we have a well-deﬁned interval over which to integrate ([0 , T ]) but also we focus our attention on values of f ∗ where we have a chance to recover it, and where it matters for us the most.

Original: That is why we decided to chose d x ( f, f ∗ )
Revised: That is why we decided to chose d x ( f, f ∗ )

Original: =
Revised: =

Original: ||
Revised: ||

Original: ( f − f ∗ ) ◦ x
Revised: ( f − f ∗ ) ◦ x

Original: || 2 as our distance function.
Revised: || 2 as our distance function.

Original: Moreover, as it is based on a familiar metric, it inherits many important properties, such as
Revised: Moreover, as it is based on a familiar metric, it inherits many important properties, such as

Original: d x is a pseudo-metric 2 , in particular, f = f ∗ =
Revised: d x is a pseudo-metric 3 , in particular, f = f ∗ =

Original: ⇒ d x ( f, f ∗ ) =2.
Revised: ⇒ d x ( f, f ∗ ) =2.

Original: d x ( f, f ∗ ) = 

Revised: d x ( f, f ∗ ) =

Original: =
Revised: =

Original: ⇒ f
Revised: ⇒ f

Original: = f ∗ on x ([0 , T ]) , the image of x 3.
Revised: = f ∗ on x ([0 , T ]) , the image of x 3.

Original: For a sequence f l such that d x ( f l , f ∗ ) → 0 it holds that f l ◦ x → f ∗ ◦ x in the mean
Revised: For a sequence f l such that d x ( f l , f ∗ ) → 0 it holds that f l ◦ x → f ∗ ◦ x in the mean

Original: The third property tells us that if we keep minimising d x ( f, f ∗ ) , we can expect f getting closer to f ∗ on x ([0 , T ]) , the image of x .
Revised: The third property tells us that if we keep minimising d x ( f, f ∗ ) , we can expect f getting closer to f ∗ on x ([0 , T ]) , the image of x .

Original: A. P ROOF OF THE THEOREM
Revised: A. P ROOF OF THE THEOREM

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Let us consider a Hilbert space L 2 [0 , T ] where the inner product is denoted by (cid:104)· , ·(cid:105) and deﬁned as: 
Revised: Let us consider a Hilbert space L 2 [0 , T ] where the inner product is denoted by (cid:104)· , ·(cid:105) and deﬁned as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where a and b are real functions which are square-integrable on the interval [0 , T ] .
Revised: where a and b are real functions which are square-integrable on the interval [0 , T ] .

Original: The L 2 norm can be written as:
Revised: The L 2 norm can be written as:

Original: [Equation]
Revised: [Equation]

Original: Using this notation we can rewrite C j ( f, (cid:98) x k , g s ) as:
Revised: Using this notation we can rewrite C j ( f, (cid:98) x k , g s ) as:

Original: [Equation]
Revised: [Equation]

Original: We assume that (cid:98) x k converges to x in mean.
Revised: We assume that (cid:98) x k converges to x in mean.

Original: For vector-valued functions that means that
Revised: For vector-valued functions that means that

Original: [Equation]
Revised: [Equation]

Original: Note that || · || 2 in this equation denotes the usual Euclidean norm as the arguments are vectors not functions.
Revised: Note that || · || 2 in this equation denotes the usual Euclidean norm as the arguments are vectors not functions.

Original: This is equivalent to saying that all components of (cid:98) x converge to the corresponding components of x .
Revised: This is equivalent to saying that all components of (cid:98) x converge to the corresponding components of x .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: As the inner product with one argument ﬁxed can be seen as a continuous operator
Revised: As the inner product with one argument ﬁxed can be seen as a continuous operator

Original: [Equation]
Revised: [Equation]

Original: we get:
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: We assume that f j is λ -Lipschitz continuous.
Revised: We assume that f j is λ -Lipschitz continuous.

Original: That means that
Revised: That means that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This means that
Revised: This means that

Original: [Equation]
Revised: [Equation]

Original: Combining equations (23) and (25) with expression (20)
Revised: Combining equations (23) and (25) with expression (20)

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: using integration by parts we get
Revised: using integration by parts we get

Original: [Equation]
Revised: [Equation]

Original: We use the assumption that g s (0) =
Revised: We use the assumption that g s (0) =

Original: g s ( T ) =
Revised: g s ( T ) =

Original: ∀ s to obtain
Revised: ∀ s to obtain

Original: [Equation]
Revised: [Equation]

Original: We can substitute it back into Equation 26 and use the linearity of the inner product to get
Revised: We can substitute it back into Equation 26 and use the linearity of the inner product to get

Original: [Equation]
Revised: [Equation]

Original: Now we can use Parseval’s identity and the fact that the set g 
Revised: Now we can use Parseval’s identity and the fact that the set g 1 , g 2 , ... forms a Hilbert basis for L 2 [

Original: [Equation]
Revised: [Equation]

Original: g 2 , ... forms a Hilbert basis for L 2 [0 , T ] to infer that:
Revised: T ] to infer that:

Original: [Equation]
Revised: [Equation]

Original: Substituting Equation 29 into Equation 30 and using the fact that the inner limit does not depend on
Revised: Substituting Equation 29 into Equation 30 and using the fact that the inner limit does not depend on

Original: S we obtain the statement in the theorem
Revised: S we obtain the statement in the theorem

Original: [Equation]
Revised: [Equation]

Original: A. D IVERGENT DERIVATIVES
Revised: A. D IVERGENT DERIVATIVES

Original: The main advantage of our algorithm lies in estimating the trajectory instead of estimating its time derivative.
Revised: The main advantage of our algorithm lies in estimating the trajectory instead of estimating its time derivative.

Original: We show how the derivatives of a uniformly convergent sequence of functions may fail to converge, and we illustrate how it impacts objectives based on estimating the derivative.
Revised: We show how the derivatives of a uniformly convergent sequence of functions may fail to converge, and we illustrate how it impacts objectives based on estimating the derivative.

Original: We highlight the following mathematical fact:
Revised: We highlight the following mathematical fact:

Original: Remark.
Revised: Remark.

Original: Uniform convergence of functions does not guarantee convergence of the derivatives
Revised: Uniform convergence of functions does not guarantee convergence of the derivatives

Original: This is in contrast to deﬁnite integrals where such implication holds.
Revised: This is in contrast to deﬁnite integrals where such implication holds.

Original: Intuitively, this follows from the fact that the derivative is a very local property of the function.
Revised: Intuitively, this follows from the fact that the derivative is a very local property of the function.

Original: To see an example of such a sequence, consider a sequence of functions ( h k ) where h k : [0 , 1] → R is given by
Revised: To see an example of such a sequence, consider a sequence of functions ( h k ) where h k : [0 , 1] → R is given by

Original: [Equation]
Revised: [Equation]

Original: It can be shown that ( h k ) converges to h ( t ) = 0 uniformly.
Revised: It can be shown that ( h k ) converges to h ( t ) = 0 uniformly.

Original: However, the derivative of h k , given by
Revised: However, the derivative of h k , given by

Original: [Equation]
Revised: [Equation]

Original: fails to converge at any point.
Revised: fails to converge at any point.

Original: We use this observation to show that the square of || ˆ˙ x j − f j ◦ ˆ x || 2 , where ˆ x j is the derivative calculated from the recovered trajectory ˆ x , does not converge to d x ( f j , f ∗ j ) 2 when ˆ x converges to the ground truth trajectory x .
Revised: We use this observation to show that the square of || ˆ˙ x j − f j ◦ ˆ x || 2 , where ˆ x j is the derivative calculated from the recovered trajectory ˆ x , does not converge to d x ( f j , f ∗ j ) 2 when ˆ x converges to the ground truth trajectory x .

Original: Consider the following ODE ( J = 1 , T = 1 ):
Revised: Consider the following ODE ( J = 1 , T = 1 ):

Original: [Equation]
Revised: [Equation]

Original: ) so f ∗ ( x ) = x .
Revised: so f ∗ ( x ) = x .

Original: We choose x ( t ) =
Revised: We choose x ( t ) =

Original: e t as the ground truth trajectory from which we sample.
Revised: e t as the ground truth trajectory from which we sample.

Original: Let (ˆ x k ) be a sequence of trajectories given by ˆ x k = e t + sin ( kt ) k .
Revised: Let (ˆ x k ) be a sequence of trajectories given by ˆ x k = e t + sin ( kt ) k .

Original: It can be shown that (ˆ x k ) converges to x uniformly.
Revised: It can be shown that (ˆ x k ) converges to x uniformly.

Original: Then for any candidate function f
Revised: Then for any candidate function f

Original: [Equation]
Revised: [Equation]

Original: If || ˆ˙ x k − f ◦ ˆ x k || 22 were to converge to d x ( f, f ∗ ) 2 then in particular ||
Revised: If || ˆ˙ x k − f ◦ ˆ x k || 22 were to converge to d x ( f, f ∗ ) 2 then in particular ||

Original: ˆ˙ x k − f ∗ ◦ ˆ x k || 22 should converge to 0 .
Revised: ˆ˙ x k − f ∗ ◦ ˆ x k || 22 should converge to 0 .

Original: However, we show this is not what happens.
Revised: However, we show this is not what happens.

Original: [Equation]
Revised: [Equation]

Original: As (cid:16) sin ( kt ) k (cid:17) 2 and 2 cos ( kt ) (cid:16) sin ( kt ) k (cid:17) converge uniformly to a 0 function, we can use the linearity of integration to obtain
Revised: As (cid:16) sin ( kt ) k (cid:17) 2 and 2 cos ( kt ) (cid:16) sin ( kt ) k (cid:17) converge uniformly to a 0 function, we can use the linearity of integration to obtain

Original: [Equation]
Revised: [Equation]

Original: This example shows that objectives that estimate the derivative are highly dependent on the type of estimation algorithm used.
Revised: This example shows that objectives that estimate the derivative are highly dependent on the type of estimation algorithm used.

Original: In contrast, Theorem 1 tells us that the only property of the estimation algorithm we need is that the trajectories converge to the ground truth in L 2 norm.Input
Revised: In contrast, Theorem 1 tells us that the only property of the estimation algorithm we need is that the trajectories converge to the ground truth in L 2 norm.

Original: Dataset t ) i ≤ N t ∈ T } Input Smoothing
Revised: Algorithm 1 D-CODE.

Original: O } for j ∈ [1 , J ] do (cid:98) f j = f j{ Initialize the j th ODE }
Revised: for j ∈ [1 , J ] do (cid:98) f j = f j{ Initialize the j th ODE }

Original: Converge =
Revised: Converge =

Original: False while Not Converge do obj =
Revised: False while Not Converge do obj =

Original: (cid:80) Ni =1 (cid:80)
Revised: (cid:80) Ni =1 (cid:80)

Original: Ss =1 C j ( f j , (cid:98)
Revised: Ss =1 C j ( f j , (cid:98)

Original: x
Revised: x

Original: i , g s){ Calculate } (cid:98) f j , Converge = O ( obj , (cid:98) f j )
Revised: i , g s )

Original: } end while end for
Revised: f j end while end for

Original: Output

Original: The discovered f j j

Original: J B D-CODE FRAMEWORK B. P SEUDOCODE
Revised: B D-CODE FRAMEWORK B. P SEUDOCODE

Original: The Pseudocode is presented in Algorithm 1.
Revised: The Pseudocode is presented in Algorithm 1.

Original: B. D ISCOVER HIGHER ORDER AND NONAUTONOMOUS ODE S
Revised: B. D ISCOVER HIGHER ORDER AND NONAUTONOMOUS ODE S

Original: In the main text, we focused on the system of ﬁrst order autonomous ODEs (Equation 1).
Revised: In the main text, we focused on the system of ﬁrst order autonomous ODEs (Equation 1).

Original: This is because higher order or nonautonomous ODEs can be transformed into ﬁrst order autonomous systems by including more variables.
Revised: This is because higher order or nonautonomous ODEs can be transformed into ﬁrst order autonomous systems by including more variables.

Original: For instance, consider the damped, driven harmonic oscillator, a model system for classical mechanics (Simmons, 1972):
Revised: For instance, consider the damped, driven harmonic oscillator, a model system for classical mechanics (Simmons, 1972):

Original: [Equation]
Revised: [Equation]

Original: The ODE is second-order as it involves the acceleration term ¨ x 

Revised: The ODE is second-order as it involves the acceleration term ¨ x

Original: and it is nonautonomous because of the time-dependent term cos( t ) .
Revised: and it is nonautonomous because of the time-dependent term cos( t ) .

Original: If we introduce two new variables: x 2 = ˙ x 1 and x 3 = t , we obtain the following ﬁrst order autonomous system:
Revised: If we introduce two new variables: x 2 = ˙ x 1 and x 3 = t , we obtain the following ﬁrst order autonomous system:

Original: [Equation]
Revised: [Equation]

Original: Since the ﬁrst and the third equations are implied by the deﬁnition of x 2 and x 3 , they do not need to be discovered from data.
Revised: Since the ﬁrst and the third equations are implied by the deﬁnition of x 2 and x 3 , they do not need to be discovered from data.

Original: One can apply the discovery algorithm to uncover the second equation from the measurements of x 1 , x 2 and x 3 .
Revised: One can apply the discovery algorithm to uncover the second equation from the measurements of x 1 , x 2 and x 3 .

Original: C D ISCUSSION ON DATASET CURATION
Revised: C D ISCUSSION ON DATASET CURATION

Original: The success of ODE discovery depends on the availability of a well-curated dataset.
Revised: The success of ODE discovery depends on the availability of a well-curated dataset.

Original: However, curating a dataset is a highly nontrivial task, and it is well beyond the scope of the current work.
Revised: However, curating a dataset is a highly nontrivial task, and it is well beyond the scope of the current work.

Original: In fact, the curation of dataset is sometimes an important scientiﬁc contribution in itself (Blasius et al., 2020).
Revised: In fact, the curation of dataset is sometimes an important scientiﬁc contribution in itself (Blasius et al., 2020).

Original: Here, we provide a brief discussion on this topic.
Revised: Here, we provide a brief discussion on this topic.

Original: Selection of variables .
Revised: Selection of variables .

Original: The researcher needs to decide which variables to include in the dataset — this deﬁnes the scope of problem.
Revised: The researcher needs to decide which variables to include in the dataset — this deﬁnes the scope of problem.

Original: In general, variable selection depends on the scientiﬁc problem being addressed and the background knowledge about the interactions between variables.
Revised: In general, variable selection depends on the scientiﬁc problem being addressed and the background knowledge about the interactions between variables.

Original: For instance, one may use the known chemical reaction network to choose a small set of compounds to include in the data.
Revised: For instance, one may use the known chemical reaction network to choose a small set of compounds to include in the data.

Original: When such knowledge is unavailable, the researcher may need to perform pilot studies to ﬁnd out the interaction network (otherwise they may face a challenging high-dimensional problem).
Revised: When such knowledge is unavailable, the researcher may need to perform pilot studies to ﬁnd out the interaction network (otherwise they may face a challenging high-dimensional problem).

Original: We note that there is a well-developed literature on uncovering the interaction network of a high-dimensional dynamical systems (Mangan et al., 2016; Qian et al., 2020).
Revised: We note that there is a well-developed literature on uncovering the interaction network of a high-dimensional dynamical systems (Mangan et al., 2016; Qian et al., 2020).

Original: Measurement settings .
Revised: Measurement settings .

Original: In practice, the measurement settings are often affected by practical constraints, e.g. the precision of measurement device, the measurement cost, the time limit, etc.
Revised: In practice, the measurement settings are often affected by practical constraints, e.g. the precision of measurement device, the measurement cost, the time limit, etc.

Original: However, the researcher should ensure that the measurement horizon [0 , T ] covers the pattern of scientiﬁc interest.
Revised: However, the researcher should ensure that the measurement horizon [0 , T ] covers the pattern of scientiﬁc interest.

Original: To illustrate this point, consider an application in climate modeling.
Revised: To illustrate this point, consider an application in climate modeling.

Original: If we are interested in studying the cyclic behavior of the system (e.g. seasonality), the measurement should cover at least one entire period (one year).
Revised: If we are interested in studying the cyclic behavior of the system (e.g. seasonality), the measurement should cover at least one entire period (one year).

Original: On the other hand, if we are interested in the system’s response to an external stimuli (e.g. short-term weather change due to volcanic activity), the measurement should cover the period before and after the stimuli (volcano eruption).
Revised: On the other hand, if we are interested in the system’s response to an external stimuli (e.g. short-term weather change due to volcanic activity), the measurement should cover the period before and after the stimuli (volcano eruption).

Original: Iterative nature of discovery .
Revised: Iterative nature of discovery .

Original: Scientiﬁc discovery is by no means a single-step monolithic process.
Revised: Scientiﬁc discovery is by no means a single-step monolithic process.

Original: Rather, it is an iterative process involving many components: hypothesis generation, data curation, modeling, validation, etc.
Revised: Rather, it is an iterative process involving many components: hypothesis generation, data curation, modeling, validation, etc.

Original: D-CODE focuses on the modeling component, but it can facilitate the entire loop of discovery because it distills an interpretable close-form ODE.
Revised: D-CODE focuses on the modeling component, but it can facilitate the entire loop of discovery because it distills an interpretable close-form ODE.

Original: For instance, if the discovered ODE converges to a wrong stationary point (as indicated by domain knowledge), the researcher may curate an additional dataset with longer time horizon and re-train D-CODE with the additional information.
Revised: For instance, if the discovered ODE converges to a wrong stationary point (as indicated by domain knowledge), the researcher may curate an additional dataset with longer time horizon and re-train D-CODE with the additional information.

Original: D U TILITIES OF THE DISCOVERED CLOSED - FORM ODE
Revised: D U TILITIES OF THE DISCOVERED CLOSED - FORM ODE

Original: In the main text, we focus on the problem of distilling closed-form ODE from data.
Revised: In the main text, we focus on the problem of distilling closed-form ODE from data.

Original: Here we illustrate how the learned closed-form ODE can facilitate various downstream tasks and applications.
Revised: Here we illustrate how the learned closed-form ODE can facilitate various downstream tasks and applications.

Original: Prediction .
Revised: Prediction .

Original: The learned closed-form ODE can be used to make predictions about the future states of a dynamical system.
Revised: The learned closed-form ODE can be used to make predictions about the future states of a dynamical system.

Original: Formally speaking, given a sequence of past observations
Revised: Formally speaking, given a sequence of past observations

Original: y ( t 1 ) , y ( t 2 ) , . . .
Revised: y ( t 1 ) , y ( t 2 ) , . . .

Original: , y ( t k ) , we would like to predict x ( t ∗ ) for a future time t ∗ > t k .
Revised: , y ( t k ) , we would like to predict x ( t ∗ ) for a future time t ∗ > t k .

Original: This prediction problem is very well studied and has many proven algorithms (e.g. the family of Kalman ﬁlters (Welch et al., 1995) and the
Revised: This prediction problem is very well studied and has many proven algorithms (e.g. the family of Kalman ﬁlters (Welch et al., 1995) and the

Original: Bayesian methods (Doucet et al., 2000)).
Revised: Bayesian methods (Doucet et al., 2000)).

Original: Simulation and synthetic data .
Revised: Simulation and synthetic data .

Original: The learned closed-form ODE can also be used as a simulation model to generate synthetic datasets.
Revised: The learned closed-form ODE can also be used as a simulation model to generate synthetic datasets.

Original: The user can specify different initial conditions x (0) and different external signals (if applicable) to generate a range of trajectories.
Revised: The user can specify different initial conditions x (0) and different external signals (if applicable) to generate a range of trajectories.

Original: The simulated trajectories may be used in downstream tasks, e.g. computing the statistical properties.
Revised: The simulated trajectories may be used in downstream tasks, e.g. computing the statistical properties.

Original: Planning and control .
Revised: Planning and control .

Original: In some cases, the system contains variables that can be intervened on (e.g. external forces), and we would like to manipulate these variables to maximize certain utility function.
Revised: In some cases, the system contains variables that can be intervened on (e.g. external forces), and we would like to manipulate these variables to maximize certain utility function.

Original: This is the canonical problem setting for Control Theory, where decades of research have been devoted to deriving the optimal policy from the closed-form governing equations and the utility function (Mehrmann, 1991; Haddad & Chellaboina, 2011).
Revised: This is the canonical problem setting for Control Theory, where decades of research have been devoted to deriving the optimal policy from the closed-form governing equations and the utility function (Mehrmann, 1991; Haddad & Chellaboina, 2011).

Original: Understanding the properties .
Revised: Understanding the properties .

Original: Often we are interested in understanding the high-level properties of the dynamical system.
Revised: Often we are interested in understanding the high-level properties of the dynamical system.

Original: Examples include asymptotic stability, ﬁxed points, periodicity, bifurcation, and so on.
Revised: Examples include asymptotic stability, ﬁxed points, periodicity, bifurcation, and so on.

Original: The closed-form ODE would facilitate the study of these properties because the analytical and the computational tools developed in the dynamical system literature are both applicable (Browder, 1983;
Revised: The closed-form ODE would facilitate the study of these properties because the analytical and the computational tools developed in the dynamical system literature are both applicable (Browder, 1983;

Original: Lyapunov, 1992; Chow & Hale, 2012).
Revised: Lyapunov, 1992; Chow & Hale, 2012).

Original: E B ACKGROUND ON NUMERICAL DIFFERENTIATION
Revised: E B ACKGROUND ON NUMERICAL DIFFERENTIATION

Original: The task of estimating time derivative ˙ x from discretely sampled and noisy data is commonly referred to as numerical differentiation .
Revised: The task of estimating time derivative ˙ x from discretely sampled and noisy data is commonly referred to as numerical differentiation .

Original: Finite difference is a family of the most basic numerical differentiation methods (Bickley, 1941).
Revised: Finite difference is a family of the most basic numerical differentiation methods (Bickley, 1941).

Original: For example, the central difference estimator calculates
Revised: For example, the central difference estimator calculates

Original: [Equation]
Revised: [Equation]

Original: The main disadvantage of ﬁnite difference is that it is highly sensitive to measurement noise (Chartrand, 2011).
Revised: The main disadvantage of ﬁnite difference is that it is highly sensitive to measurement noise (Chartrand, 2011).

Original: A common way to address this shortcoming is to estimate the derivative on the smoothed trajectory.
Revised: A common way to address this shortcoming is to estimate the derivative on the smoothed trajectory.

Original: For example, spline-smoothed differentiation smooths the observed trajectories with cubic spline and estimates ˙ x from the ﬁtted spline functions (Shen et al., 1998; Zhou & Wolfe, 2000).
Revised: For example, spline-smoothed differentiation smooths the observed trajectories with cubic spline and estimates ˙ x from the ﬁtted spline functions (Shen et al., 1998; Zhou & Wolfe, 2000).

Original: (This method is used in the benchmarks SR-S).
Revised: (This method is used in the benchmarks SR-S).

Original: However, this family of methods is usually based on assumptions on
Revised: However, this family of methods is usually based on assumptions on

Original: [Table]
Revised: [Table]

Original: the smoothness of the trajectory and its (higher-order) derivatives (Zhou & Wolfe, 2000): they encode a bias towards smooth and slow-changing derivatives.
Revised: the smoothness of the trajectory and its (higher-order) derivatives (Zhou & Wolfe, 2000): they encode a bias towards smooth and slow-changing derivatives.

Original: Such bias may the hamper the estimation of dynamical systems with abrupt changes (Chartrand, 2011)
Revised: Such bias may the hamper the estimation of dynamical systems with abrupt changes (Chartrand, 2011)

Original: [Table]
Revised: [Table]

Original: To address this challenge
Revised: To address this challenge

Original: [Table]
Revised: [Table]

Original: Chartrand (2011) proposes to use explicit regularization on the ﬁnite difference estimator.
Revised: Chartrand (2011) proposes to use explicit regularization on the ﬁnite difference estimator.

Original: As we pointed out in Appendix A
Revised: As we pointed out in Appendix A

Original: [Table]
Revised: [Table]

Original: the standard L2 regularization is not suitable for derivative estimation.
Revised: the standard L2 regularization is not suitable for derivative estimation.

Original: The authors propose to use total variation regularization, a method originally developed in computer vision to obtain sharp denoised images (Strong & Chan, 2003).
Revised: The authors propose to use total variation regularization, a method originally developed in computer vision to obtain sharp denoised images (Strong & Chan, 2003).

Original: This type of regularization allows rapid (or even discountinous) change in ˙ x .
Revised: This type of regularization allows rapid (or even discountinous) change in ˙ x .

Original: The method has been adopted in the two-step sparse regression literature (Brunton et al., 2016)
Revised: The method has been adopted in the two-step sparse regression literature (Brunton et al., 2016)

Original: [Table]
Revised: [Table]

Original: and it is used in the benchmark SR-T.
Revised: and it is used in the benchmark SR-T.

Original: F I MPLEMENTATION DETAILS OF THE EXPERIMENTS F. D ETAILED SETTINGS FOR EACH EXPERIMENT
Revised: F I MPLEMENTATION DETAILS OF THE EXPERIMENTS F. D ETAILED SETTINGS FOR EACH EXPERIMENT

Original: The detailed settings for each experiment in Section 5 is shown in Table 3.
Revised: The detailed settings for each experiment in Section 5 is shown in Table 3.

Original: The time horizon T and the range of initial conditions [ a, b ] are chosen based on the scale and properties of the system.
Revised: The time horizon T and the range of initial conditions [ a, b ] are chosen based on the scale and properties of the system.

Original: For example, we observe that the Gompertz model reaches saturation and converges to the constant x =before t = 4 (hence we set the time horizon T = 4 ).
Revised: For example, we observe that the Gompertz model reaches saturation and converges to the constant x =before t = 4 (hence we set the time horizon T = 4 ).

Original: In practical applications, the time horizon is decided by the problem scope and the availability of experimental or observational data (see the discussion on dataset curation in Appendix C).
Revised: In practical applications, the time horizon is decided by the problem scope and the availability of experimental or observational data (see the discussion on dataset curation in Appendix C).

Original: F. H YPER - PARAMETER SETTINGS
Revised: F. H YPER - PARAMETER SETTINGS

Original: Smoothing algorithm .
Revised: Smoothing algorithm .

Original: In the experiments, D-CODE uses Gaussian process in the pre-processing step.
Revised: In the experiments, D-CODE uses Gaussian process in the pre-processing step.

Original: We use the radial basis kernel with the hyper-parameters tuned based on marginal log-likelihood (Bernardo et al., 1998).
Revised: We use the radial basis kernel with the hyper-parameters tuned based on marginal log-likelihood (Bernardo et al., 1998).

Original: Genetic programming .
Revised: Genetic programming .

Original: The hyperparameters of genetic programming is decided based on a pilot study.
Revised: The hyperparameters of genetic programming is decided based on a pilot study.

Original: In the pilot study, we consider a noiseless and frequently sampled Gompertz growth model ( σ = 0 , ∆ t = 0 . 01 ).
Revised: In the pilot study, we consider a noiseless and frequently sampled Gompertz growth model ( σ = 0 , ∆ t = 0 . 01 ).

Original: We use randomized search on Two-step Symbolic Regression to ﬁnd the hyperparameters that achieves the best RMSE loss (Table 1).
Revised: We use randomized search on Two-step Symbolic Regression to ﬁnd the hyperparameters that achieves the best RMSE loss (Table 1).

Original: The same hyperparameters are used by all algorithms on all experiments.
Revised: The same hyperparameters are used by all algorithms on all experiments.

Original: The values are listed below.
Revised: The values are listed below.

Original: We refer the reader to the documentation of gplearn library for a detailed explanation of these hyperparameters (Stephens, 2021).
Revised: We refer the reader to the documentation of gplearn library for a detailed explanation of these hyperparameters (Stephens, 2021).

Original: population size:2.
Revised: population size:2.

Original: tournament size:3.
Revised: tournament size:3.

Original: p crossover: 0.
Revised: p crossover: 0.

Original: p subtree mutation: 0.
Revised: p subtree mutation: 0.

Original: p hoist mutation: 0.
Revised: p hoist mutation: 0.

Original: p point mutation: 0.
Revised: p point mutation: 0.

Original: generations:8.
Revised: generations:8.

Original: parsimony coefﬁcient: 0.
Revised: parsimony coefﬁcient: 0.

Original: The set of allowed mathematical operations is
Revised: The set of allowed mathematical operations is

Original: [Equation]
Revised: [Equation]

Original:  D-CODE and all benchmarks use the same set of mathematical operations for fairness of comparison.
Revised: power [Equation] D-CODE and all benchmarks use the same set of mathematical operations for fairness of comparison.

Original: To speed up computation, we run a pilot study for each dynamical system in the noiseless setting described above to select a subset of these operations for the main experiments.
Revised: To speed up computation, we run a pilot study for each dynamical system in the noiseless setting described above to select a subset of these operations for the main experiments.

Original: Numerical differentiation .
Revised: Numerical differentiation .

Original: We use the python package derivative for numerical differentiation (Quade & Goldschmidt, 2020).
Revised: We use the python package derivative for numerical differentiation (Quade & Goldschmidt, 2020).

Original: The library contain implementations of the methods discussed in
Revised: The library contain implementations of the methods discussed in

Original: Appendix E. Currently, there lacks a principled way to tune the hyperparameters of numerical differentiation because the time derivative is not observed.
Revised: Appendix E. Currently, there lacks a principled way to tune the hyperparameters of numerical differentiation because the time derivative is not observed.

Original: Existing works often rely on heuristics (Chartrand, 2011).
Revised: Existing works often rely on heuristics (Chartrand, 2011).

Original: Here, we tune the hyperparameters based on the true derivative using grid search over (0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000) .
Revised: Here, we tune the hyperparameters based on the true derivative using grid search over (0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000) .

Original: Note that this procedure may allow the time derivative to be estimated more accurately than what is possible (because the true derivative is not available in real data).
Revised: Note that this procedure may allow the time derivative to be estimated more accurately than what is possible (because the true derivative is not available in real data).

Original: Hence, it will give an advantage to SR-T and SR-S because D-CODE does not use the true derivative in any way.
Revised: Hence, it will give an advantage to SR-T and SR-S because D-CODE does not use the true derivative in any way.

Original: SR-G uses the same Gaussian process as D-CODE, which is tuned by the marginal log-likelihood on the observed trajectory.
Revised: SR-G uses the same Gaussian process as D-CODE, which is tuned by the marginal log-likelihood on the observed trajectory.

Original: Correctness of the functional form .
Revised: Correctness of the functional form .

Original: As discussed in Section 2, the functional form of a closedform function f is the expression with all the numeric constants replaced by placeholders θ 1 , θ 2 , . . .
Revised: As discussed in Section 2, the functional form of a closedform function f is the expression with all the numeric constants replaced by placeholders θ 1 , θ 2 , . . .

Original: In practice, the uncovered function ˆ f may contain constants that are close to 0 .
Revised: In practice, the uncovered function ˆ f may contain constants that are close to 0 .

Original: For instance f ( x ) = x + 0 .
Revised: For instance f ( x ) = x + 0 .

Original: 001 x 2 is close to f ( x ) =
Revised: 001 x 2 is close to f ( x ) =

Original: x when x is small.
Revised: x when x is small.

Original: Hence, we apply thresholding on the estimated constants and set all constants between [ − 0 .
Revised: Hence, we apply thresholding on the estimated constants and set all constants between [ − 0 .

Original: as 0 (only for logistic model and
Revised: as 0 (only for logistic model and

Original: Glycolytic oscillator, where we found thresholding to be helpful).
Revised: Glycolytic oscillator, where we found thresholding to be helpful).

Original: The thresholding is applied to the results of all methods to evaluate success probability of recovering the functional form – and it is not used when calculating the RMSE of parameter estimation or the distance d x .
Revised: The thresholding is applied to the results of all methods to evaluate success probability of recovering the functional form – and it is not used when calculating the RMSE of parameter estimation or the distance d x .

Original: We use Python package sympy to decide whether the discovered functional form is correct (Meurer et al., 2017), i.e. success when simplify ( f ∗ − ˆ f ) returns 0 .
Revised: We use Python package sympy to decide whether the discovered functional form is correct (Meurer et al., 2017), i.e. success when simplify ( f ∗ − ˆ f ) returns 0 .

Original: F. C LEANING TUMOR GROWTH DATASET
Revised: F. C LEANING TUMOR GROWTH DATASET

Original: The original tumor growth dataset (Section 6) contains heterogeneous growth patterns and outliers.
Revised: The original tumor growth dataset (Section 6) contains heterogeneous growth patterns and outliers.

Original: However, the dataset does not include any patient covariate that could help to model or explain the heterogeneity, i.e. some variables that would drive the tumor growth dynamics are not recorded in the data, e.g. age and comorbidity (see the discussion in Section 7).
Revised: However, the dataset does not include any patient covariate that could help to model or explain the heterogeneity, i.e. some variables that would drive the tumor growth dynamics are not recorded in the data, e.g. age and comorbidity (see the discussion in Section 7).

Original: To address this issue, we performed k-means clustering on the observed trajectories (with 10 clusters), such that the growth pattern within each cluster is more or less homogeneous.
Revised: To address this issue, we performed k-means clustering on the observed trajectories (with 10 clusters), such that the growth pattern within each cluster is more or less homogeneous.

Original: We selected the largest cluster with 310 trajectories for the analysis.
Revised: We selected the largest cluster with 310 trajectories for the analysis.

Original: G A DDITIONAL RESULTS G. A DDITIONAL EVALUATION METRIC : d x
Revised: G A DDITIONAL RESULTS G. A DDITIONAL EVALUATION METRIC : d x

Original: Here we report the distance function d x ( ˆ f, f ∗ ) between the true and estimated ODEs.
Revised: Here we report the distance function d x ( ˆ f, f ∗ ) between the true and estimated ODEs.

Original: This metric tracks how well ˆ f approximates f ∗ .
Revised: This metric tracks how well ˆ f approximates f ∗ .

Original: The rationale for reporting this metric is that it may be possible that ˆ f has the wrong functional form but it still well approximates f ∗ .
Revised: The rationale for reporting this metric is that it may be possible that ˆ f has the wrong functional form but it still well approximates f ∗ .

Original: Hence, we need the distance d x in addition to the discovery probability.
Revised: Hence, we need the distance d x in addition to the discovery probability.

Original: The results are presented in Figure 7 and 8.
Revised: The results are presented in Figure 7 and 8.

Original: We observe that the patterns of d x and discovery probability (reported in the main text Section 5) are very similar — the performance gap between the methods is minor in the low-noise frequent-sampling scenarios, but D-CODE signiﬁcantly outperforms the benchmarks in the more challenging measurement settings.
Revised: We observe that the patterns of d x and discovery probability (reported in the main text Section 5) are very similar — the performance gap between the methods is minor in the low-noise frequent-sampling scenarios, but D-CODE signiﬁcantly outperforms the benchmarks in the more challenging measurement settings.

Original: The agreement between d x and discovery probability suggests that when the correct functional form is not recovered, the estimated ˆ f tends to perform poorly in approximating the true function as well.
Revised: The agreement between d x and discovery probability suggests that when the correct functional form is not recovered, the estimated ˆ f tends to perform poorly in approximating the true function as well.

Original: G. A DDITIONAL DYNAMICAL SYSTEMS
Revised: G. A DDITIONAL DYNAMICAL SYSTEMS

Original: Here we consider an additional dynamical system, which is an extension to the quadratic integrate and ﬁre (QIF) model in Neuroscience (Brunel & Latham, 2003).
Revised: Here we consider an additional dynamical system, which is an extension to the quadratic integrate and ﬁre (QIF) model in Neuroscience (Brunel & Latham, 2003).

Original: The governing equation is
Revised: The governing equation is

Original: [Equation]
Revised: [Equation]

Original: We highlight that this ODE is time-dependent and it involves a fraction , which makes it impossible to be symbolically represented in a linear form f ( x ) = (cid:80) k
Revised: We highlight that this ODE is time-dependent and it involves a fraction , which makes it impossible to be symbolically represented in a linear form f ( x ) = (cid:80) k

Original: θ k b k ( x ) (as required by the two-step sparse regression method).
Revised: θ k b k ( x ) (as required by the two-step sparse regression method).

Original: A similar example was given in Appendix B-2 of Brunton et al.
Revised: A similar example was given in Appendix B-2 of Brunton et al.

Original: (2016) to illustrate this failure mode of two-step sparse regression method.
Revised: (2016) to illustrate this failure mode of two-step sparse regression method.

Original: The simulation results are shown in Figure 9.
Revised: The simulation results are shown in Figure 9.

Original: We observe the same pattern as the other dynamical systems considered in the main text.
Revised: We observe the same pattern as the other dynamical systems considered in the main text.

Original: G. S ENSITIVITY ANALYSIS FOR TESTING FUNCTION
Revised: G. S ENSITIVITY ANALYSIS FOR TESTING FUNCTION

Original: Here we study the sensitivity of D-CODE to the choice and number of testing functions.
Revised: Here we study the sensitivity of D-CODE to the choice and number of testing functions.

Original: We consider two families of testing function: sine functions and cubic spline functions, both of which satisfy the conditions in Section 4.2.
Revised: We consider two families of testing function: sine functions and cubic spline functions, both of which satisfy the conditions in Section 4.2.

Original: We refer the readers to De Boor (1978) for the exact recursive deﬁnition of cubic spline functions.
Revised: We refer the readers to De Boor (1978) for the exact recursive deﬁnition of cubic spline functions.

Original: Figure 10 reports the discovery probability for Gompertz model (ﬁrst row) and Lorenz system (second row).
Revised: Figure 10 reports the discovery probability for Gompertz model (ﬁrst row) and Lorenz system (second row).

Original: We ﬁrst observe that using too few testing functions (i.e. < 40) leads to performance loss.
Revised: We ﬁrst observe that using too few testing functions (i.e. < 40) leads to performance loss.

Original: This agrees with Theorem 1 because the objective function Eq.
Revised: This agrees with Theorem 1 because the objective function Eq.

Original: 5 cannot well-approximate the distance d x (Eq. 6) with too few testing functions.
Revised: 5 cannot well-approximate the distance d x (Eq. 6) with too few testing functions.

Original: We also observe that the performance is reasonably stable when enough testing functions are included (e.g. 40 - 80 sine functions, and 80 - 100 cubic splines).
Revised: We also observe that the performance is reasonably stable when enough testing functions are included (e.g. 40 - 80 sine functions, and 80 - 100 cubic splines).

Original: Finally, we see a performance drop for using too many (e.g. 100) sine functions.
Revised: Finally, we see a performance drop for using too many (e.g. 100) sine functions.

Original: After investigation, we ﬁnd that the drop is due to the numerical error in the integration procedure — sine functions with very high frequency are oscillating too rapidly for accurate numerical integration.
Revised: After investigation, we ﬁnd that the drop is due to the numerical error in the integration procedure — sine functions with very high frequency are oscillating too rapidly for accurate numerical integration.

Original: One may resolve this issue by using a smaller integration step size or a more advanced numerical integration method.
Revised: One may resolve this issue by using a smaller integration step size or a more advanced numerical integration method.

Original: For practical applications, the user may experiment with different numbers of testing functions and examine the discovered ODEs.
Revised: For practical applications, the user may experiment with different numbers of testing functions and examine the discovered ODEs.

Original: A key advantage of discovering close-form ODEs is that it allows the user to examine and reﬁne the ODEs in an iterative way.
Revised: A key advantage of discovering close-form ODEs is that it allows the user to examine and reﬁne the ODEs in an iterative way.

Original: G. T RAINING TIME
Revised: G. T RAINING TIME

Original: Figure 11 compares the average training time of D-CODE and two-step symbolic regression (SR-T).
Revised: Figure 11 compares the average training time of D-CODE and two-step symbolic regression (SR-T).

Original: We expect D-CODE to have longer training time due to the numerical integration step in evaluating the functional C (Eq. 3).
Revised: We expect D-CODE to have longer training time due to the numerical integration step in evaluating the functional C (Eq. 3).

Original: However, even for the longest-running task, D-CODE ﬁnishes training within 30 minutes on average.
Revised: However, even for the longest-running task, D-CODE ﬁnishes training within 30 minutes on average.

Original: This can still be a very signiﬁcant speed up compared with manual discovery by human experts.
Revised: This can still be a very signiﬁcant speed up compared with manual discovery by human experts.

Original: Experiments are run on a computer with Intel Xeon E3-12xx v2 CPU (16 cores) and 60 GB memory.
Revised: Experiments are run on a computer with Intel Xeon E3-12xx v2 CPU (16 cores) and 60 GB memory.

Original: G. T ABLE OF DISCOVERED ODE S
Revised: G. T ABLE OF DISCOVERED ODE S

Original: The top-3 most common equations discovered by different methods for each dynamical 
Revised: The top-3 most common equations discovered by different methods for each dynamical system

Original: U nd e r r e v i e w a s a c on f e r e n ce p a p e r a t
Revised: P ub li s h e d a s a c on f e r e n ce p a p e r a t

Original: I C L
Revised: I C L

Original: R
Revised: R

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Lorenz
Revised: Lorenz

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Lorenz
Revised: Lorenz

Original: [Table]
Revised: [Table]

Original: G. V ISUALIZATION OF SMOOTHED AND TRUE TRAJECTORIES
Revised: G. V ISUALIZATION OF SMOOTHED AND TRUE TRAJECTORIES

Original: In this section we study whether D-CODE can still discover the ODE when the smoothed trajectory ˆ x ( t ) is signiﬁcantly different from the true trajectory x ( t ) ; for instance, when the signal-to-noise ratio is low or the observations are sparse.
Revised: In this section we study whether D-CODE can still discover the ODE when the smoothed trajectory ˆ x ( t ) is signiﬁcantly different from the true trajectory x ( t ) ; for instance, when the signal-to-noise ratio is low or the observations are sparse.

Original: In Section 5 we provided quantitative evaluation under scenarios with different noise levels ( σ R ) and sampling sparsity ( ∆ t ).
Revised: In Section 5 we provided quantitative evaluation under scenarios with different noise levels ( σ R ) and sampling sparsity ( ∆ t ).

Original: 4 For further illustration, here we visualize ˆ x ( t ) and x ( t ) together with the observations y ( t ) in Figure 12.
Revised: For further illustration, here we visualize ˆ x ( t ) and x ( t ) together with the observations y ( t ) in Figure 12.

Original: In the most favorable setting (top left), ˆ x ( t ) is indistinguishable from x ( t ) , and D-CODE achieves very high success probability as expected.
Revised: In the most favorable setting (top left), ˆ x ( t ) is indistinguishable from x ( t ) , and D-CODE achieves very high success probability as expected.

Original: As the observation setting becomes more challenging (middle column), ˆ x ( t ) may recover the overall trend of x ( t ) but it may also contain some estimation artifacts (e.g. the unnecessary zigzag pattern).
Revised: As the observation setting becomes more challenging (middle column), ˆ x ( t ) may recover the overall trend of x ( t ) but it may also contain some estimation artifacts (e.g. the unnecessary zigzag pattern).

Original: However, D-CODE is still able to recover the true ODE with very high probability.
Revised: However, D-CODE is still able to recover the true ODE with very high probability.

Original: In the most challenging settings (right column), ˆ x ( t ) differs signiﬁcantly from x ( t ) for extended periods of time (top right t > 2 , bottom right t < 2 and t > 4 ).
Revised: In the most challenging settings (right column), ˆ x ( t ) differs signiﬁcantly from x ( t ) for extended periods of time (top right t > 2 , bottom right t < 2 and t > 4 ).

Original: Yet, D-CODE still makes successful discovery at times.
Revised: Yet, D-CODE still makes successful discovery at times.

Original: H D ISCUSSION ON ALTERNATIVE METHODS
Revised: H D ISCUSSION ON ALTERNATIVE METHODS

Original: In this section, we discuss the alternative methods that, in theory, could also be applied to discovering closed form ODEs from data.
Revised: In this section, we discuss the alternative methods that, in theory, could also be applied to discovering closed form ODEs from data.

Original: However, as we will show, these methods suffer from various drawbacks that may severely limit their practical utility.
Revised: However, as we will show, these methods suffer from various drawbacks that may severely limit their practical utility.

Original: H. D OUBLE OPTIMIZATION METHOD
Revised: H. D OUBLE OPTIMIZATION METHOD

Original: The double optimization method (DO) was ﬁrst proposed in the literature of ODE parametric estimation, where the functional form of f is known but it contains unknown parameters θ (see
Revised: The double optimization method (DO) was ﬁrst proposed in the literature of ODE parametric estimation, where the functional form of f is known but it contains unknown parameters θ (see

Original: Ramsay et al.
Revised: Ramsay et al.

Original: (2007) Section 1.3 for a review).
Revised: (2007) Section 1.3 for a review).

Original: 5 DO solves the following optimization problem (we consider the one-dimensional case to simplify notations):
Revised: 6 DO solves the following optimization problem (we consider the one-dimensional case to simplify notations):

Original: [Equation]
Revised: [Equation]

Original: This is a nested optimization problem where the outer part searches for the unknown function f and the inner part searches for the unknown initial conditions x i (0) given the current guess of f .
Revised: This is a nested optimization problem where the outer part searches for the unknown function f and the inner part searches for the unknown initial conditions x i (0) given the current guess of f .

Original: Note that the inner and outer parts are coupled because the optimal ˆ x i (0) depends on f , e.g. the best-ﬁtting exponential curve would have a different initial condition from the best linear ﬁt.
Revised: Note that the inner and outer parts are coupled because the optimal ˆ x i (0) depends on f , e.g. the best-ﬁtting exponential curve would have a different initial condition from the best linear ﬁt.

Original: A well-known shortcoming of DO is its computational complexity (Gugushvili & Klaassen, 2012).
Revised: A well-known shortcoming of DO is its computational complexity (Gugushvili & Klaassen, 2012).

Original: The inner optimization over x i (0) is typically non-convex and may take many steps to converge.
Revised: The inner optimization over x i (0) is typically non-convex and may take many steps to converge.

Original: To further aggravate the problem, each inner optimization step requires solving the IVP numerically.
Revised: To further aggravate the problem, each inner optimization step requires solving the IVP numerically.

Original: The computational challenge becomes even more pronounced in our setting, where the functional form of f is also unknown.
Revised: The computational challenge becomes even more pronounced in our setting, where the functional form of f is also unknown.

Original: Genetic programming needs to evaluate many candidate functions f
Revised: Genetic programming needs to evaluate many candidate functions f

Original: but
Revised: but

Original: DO requires solving the inner optimization for each evaluation.
Revised: DO requires solving the inner optimization for each evaluation.

Original: In contrast, D-CODE (Equation 5) does not involve any nested optimization.
Revised: In contrast, D-CODE (Equation 5) does not involve any nested optimization.

Original: To illustrate this point, we have implemented DO and measured its computation time to evaluate a candidate f .
Revised: To illustrate this point, we have implemented DO and measured its computation time to evaluate a candidate f .

Original: The results are shown in Figure 13.
Revised: The results are shown in Figure 13.

Original: We found that DO is typically 30 - 50 times slower than D-CODE.
Revised: We found that DO is typically 30 - 50 times slower than D-CODE.

Original: Its computational cost is too high to be considered as a feasible method for discovering closed-form ODEs.
Revised: Its computational cost is too high to be considered as a feasible method for discovering closed-form ODEs.

Original: H. N EURAL ODE WITH POST - HOC DISTILLATION
Revised: H. N EURAL ODE WITH POST - HOC DISTILLATION

Original: Although Neural ODE (NODE) does not directly produce closed-form ODEs, one may attempt to distill a closed-form equation from the trained network.
Revised: Although Neural ODE (NODE) does not directly produce closed-form ODEs, one may attempt to distill a closed-form equation from the trained network.

Original: The method takes two steps: (1) train a NODE f N using observed trajectories and query the trained NODE to produce a dataset { x i ( t ) , f N ( x i ( t )) } i,t ; (2) apply symbolic regression on this dataset.
Revised: The method takes two steps: (1) train a NODE f N using observed trajectories and query the trained NODE to produce a dataset { x i ( t ) , f N ( x i ( t )) } i,t ; (2) apply symbolic regression on this dataset.

Original: This approach can also be viewed as a two-step regression method (Section 3), where the derivatives are estimated by a NODE.
Revised: This approach can also be viewed as a two-step regression method (Section 3), where the derivatives are estimated by a NODE.

Original: The standard approach to train a NODE for time series modeling is to minimize the error over a prediction horizon T ∗ =
Revised: The standard approach to train a NODE for time series modeling is to minimize the error over a prediction horizon T ∗ =

Original: [ t 1 , t 2 , . .
Revised: [ t 1 , t 2 , . .

Original: , T ∗ ] (Rubanova et al., 2019), i.e. to minimize
Revised: , T ∗ ] (Rubanova et al., 2019), i.e. to minimize

Original: [Equation]
Revised: [Equation]

Original: where ˆ x i ( t ) is obtained by solving the IVP of the NODE.
Revised: where ˆ x i ( t ) is obtained by solving the IVP of the NODE.

Original: However, the choice of prediction horizon T ∗ is non-trivial and plays an important role in the training.
Revised: However, the choice of prediction horizon T ∗ is non-trivial and plays an important role in the training.

Original: For chaotic systems, the user should choose a smaller T ∗ because the system is unpredictable in the long term.
Revised: For chaotic systems, the user should choose a smaller T ∗ because the system is unpredictable in the long term.

Original: On the other hand, making T ∗ too small may lead to a myopic model that fails to capture long-term patterns (Schmidt et al., 2020; Kantz & Schreiber, 2004).
Revised: On the other hand, making T ∗ too small may lead to a myopic model that fails to capture long-term patterns (Schmidt et al., 2020; Kantz & Schreiber, 2004).

Original: In practice, the prediction horizon T ∗ is a hyper-parameter that may be difﬁcult to decide (note that the Lyapunov time of the true system is unknown).
Revised: In practice, the prediction horizon T ∗ is a hyper-parameter that may be difﬁcult to decide (note that the Lyapunov time of the true system is unknown).

Original: In contrast, D-CODE does not require user to specify the prediction horizon
Revised: In contrast, D-CODE does not require user to specify the prediction horizon

Original: T ∗ .
Revised: T ∗ .

Original: Furthermore, as we alluded to earlier, this approach is an instance of two-step regression, where the derivative is estimated by NODE instead of numerical differentiation.
Revised: Furthermore, as we alluded to earlier, this approach is an instance of two-step regression, where the derivative is estimated by NODE instead of numerical differentiation.

Original: Although NODE is known to be a universal function approximator, its estimation efﬁciency (e.g. √ n consistency), ﬁnite sample performance, and robustness to noise are still largely under investigation.
Revised: Although NODE is known to be a universal function approximator, its estimation efﬁciency (e.g. √ n consistency), ﬁnite sample performance, and robustness to noise are still largely under investigation.

Original: Hence, we should remind the readers that NODE is not guaranteed to outperform numerical differentiation in estimating derivatives.
Revised: Hence, we should remind the readers that NODE is not guaranteed to outperform numerical differentiation in estimating derivatives.

Original: To illustrate these points, we evaluate the performance of NODE with post-hoc distillation.
Revised: To illustrate these points, we evaluate the performance of NODE with post-hoc distillation.

Original: We consider two versions of NODE: the NODE-L is trained on long term prediction ( T ∗ = 25 ) and the NODE-S is trained on short term prediction ( T ∗ = 0 . 04 ).
Revised: We consider two versions of NODE: the NODE-L is trained on long term prediction ( T ∗ = 25 ) and the NODE-S is trained on short term prediction ( T ∗ = 0 . 04 ).

Original: The evaluation is performed on the
Revised: The evaluation is performed on the

Original: Gompertz ODE (non-chaotic, one dimensional) and the Lorenz system (chaotic, three dimensional).
Revised: Gompertz ODE (non-chaotic, one dimensional) and the Lorenz system (chaotic, three dimensional).

Original: Figure 14 shows the success probability (bigger better) and Figure 15 shows the distance d x (Equation 6, smaller better).
Revised: Figure 14 shows the success probability (bigger better) and Figure 15 shows the distance d x (Equation 6, smaller better).

Original: For the non-chaotic Gompertz ODE, both NODE-L and NODE-S are only able to recover the correct functional form when the noise is very close to zero ( σ R = 0 . 01 ).
Revised: For the non-chaotic Gompertz ODE, both NODE-L and NODE-S are only able to recover the correct functional form when the noise is very close to zero ( σ R = 0 . 01 ).

Original: Note that this behavior is very similar to other two-step regression methods (SR-T, SR-S and SR-G, Figure 3).
Revised: Note that this behavior is very similar to other two-step regression methods (SR-T, SR-S and SR-G, Figure 3).

Original: We also observe that the NODE-L achieves consistently better d x than NODE-S.
Revised: We also observe that the NODE-L achieves consistently better d x than NODE-S.

Original: This conﬁrms that training over longer time horizon leads to better estimate of the ODE when the system is non-chaotic.
Revised: This conﬁrms that training over longer time horizon leads to better estimate of the ODE when the system is non-chaotic.

Original: Turning to the chaotic Lorenz system, we observe that NODE-L always fails to discover the correct
Revised: Turning to the chaotic Lorenz system, we observe that NODE-L always fails to discover the correct

Original: ODE and its d x is higher than the other two methods.
Revised: ODE and its d x is higher than the other two methods.

Original: This veriﬁes our expectation that training with long-term prediction is not feasible for chaotic systems.
Revised: This veriﬁes our expectation that training with long-term prediction is not feasible for chaotic systems.

Original: As before, NODE-S performs well only when the noise level is relatively small.
Revised: As before, NODE-S performs well only when the noise level is relatively small.

Original: We also observe that its d x monotonically increases with the noise level.
Revised: We also observe that its d x monotonically increases with the noise level.

Original: This suggests that higher noise level signiﬁcantly reduces NODE-S
Revised: This suggests that higher noise level signiﬁcantly reduces NODE-S

Original: ’s ability to approximate the unknown ODE, limiting its performance on noisy data.
Revised: ’s ability to approximate the unknown ODE, limiting its performance on noisy data.