Original: E FFICIENT D ISCOVERY OF D YNAMICAL L AWS IN S YMBOLIC F ORM
Revised: E FFICIENT D ISCOVERY OF D YNAMICAL L AWS IN S YMBOLIC F ORM

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a transformer-based sequence-to-sequence model that recovers scalar ordinary differential equations (ODEs) in symbolic form from time-series data of a single observed solution trajectory of the ODE.
Revised: We propose a transformer-based sequence-to-sequence model that recovers scalar ordinary differential equations (ODEs) in symbolic form from time-series data of a single observed solution trajectory of the ODE.

Original: Our method is efficiently scalable: after one-time pretraining on a large set of ODEs, we can infer the governing laws of a new observed solution in a few forward passes of the model.
Revised: Our method is efficiently scalable: after one-time pretraining on a large set of ODEs, we can infer the governing laws of a new observed solution in a few forward passes of the model.

Original: First, we generate and make available a large dataset of more than 3M ODEs together with more than 63M numerical solutions for different initial conditions that may serve as a useful benchmark for future work on machine learning for dynamical systems.
Revised: First, we generate and make available a large dataset of more than 3M ODEs together with more than 63M numerical solutions for different initial conditions that may serve as a useful benchmark for future work on machine learning for dynamical systems.

Original: Then we show that our model performs better or on par with existing methods in various test cases in terms of accurate symbolic recovery of the ODE, especially for more complex expressions.
Revised: Then we show that our model performs better or on par with existing methods in various test cases in terms of accurate symbolic recovery of the ODE, especially for more complex expressions.

Original: Reliably recovering the symbolic form of dynamical laws is important as it allows for further dissemination of the inferred dynamics as well as meaningful modifications for predictions under interventions.
Revised: Reliably recovering the symbolic form of dynamical laws is important as it allows for further dissemination of the inferred dynamics as well as meaningful modifications for predictions under interventions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Science is commonly described as the “discovery of natural laws through experimentation and observation”.
Revised: Science is commonly described as the “discovery of natural laws through experimentation and observation”.

Original: Researchers in the natural sciences increasingly turn to machine learning (ML) to aid the discovery of natural laws from observational data alone, which is often abundantly available, hoping to bypass expensive and cumbersome targeted experimentation.
Revised: Researchers in the natural sciences increasingly turn to machine learning (ML) to aid the discovery of natural laws from observational data alone, which is often abundantly available, hoping to bypass expensive and cumbersome targeted experimentation.

Original: While there may be fundamental limitations to what can be extracted from observations alone, recent successes of ML in the entire range of natural sciences provide ample reason for excitement.
Revised: While there may be fundamental limitations to what can be extracted from observations alone, recent successes of ML in the entire range of natural sciences provide ample reason for excitement.

Original: In this work, we focus on ordinary differential equations, a ubiquitous description of dynamical natural laws in physics, chemistry, and systems biology.
Revised: In this work, we focus on ordinary differential equations, a ubiquitous description of dynamical natural laws in physics, chemistry, and systems biology.

Original: For a first order ODE ˙ y :
Revised: For a first order ODE ˙ y :

Original: =
Revised: =

Original: ∂y / ∂t = f
Revised: ∂y / ∂t = f

Original: ( y, t ) , we call f (which uniquely defines the
Revised: ( y, t ) , we call f (which uniquely defines the

Original: ODE) the underlying dynamical law.
Revised: ODE) the underlying dynamical law.

Original: Informally, our goal is then to infer f in symbolic form given discrete time-series observations of a single solution { y i :=
Revised: Informally, our goal is then to infer f in symbolic form given discrete time-series observations of a single solution { y i :=

Original: y ( t i ) } ni
Revised: y ( t i ) } ni

Original: =1 of the underlying ODE.
Revised: =1 of the underlying ODE.

Original: Contrary to “black-box-techniques” such as Neural Ordinary Differential Equations (NODE) (Chen et al., 2018) that aim at inferring a possible f as an arguably opaque neural network, we focus specifically on symbolic regression.
Revised: Contrary to “black-box-techniques” such as Neural Ordinary Differential Equations (NODE) (Chen et al., 2018) that aim at inferring a possible f as an arguably opaque neural network, we focus specifically on symbolic regression.

Original: From the perspective of the sciences, a law of nature is useful insofar as it is more broadly applicable than to merely describe a single observation.
Revised: From the perspective of the sciences, a law of nature is useful insofar as it is more broadly applicable than to merely describe a single observation.

Original: In particular, the reason to learn a dynamical law in the first place is to dissect and understand it as well as to make predictions about situations that differ from the observed one.
Revised: In particular, the reason to learn a dynamical law in the first place is to dissect and understand it as well as to make predictions about situations that differ from the observed one.

Original: From this perspective, a symbolic representation of the law (in our case the function f ) has several advantages over block-box representations: they are compact and directly interpretable, they are amenable to analytic analysis, they allow for meaningful changes and thus enable assessment of interventions and counterfactuals.
Revised: From this perspective, a symbolic representation of the law (in our case the function f ) has several advantages over block-box representations: they are compact and directly interpretable, they are amenable to analytic analysis, they allow for meaningful changes and thus enable assessment of interventions and counterfactuals.

Original: In this work, we develop N eural S ymbolic O rdinary D ifferential E quation ( NSODE ), a sequence-tosequence transformer to efficiently infer governing ODEs in symbolic form from a single observed solution trajectory that makes use of massive pretraining.
Revised: In this work, we develop N eural S ymbolic O rdinary D ifferential E quation ( NSODE ), a sequence-tosequence transformer to efficiently infer governing ODEs in symbolic form from a single observed solution trajectory that makes use of massive pretraining.

Original: We first (randomly) generate a total of > 3M scalar, autonomous, non-linear, first-order ODEs, together with a total of > 63M numerical solutions from various (random) initial conditions.
Revised: We first (randomly) generate a total of > 3M scalar, autonomous, non-linear, first-order ODEs, together with a total of > 63M numerical solutions from various (random) initial conditions.

Original: All solutions are carefully checked for convergence of the numerical integration.
Revised: All solutions are carefully checked for convergence of the numerical integration.

Original: This dataset is unprecedented in both its scale and diversity and will be made publicly available alongside the code that was used to generate it.
Revised: This dataset is unprecedented in both its scale and diversity and will be made publicly available alongside the code that was used to generate it.

Original: We then devise NSODE, a sequence-to-sequence transformer that maps observed trajectories, i.e., numeric sequences of the form { ( t i , y i ) } ni =1 , directly to symbolic equations as strings, e.g., "y ** 2+1.64 * cos(y)" , which is the prediction for f .
Revised: We then devise NSODE, a sequence-to-sequence transformer that maps observed trajectories, i.e., numeric sequences of the form { ( t i , y i ) } ni =1 , directly to symbolic equations as strings, e.g., "y ** 2+1.64 * cos(y)" , which is the prediction for f .

Original: This example directly highlights the benefit
Revised: This example directly highlights the benefit

Original: [Figure]
Revised: [Figure]

Original: over ODEs
Revised: over ODEs

Original: O pe r a t o r s
Revised: O pe r a t o r s

Original: S y m bo l s e x p + l og c
Revised: S y m bo l s e x p + l og c

Original: o s s i
Revised: o s s i

Original: n y y
Revised: n y y

Original: ×
Revised: ×

Original: ÷ po w + × y y 2.
Revised: ÷ po w + × y y 2.

Original: exp cos f i x ed nu m be r
Revised: exp cos f i x ed nu m be r

Original: o f ope r a t
Revised: o f ope r a t

Original: o r s b i na r
Revised: o r s b i na r

Original: i
Revised: i

Original: z
Revised: z

Original: e
Revised: e

Original: ( I EEE - 754 )
Revised: ( I EEE - 754 )

Original: ODE solver y*2.3+exp(cos(y)) scipy.integrate.odeint sample initial values sample f(y) ẏ = f(y)
Revised: ODE solver y*2.3+exp(cos(y)) scipy.integrate.odeint sample initial values sample f(y) ẏ = f(y)

Original: y 

Revised: y

Original: ~
Revised: ~

Original: [Figure] 
Revised: [Figure] >

Original: Model inputs
Revised: Model inputs

Original: Model target y t t y n                    t
Revised: Model target y t t y n                    t

Original: y n
Revised: y n

Original: Transformer
Revised: Transformer

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: y

Original: [Figure]
Revised: [Figure]

Original: b i na r y una
Revised: b i na r y una

Original: r
Revised: r

Original: y of symbolic representations in that the y 2 and cos( y ) terms tell us something about the fundamental dynamics of the observed system; the constant 1.64 will have semantic meaning in a given context and we can, for example, make predictions about settings in which this constant takes a different value.
Revised: y of symbolic representations in that the y 2 and cos( y ) terms tell us something about the fundamental dynamics of the observed system; the constant 1.64 will have semantic meaning in a given context and we can, for example, make predictions about settings in which this constant takes a different value.

Original: NSODE combines and innovates on technical advances regarding input representations and an efficiently optimizable loss formulation.
Revised: NSODE combines and innovates on technical advances regarding input representations and an efficiently optimizable loss formulation.

Original: Our model outperforms scalable existing methods in terms of skeleton recovery, especially on more complex expressions.
Revised: Our model outperforms scalable existing methods in terms of skeleton recovery, especially on more complex expressions.

Original: While other methods still perform better on simple expressions found in (adjusted) existing benchmark datasets (which these methods have been tuned to) as well as a novel set of simple ODEs that we manually collected from different domains, they are typically orders of magnitude slower than NSODE.
Revised: While other methods still perform better on simple expressions found in (adjusted) existing benchmark datasets (which these methods have been tuned to) as well as a novel set of simple ODEs that we manually collected from different domains, they are typically orders of magnitude slower than NSODE.

Original: We publish all code (including benchmarks) and final model checkpoints.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original:  While NODE (Chen et al., 2018) (with a large body of follow up work) is perhaps the most prominent method to learn ODEs from data in black-box form, we focus on various works have aimed at inferring governing laws in symbolic form.
Revised: Modeling dynamics and forecasting their behavior has a long history in machine learning. While NODEs (Chen et al., 2018) (with a large body of follow up work) are perhaps the most prominent approach, their inherent black-box character complicates scientific understanding of the observed phenomena.

Original: Classically, symbolic regression aims at regular functional relationships (mapping ( x, f ( x )) pairs to f instead of mapping trajectories t, y ( t ) to the governing ODE ˙ y =
Revised: Recent alternatives such as tractable dendritic RNNs (Brenner et al., 2022) and Neural Operators (Kovachki et al., 2021; Li et al., 2021) set out to facilitate scientific discovery by combining accurate predictions with improved interpretability.

Original: f
Revised: A considerable advantage of these and similar approaches is their scalability to high dimensional systems as well as their relative robustness to noise, missing or irregularly sampled data (Iakovlev et al., 2021) and challenging properties such as multiscale dynamics (Vlachas et al., 2022) or chaos (Park et al., 2022; Patel & Ott, 2022). Despite these advantages, we turn the focus to a different class of models in this paper and look at approaches that explicitly predict a mathematical expression in symbolic form that describes the observed dynamics.

Original: ( y, t ) )
Revised: Such models are in a sense situated on the other side of the spectrum: while less scalable, the predicted symbolic expression is compact and readily interpretable so that dynamical properties can be analytically deduced and investigated. A recent benchmark study and great overview including deep learning-based as well as symbolic models can be found in Gilpin (2021). Evolutionary algorithms.

Original: and has been approached by heuristics-based search, most prominently via genetic programming (Koza, 1993).
Revised: Classically, symbolic regression is approached through evolutionary algorithms such as genetic programming (Koza, 1993).

Original: Genetic programming randomly evolves a population of prospective mathematical expressions over many iterations and mimics natural selection by keeping only the best contenders across iterations, where superiority is measured by user-definedand problem-specific fitness functions.
Revised: Genetic programming randomly evolves a population of prospective mathematical expressions over many iterations and mimics natural selection by keeping only the best contenders across iterations, where superiority is measured by user-defined fitness functions (Schmidt & Lipson, 2009).

Revised: Process-based modeling follows a similar approach but includes domain knowledge-informed constraints on particular components of the system in order to reduce the search space to reasonable candidates (Todorovski & Dzeroski, 1997; Bridewell et al., 2008; Simidjievski et al., 2020).

Original:  More recently, symbolic regression has been approached with machine learning methods which exploit gradient information to optimize within the space of (finite) compositions of pre-defined basis functions.
Revised: Gradient-based search. More recently, symbolic regression has been approached with machine learning methods which exploit gradient information to optimize within the space of (finite) compositions of pre-defined basis functions.

Original: Brunton et al.
Revised: Brunton et al.

Original: (2016) use linear regression to identify a(sparse) linear combination of basis functions that yields the best fit for the observed data, while other approaches use neural networks with a diverse set of activation functions (Sahoo et al., 2018;
Revised: (2016) builds on sparse linear regression to identify a linear combination of basis functions that yields the best fit for the observed data.

Original: Long et al., 2019; Liu et al., 2020).
Revised: This approach has inspired a large body of follow-up work that generalize the idea to partial observations (Bakarji et al., 2022), parameterized functions (Lejarza & Baldea, 2022), simultaneous discovery of coordinates (Champion et al., 2019), coordinate transformations that linearize the dynamics (Lusch et al., 2018) and partial differential equations (Rudy et al., 2017).

Original: All these techniques  deploystrong sparsity-promoting regularizers and train one model for each set of observations.
Revised: These techniques often deploy sparsity-promoting regularizers and train one model for each set of observations.

Original: Once trained, the model itself represents the predicted symbolic expression, which can be read off the non-zero coefficients.
Revised: Once trained, the model itself represents the predicted symbolic expression, which can be read off the non-zero coefficients.

Revised: This modeling principle is also employed by many other approaches that replace linear regression by neural networks with diverse sets of activation functions, both for differential equations (Long et al., 2019; Liu et al., 2020) and non-differential algebraic equations (Sahoo et al., 2018).

Original:  Alternatively, one can train a model to directly output the symbolic expressions (for example as a string).
Revised: Hybrid models. Alternatively, one can train a model to directly output the symbolic expressions (for example as a string).

Original: Supervised learning with gradient-based optimization for this approach is challenged by the formulation of a differentiable loss that measures the fit between the predicted symbolic expression and the observed data.
Revised: Supervised learning with gradient-based optimization for this approach is challenged by the formulation of a differentiable loss that measures the fit between the predicted symbolic expression and the observed data.

Original: Thus, prior work resorted to reinforcement learning (Petersen et al., 2021) or  evolutionary algorithms (Atkinson et al., 2019; Costa et al., 2021) for gradient-free optimization.
Revised: Thus, prior work, mostly in the context of non-differential equation prediction, resorted to reinforcement learning (Petersen et al., 2021) or combinations of neural networks and evolutionary algorithms (Atkinson et al., 2019; Costa et al., 2021).

Original: A hybrid approach combining gradient-free, human intuition-guided heuristic search via genetic programming with neural network-based optimization has been presented  by Udrescu et al.
Revised: A hybrid approach combining gradient-free, human intuition-guided heuristic search via genetic programming with neural network-based optimization has been presented for non-differential equations by Udrescu et al.

Original: and extended towork with dynamical systems by Weilbach et al.
Revised: and extended to dynamical systems by Weilbach et al.

Original:  The closest works to ours use pre-trained, attention-based sequence-to-sequence models for symbolic regression of functional relationships (Biggio et al., 2021;
Revised: Sequence-to-sequence models. The closest works to ours use pre-trained, attention-based sequenceto-sequence models for symbolic regression of functional relationships (Biggio et al., 2021;

Original: Valipour et al., 2021; Kamienny et al., 2022; Vastl et al.,  2022).
Revised: Valipour et al., 2021; Kamienny et al., 2022; Vastl et al., 2022) or (discrete) recurrence relations (D’Ascoli et al., 2022).

Original: They exploit the fact that symbolic expressions for (multi-variate) scalar functions can be both generated and evaluated on random inputs cheaply, resulting in essentially unlimited training data.
Revised: They exploit the fact that symbolic expressions for (multi-variate) scalar functions can be both generated and evaluated on random inputs cheaply, resulting in essentially unlimited training data.

Original: Large data including ground-truth expressions in symbolic form allow for a differentiable cross-entropy loss based directly on the symbols of the expression, instead of the numerical proximity between evaluations of the predicted and true expression.
Revised: Large data including ground-truth expressions in symbolic form allow for a differentiable cross-entropy loss based directly on the symbols of the expression, instead of the numerical proximity between evaluations of the predicted and true expression.

Original: While the crossentropy loss works well for operators and symbols (e.g. +,exp,sin,x,y ), a naive implementation is inefficient for numerical constants, e.g., 1.452 .
Revised: While the cross-entropy loss works well for operators and symbols (e.g. +,exp,sin,x,y ), a naive implementation is inefficient for numerical constants, e.g. 1.452 .

Original: As a workaround previous works resort to one of two strategies: 1) represent all constants with a special <const> token when training the sequence-to-sequence model and predict only the presence of a constant.
Revised: As a workaround previous works resort to one of two strategies: 1) represent all constants with a special <const> token when training the sequence-to-sequence model and predict only the presence of a constant.

Original: Actual values are then inferred in a second, subsequent optimization step that minimizes (cid:80) ni =1 ∥
Revised: Actual values are then inferred in a second, subsequent optimization step that minimizes (cid:80) ni =1 ∥

Original: y i − ˆ y c 1 ,c 2 ,...
Revised: y i − ˆ y c 1 ,c 2 ,...

Original: ( x i ) ∥ 2 .
Revised: ( x i ) ∥ 2 .

Original: This second optimization procedure comes with substantial computational cost as constants have to be fit per inferred expression.
Revised: This second optimization procedure comes with substantial computational cost as constants have to be fit per inferred expression.

Original: In particular, we highlight that it does not transfer to inferring ODEs as it would require to first solve the predicted ODE ˙ y
Revised: In particular, we highlight that it does not transfer to inferring ODEs as it would require to first solve the predicted ODE ˙ y

Original: = ˆ f ( y ) to obtain predicted { ˆ y i } ni =1 values that can be compared to the set of observations { y i } ni =1 .
Revised: = ˆ f ( y ) to obtain predicted { ˆ y i } ni =1 values that can be compared to the set of observations { y i } ni =1 .

Original: While differentiable ODE solvers exist, optimizing constants this way is prohibitively expensive and typically highly unstable.
Revised: While differentiable ODE solvers exist, optimizing constants this way is prohibitively expensive and typically highly unstable.

Original: A second popular strategy consists in rounding constants within the range of interest so that they can be represented with a finite number of tokens.
Revised: A second popular strategy consists in rounding constants within the range of interest so that they can be represented with a finite number of tokens.

Original: This second strategy avoids a subsequent optimization step and enjoys clever encoding schemes with improved token efficiency yet represents values with an inherent loss in precision.
Revised: This second strategy avoids a subsequent optimization step and enjoys clever encoding schemes with improved token efficiency yet represents values with an inherent loss in precision.

Original: As an alternative, we develop a representation based on a “two-hot” encoding which avoids subsequent optimization steps as well as rounding.
Revised: As an alternative, we develop a representation based on a “two-hot” encoding which avoids subsequent optimization steps as well as rounding.

Original: We note that many previous symbolic regression methods have been described as “discovering natural laws”.
Revised: We note that many previous symbolic regression methods have been described as “discovering natural laws”.

Original: However, most of them learn fixed functional relationships from input-output pairs, whereas we seek to actually infer the underlying dynamic law that governed the behavior of the observed solution trajectory.
Revised: However, most of them learn fixed functional relationships from input-output pairs, whereas we seek to actually infer the underlying dynamic law that governed the behavior of the observed solution trajectory.

Original: A common obstacle to inferring dynamical laws is the lack of temporal derivatives in empirical measurements, which can hence not serve as regression targets.
Revised: A common obstacle to inferring dynamical laws is the lack of temporal derivatives in empirical measurements, which can hence not serve as regression targets.

Original: While derivatives can be estimated numerically, for example via finite differences, this requires dense, regularly sampled observations with a high signal-to-noise ratio to prevent fatal propagation of approximation errors.
Revised: While derivatives can be estimated numerically, for example via finite differences, this requires dense, regularly sampled observations with a high signal-to-noise ratio to prevent fatal propagation of approximation errors.

Original: Alternative loss formulations that bypass unobserved time derivatives have been identified as an open challenge in symbolic regression for dynamical systems recently (Qian et al., 2022).
Revised: Alternative loss formulations that bypass unobserved time derivatives have been identified as an open challenge in symbolic regression for dynamical systems recently (Qian et al., 2022).

Original: With NSODE, we propose a solution that does not rely on temporal derivatives and thus avoids these complications.
Revised: With NSODE, we propose a solution that does not rely on temporal derivatives and thus avoids these complications.

Original:  M ETHOD
Revised:  M ETHOD

Original: Problem setting.
Revised: Problem setting.

Original: Given observations { ( t i , y i ) } ni =1 of a trajectory y :
Revised: Given observations { ( t i , y i ) } ni =1 of a trajectory y :

Original: [ t 1 , t n ] → R that is a solution of the ODE ˙ y =
Revised: [ t 1 , t n ] → R that is a solution of the ODE ˙ y =

Original: f ( y ) , we aim to recover the function f in symbolic form—in our case as a string.
Revised: f ( y ) , we aim to recover the function f in symbolic form—in our case as a string.

Original: In this work, we focus on time-invariant (or autonomous) ODEs (i.e., f ( y, t ) =
Revised: In this work, we focus on time-invariant (or autonomous) ODEs (i.e., f ( y, t ) =

Original: f ( y ) ).
Revised: f ( y ) ).

Original: Such settings are a good starting point for investigation as they are commonly studied and can be thought of as “evolving on their own” without external driving forces or controls, i.e., once an initial condition is fixed the absolute time does not directly influence the evolution.
Revised: Such settings are a good starting point for investigation as they are commonly studied and can be thought of as “evolving on their own” without external driving forces or controls, i.e., once an initial condition is fixed the absolute time does not directly influence the evolution.

Original: We explicitly assume that the observed system actually evolves according to an ODE in canonical form ˙ y =
Revised: We explicitly assume that the observed system actually evolves according to an ODE in canonical form ˙ y =

Original: f ( y ) such that f can be expressed in closed form using the mathematical operators seen during training (see Section 3.1).
Revised: f ( y ) such that f can be expressed in closed form using the mathematical operators seen during training (see Section 3.1).

Original: To sum up, in this work we study the rich class of non-linear, scalar, first-order, autonomous ODEs, even though our approach is not fundamentally limited to this setting.
Revised: To sum up, in this work we study the rich class of non-linear, scalar, first-order, autonomous ODEs, even though our approach is not fundamentally limited to this setting.

Original: In Appendix A, we discuss extensions of NSODE to higher-order systems of coupled non-autonomous ODEs.
Revised: In Appendix A, we discuss extensions of NSODE to higher-order systems of coupled non-autonomous ODEs.

Original: 3. D ATA G ENERATION
Revised: 3. D ATA G ENERATION

Original: Sampling symbolic expressions.
Revised: Sampling symbolic expressions.

Original: To exploit large-scale supervised pretraining we randomly generate a dataset of ∼ 63M ODEs in symbolic form along with their numerical solutions for randomly sampled initial values.
Revised: To exploit large-scale supervised pretraining we randomly generate a dataset of ∼ 63M ODEs in symbolic form along with their numerical solutions for randomly sampled initial values.

Original: Since we assume ODEs to be in canonical form ˙ y = f ( y ) , generating an ODE is equivalent to generating a symbolic expression f ( y ) .
Revised: Since we assume ODEs to be in canonical form ˙ y = f ( y ) , generating an ODE is equivalent to generating a symbolic expression f ( y ) .

Original: We follow Lample & Charton (2019), who sample such an expression f ( y ) as a unary-binary tree, where each internal node corresponds to an operator and each leaf node corresponds to a constant or variable.
Revised: We follow Lample & Charton (2019), who sample such an expression f ( y ) as a unary-binary tree, where each internal node corresponds to an operator and each leaf node corresponds to a constant or variable.

Original: The algorithm consists of two[Table] phases: (1) A unary-binary tree is sampled uniformly from the distribution of unary-binary trees with up to k ∈ N internal nodes, which crucially does not overrepresent small trees corresponding to short expressions.
Revised: The algorithm consists of two phases: (1) A unary-binary tree is sampled uniformly from the distribution of unary-binary trees with up to k ∈ N internal nodes, which crucially does not overrepresent small trees corresponding to short expressions.

Original: Here the maximum number of internal nodes K is a hyperparameter of the algorithm.
Revised: Here the maximum number of internal nodes K is a hyperparameter of the algorithm.

Original: The sampled tree is “decorated”, that is, each binary internal node is assigned a binary operator, each unary internal node is assigned a unary operator, and each leaf is assigned a variable or constant.
Revised: The sampled tree is “decorated”, that is, each binary internal node is assigned a binary operator, each unary internal node is assigned a unary operator, and each leaf is assigned a variable or constant.

Original: Hence, we have to specify a distribution over the N bin binary operators, one over the N una unary operators, a probability p sym ∈ (0 , 1) to decide between symbols and constants, as well as a distribution p c over constants.
Revised: Hence, we have to specify a distribution over the N bin binary operators, one over the N una unary operators, a probability p sym ∈ (0 , 1) to decide between symbols and constants, as well as a distribution p c over constants.

Original: For constants we distinguish explicitly between sampling an integer or a real value.
Revised: For constants we distinguish explicitly between sampling an integer or a real value.

Original: Together with K , these choices uniquely determine a distribution over equations f and are described in detail in Appendix B. Figure 1 depicts an overview of the data generation procedure.
Revised: Together with K , these choices uniquely determine a distribution over equations f and are described in detail in Appendix B. Figure 1 depicts an overview of the data generation procedure.

Original: The pre-order traversal of a sampled tree results in the symbolic expression for f in prefix notation.
Revised: The pre-order traversal of a sampled tree results in the symbolic expression for f in prefix notation.

Original: After conversion to the more common mathematical infix notation, we simplify each expression using the computer algebra system SymPy (Meurer et al., 2017), and filter out constant equations f ( y )
Revised: After conversion to the more common mathematical infix notation, we simplify each expression using the computer algebra system SymPy (Meurer et al., 2017), and filter out constant equations f ( y )

Original: =
Revised: =

Original: c as well as expressions that contain operators or symbols that were not part of the original distribution 1 .
Revised: c as well as expressions that contain operators or symbols that were not part of the original distribution 1 .

Original: We call the structure modulo the value of the constants of such an expression a skeleton .
Revised: We call the structure modulo the value of the constants of such an expression a skeleton .

Original: Any skeleton containing at least one binary operator or constant can be represented by different unary-binary trees.
Revised: Any skeleton containing at least one binary operator or constant can be represented by different unary-binary trees.

Original: Vice versa many of the generated trees will be simplified to the same skeleton.
Revised: Vice versa many of the generated trees will be simplified to the same skeleton.

Original: To ensure diversity and to mitigate potential dataset bias towards particular expressions, we discard duplicates on the skeleton level.
Revised: To ensure diversity and to mitigate potential dataset bias towards particular expressions, we discard duplicates on the skeleton level.

Original: To further cheaply increase the variability of ODEs we sample N const unique sets of constants per skeleton.
Revised: To further cheaply increase the variability of ODEs we sample N const unique sets of constants per skeleton.

Original: When sampling constants we take care not to modify the canonical expression by adhering to the rules listed in Appendix B.1.
Revised: When sampling constants we take care not to modify the canonical expression by adhering to the rules listed in Appendix B.1.

Original: Our final dataset contains linear and non-linear as well as homogeneous and inhomogeneous ODEs and we provide summary statistics about the distribution over equations in Appendix D. Besides the number of internal nodes, we also report a simple yet common measure of complexity for each symbolic equation, which is the overall count of symbols (e.g., y , or constants) as well as operators in an expression.
Revised: Our final dataset contains linear and non-linear as well as homogeneous and inhomogeneous ODEs and we provide summary statistics about the distribution over equations in Appendix D. Besides the number of internal nodes, we also report a simple yet common measure of complexity for each symbolic equation, which is the overall count of symbols (e.g., y , or constants) as well as operators in an expression.

Original: Computing numerical solutions.
Revised: Computing numerical solutions.

Original: We obtain numerical solutions for all ODEs via SciPy’s interface (Virtanen et al., 2020) to the LSODA software package (Hindmarsh & Laboratory, 1982) with both relative and absolute tolerances set to 10 − 9 .
Revised: We obtain numerical solutions for all ODEs via SciPy’s interface (Virtanen et al., 2020) to the LSODA software package (Hindmarsh & Laboratory, 1982) with both relative and absolute tolerances set to 10 − 9 .

Original: LSODA consists of a collection of ODE solvers and implements a strategy to automatically choose an appropriate solver for the problem at hand (e.g., recognizing stiff problems).
Revised: LSODA consists of a collection of ODE solvers and implements a strategy to automatically choose an appropriate solver for the problem at hand (e.g., recognizing stiff problems).

Original: We solve each equation on a fixed time interval t ∈ [0 , T ] and store solutions on a regular grid of N grid points.
Revised: We solve each equation on a fixed time interval t ∈ [0 , T ] and store solutions on a regular grid of N grid points.

Original: For each ODE, we sample up to N iv initial values y (0) =
Revised: For each ODE, we sample up to N iv initial values y (0) =

Original: y 0 uniformly from ( y min0 , y max0 ) .
Revised: y 0 uniformly from ( y min0 , y max0 ) .

Original: 2 While LSODA attempts to select an appropriate solver, numerical solutions still cannot be trusted in all cases.
Revised: 2 While LSODA attempts to select an appropriate solver, numerical solutions still cannot be trusted in all cases.

Original: Therefore, we check the validity of solutions  via the following quality control check: we use 9th order central finite differences to approximate the temporal derivative of the solution trajectory (on the same temporal grid as the proposed solution), denoted by ˙ y fd , and filter out any solution for which ∥ ˙
Revised: Therefore, we check the validity of solutions , which we not [Table] via the following quality control check: we use 9th order central finite differences to approximate the temporal derivative of the solution trajectory (on the same temporal grid as the proposed solution), denoted by ˙ y fd , and filter out any solution for which ∥ ˙

Original: y fd − ˙ y ∥ ∞ > ϵ , where we use ϵ = 1 .
Revised: y fd − ˙ y ∥ ∞ > ϵ , where we use ϵ = 1 .

Original: 3. M ODEL D ESIGN C HOICES
Revised: 3. M ODEL D ESIGN C HOICES

Original: NSODE consists of an encoder-decoder transformer with architecture choices listed in Table 1.
Revised: NSODE consists of an encoder-decoder transformer with architecture choices listed in Table 1.

Original: We provide a visual overview in Figure 1.
Revised: We provide a visual overview in Figure 1.

Original: samples.

Original: Representing input trajectories.
Revised: Representing input trajectories.

Original: A key difficulty in feeding numerical solutions { y i } ni =1 as input is that their range may differ greatly both within a single solution as well as across ODEs.
Revised: A key difficulty in feeding numerical solutions { y i } ni =1 as input is that their range may differ greatly both within a single solution as well as across ODEs.

Original: For example, the linear ODE ˙ y = c · y for a constant c is solved by an exponential y ( t ) =
Revised: For example, the linear ODE ˙ y = c · y for a constant c is solved by an exponential y ( t ) =

Original: y 0 exp( ct ) for initial value y (0) =
Revised: y 0 exp( ct ) for initial value y (0) =

Original: y 0 , which may span many orders of magnitude on a fixed time interval.
Revised: y 0 , which may span many orders of magnitude on a fixed time interval.

Original: To prevent numerical errors and vanishing or exploding gradients caused by the large range of values, we assume each representable 64-bit float value is a token and use its IEEE-754 encoding as the token representation (Biggio et al., 2021).
Revised: To prevent numerical errors and vanishing or exploding gradients caused by the large range of values, we assume each representable 64-bit float value is a token and use its IEEE-754 encoding as the token representation (Biggio et al., 2021).

Original: We thus convert all pairs ( t i , y i ) to their IEEE-754 64 bit representations, channel them through a linear embedding layer, and then feed them to the encoder.
Revised: We thus convert all pairs ( t i , y i ) to their IEEE-754 64 bit representations, channel them through a linear embedding layer, and then feed them to the encoder.

Original: Representing symbolic expressions.
Revised: Representing symbolic expressions.

Original: The target sequence (i.e., the string for the symbolic expression of f ) is tokenized on the symbol-level.
Revised: The target sequence (i.e., the string for the symbolic expression of f ) is tokenized on the symbol-level.

Original: We distinguish two cases: (1) Operators and variables: for each operator and variable we include a unique token in the vocabulary.
Revised: We distinguish two cases: (1) Operators and variables: for each operator and variable we include a unique token in the vocabulary.

Original: These tokens are one-hot encoded and passed through a learnable embedding layer before their embedded representations are fed to the decoder.(2) Numerical constants: constants may come from both discrete (integers) as well as continuous distributions, as for example in y**2+1.64*cos(y) .
Revised: These tokens are one-hot encoded and passed through a learnable embedding layer before their embedded representations are fed to the decoder.(2) Numerical constants: constants may come from both discrete (integers) as well as continuous distributions, as for example in y**2+1.64*cos(y) .

Original: Hence, it is unfeasible to include individual tokens “for each constant”.
Revised: Hence, it is unfeasible to include individual tokens “for each constant”.

Original: Naively tokenizing on the digit level, i.e., representing real values literally as the sequence of characters (e.g., "1.64" ), not only significantly expands the length of target sequences and thus the computational cost, but also requires a variable number of prediction steps for every single constant.
Revised: Naively tokenizing on the digit level, i.e., representing real values literally as the sequence of characters (e.g., "1.64" ), not only significantly expands the length of target sequences and thus the computational cost, but also requires a variable number of prediction steps for every single constant.

Original: Instead, we take inspiration from Schrittwieser et al.
Revised: Instead, we take inspiration from Schrittwieser et al.

Original: (2020) and encode constants in a two-hot fashion.
Revised: (2020) and encode constants in a two-hot fashion.

Original: We fix a finite homogeneous grid on the real numbers x 1 < x 2 < . . .
Revised: We fix a finite homogeneous grid on the real numbers x 1 < x 2 < . . .

Original: < x m for some m ∈ N , which we add as tokens to the vocabulary.
Revised: < x m for some m ∈ N , which we add as tokens to the vocabulary.

Original: The grid range ( x 1 , x m ) and number of grid points m can be tuned for performance.
Revised: The grid range ( x 1 , x m ) and number of grid points m can be tuned for performance.

Original: Our choices are described in Appendix B.3.
Revised: Our choices are described in Appendix B.3.

Original: For any constant c in the target sequence we then find i ∈ { 1 , . . .
Revised: For any constant c in the target sequence we then find i ∈ { 1 , . . .

Original: , m − 1 } such that x i ≤ c < x i +1 and encode c as a distribution supported on x i , x i +1 with weights α, β such that αx i + βx
Revised: , m − 1 } such that x i ≤ c < x i +1 and encode c as a distribution supported on x i , x i +1 with weights α, β such that αx i + βx

Original: i +

Revised: i +

Original: =
Revised: =

Original: c .
Revised: c .

Original: That is, the target in the cross-entropy loss for a constant token is not a strict one-hot encoding, but a distribution supported on two (neighboring) vocabulary tokens resulting in a lossless encoding of continuous values in [ x 1 , x m ] .
Revised: That is, the target in the cross-entropy loss for a constant token is not a strict one-hot encoding, but a distribution supported on two (neighboring) vocabulary tokens resulting in a lossless encoding of continuous values in [ x 1 , x m ] .

Original: While this two-hot representation can be used directly in the cross-entropy loss function and thus greatly facilitates training, it can not be passed directly through an embedding layer.
Revised: While this two-hot representation can be used directly in the cross-entropy loss function and thus greatly facilitates training, it can not be passed directly through an embedding layer.

Original: For a generic constant in the target sequence represented as αx i + βx
Revised: For a generic constant in the target sequence represented as αx i + βx

Original: i +1 , we thus instead provide the linear combination of the two embeddings α embed(x i )
Revised: i +1 , we thus instead provide the linear combination of the two embeddings α embed(x i )

Original: +
Revised: +

Original: β embed(x i+1 ) as decoder input.
Revised: β embed(x i+1 ) as decoder input.

Original: Decoding constants.
Revised: Decoding constants.

Original: When decoding a predicted sequence, we check at each prediction step whether the arg max of the logits corresponds to one of the m constant tokens { x 1 , . .
Revised: When decoding a predicted sequence, we check at each prediction step whether the arg max of the logits corresponds to one of the m constant tokens { x 1 , . .

Original: , x m } .
Revised: , x m } .

Original: If not, we proceed by conventional 1-hot decoding to obtain predicted operators and variables.
Revised: If not, we proceed by conventional 1-hot decoding to obtain predicted operators and variables.

Original: If instead the argmax corresponds to, for example, x i , we also pick its largest-logit neighbor ( x i − 1 or x i +1 ; suppose x i +1 ), renormalize their probabilities by applying a softmax to all logits and use the resulting two probability estimates as weights α, β .
Revised: If instead the argmax corresponds to, for example, x i , we also pick its largest-logit neighbor ( x i − 1 or x i +1 ; suppose x i +1 ), renormalize their probabilities by applying a softmax to all logits and use the resulting two probability estimates as weights α, β .

Original: Constants are then ultimately decoded as αx i + βx i +1 .
Revised: Constants are then ultimately decoded as αx i + βx i +1 .

Original: 3. E VALUATION AND M ETRICS
Revised: 3. E VALUATION AND M ETRICS

Original: Sampling solutions.
Revised: Sampling solutions.

Original: To infer a symbolic expression for the governing ODE of a new observed solution trajectory { ( t i , y i ) } ni =1 , all the typical policies such as greedy, sampling, or beam search are available.
Revised: To infer a symbolic expression for the governing ODE of a new observed solution trajectory { ( t i , y i ) } ni =1 , all the typical policies such as greedy, sampling, or beam search are available.

Original: In our evaluation, we leverage computationally cheap forward passes to perform beam search with 1536 beams and report top-
Revised: In our evaluation, we leverage computationally cheap forward passes to perform beam search with 1536 beams and report top-

Original: k results with k ranging from 1 to 1536.
Revised: k results with k ranging from 1 to 1536.

Original: To pick from the candidate expressions provided by the beam search, one could for example only provide a fraction of the trajectory as input to the model (say the first half) and then pick the candidate ODE whose numerical solution best predicts the trajectory on the remaining observation.
Revised: To pick from the candidate expressions provided by the beam search, one could for example only provide a fraction of the trajectory as input to the model (say the first half) and then pick the candidate ODE whose numerical solution best predicts the trajectory on the remaining observation.

Original: Metrics.
Revised: Metrics.

Original: We evaluate model performance both numerically and symbolically.
Revised: We evaluate model performance both numerically and symbolically.

Original: For numerical evaluation we follow Biggio et al.
Revised: For numerical evaluation we follow Biggio et al.

Original: (2020): suppose the ground truth ODE is given by ˙ y
Revised: (2020): suppose the ground truth ODE is given by ˙ y

Original: = f ( y ) with (numerical) solution y ( t ) and the predicted ODE is given by ˆ˙ y
Revised: = f ( y ) with (numerical) solution y ( t ) and the predicted ODE is given by ˆ˙ y

Original: =
Revised: =

Original: ˆ f ( y ) .
Revised: ˆ f ( y ) .

Original: To compute numerical accuracy we first evaluate f and ˆ f on N eval points in the interval [min( y ( t )) , max( y ( t ))]
Revised: To compute numerical accuracy we first evaluate f and ˆ f on N eval points in the interval [min( y ( t )) , max( y ( t ))]

Original: (i.e., the interval traced out by the observed solution), which yields function evaluations gt = { ˙ y i } N eval i = and pred = { ˆ˙ y i } N eval
Revised: (i.e., the interval traced out by the observed solution), which yields function evaluations gt = { ˙ y i } N eval i = and pred = { ˆ˙ y i } N eval

Original: i
Revised: i

Original: =1 .
Revised: =1 .

Original: We then assess whether numpy.allclose 3 returns True as well as whether the coefficient of determination R 2 ≥ 0 .
Revised: We then assess whether numpy.allclose 3 returns True as well as whether the coefficient of determination R 2 ≥ 0 .

Original: 4 Numerical evaluations capture how closely the predicted function approximates the ground truth function within the interval [min( y ( t )) , max( y ( t ))] .
Revised: 4 Numerical evaluations capture how closely the predicted function approximates the ground truth function within the interval [min( y ( t )) , max( y ( t ))] .

Original: However, a key motivation for symbolic regression is to uncover a symbolic mathematical expression that governs the observations.
Revised: However, a key motivation for symbolic regression is to uncover a symbolic mathematical expression that governs the observations.

Original: Perhaps surprisingly, previous works on symbolic regression have paid little attention to directly testing for symbolic equivalence, which need not be implied by numerical fit.
Revised: Perhaps surprisingly, previous works on symbolic regression have paid little attention to directly testing for symbolic equivalence, which need not be implied by numerical fit.

Original: Testing for symbolic equivalence between ground truth expression f ( y ) and a predicted expression ˆ f ( y ) is unsuitable in the presence of real-valued constants as even minor deviations between true and predicted constants render the equivalence false.
Revised: Testing for symbolic equivalence between ground truth expression f ( y ) and a predicted expression ˆ f ( y ) is unsuitable in the presence of real-valued constants as even minor deviations between true and predicted constants render the equivalence false.

Original: Instead, we regard the predicted expression ˆ f ( y ) to be symbolically correct if f ( y ) and ˆ f ( y ) can be made equivalent by modifying only the values of constants in the predicted expression ˆ f ( y ) .
Revised: Instead, we regard the predicted expression ˆ f ( y ) to be symbolically correct if f ( y ) and ˆ f ( y ) can be made equivalent by modifying only the values of constants in the predicted expression ˆ f ( y ) .

Original: This is implemented using SymPy’s match function.
Revised: This is implemented using SymPy’s match function.

Original: In order not to alter the structure of the predicted expression, we constrain modifications of constants such that all constants remain non-zero and retain their original sign.
Revised: In order not to alter the structure of the predicted expression, we constrain modifications of constants such that all constants remain non-zero and retain their original sign.

Original: This definition is thus primarily concerned with the structure of an expression, rather than precise numerical agreement.
Revised: This definition is thus primarily concerned with the structure of an expression, rather than precise numerical agreement.

Original: Once the structure is known, the inference problem becomes conventional parameter estimation.
Revised: Once the structure is known, the inference problem becomes conventional parameter estimation.

Original: We report percentages of samples in a given test set that satisfies any individual metric (numerical and symbolic), as well as percentages satisfying symbolic and numerical metrics simultaneously.
Revised: We report percentages of samples in a given test set that satisfies any individual metric (numerical and symbolic), as well as percentages satisfying symbolic and numerical metrics simultaneously.

Original:  E XPERIMENTS 4. B ENCHMARK D ATASETS
Revised:  E XPERIMENTS 4. B ENCHMARK D ATASETS

Original: We construct several test sets to evaluate model performance and generalization in different settings.
Revised: We construct several test sets to evaluate model performance and generalization in different settings.

Original: • testset-iv : Our first test set assesses generalization within initial values not seen during training.
Revised: • testset-iv : Our first test set assesses generalization within initial values not seen during training.

Original: It consists of 5793 ODEs picked uniformly at random from our generated dataset but re-sampled initial values.
Revised: It consists of 5793 ODEs picked uniformly at random from our generated dataset but re-sampled initial values.

Original: We also employ the following constraints via rejection sampling: (a) All skeletons in testset-iv are unique.
Revised: We also employ the following constraints via rejection sampling: (a) All skeletons in testset-iv are unique.

Original: (b) As the number of unique skeletons increases with the number of operators, we allow at most 2000 examples per number of operators (with substantially fewer unique skeletons existing for few operators).
Revised: (b) As the number of unique skeletons increases with the number of operators, we allow at most 2000 examples per number of operators (with substantially fewer unique skeletons existing for few operators).

Original: • testset-constants : Our second test set assesses generalization within unseen initial values and constants.
Revised: • testset-constants : Our second test set assesses generalization within unseen initial values and constants.

Original: It consists of 2720 ODEs picked uniformly at random from our dataset (ensuring unique skeletons and at most 1000 examples per number of operators as above), but re-sampled intial values and constants.
Revised: It consists of 2720 ODEs picked uniformly at random from our dataset (ensuring unique skeletons and at most 1000 examples per number of operators as above), but re-sampled intial values and constants.

Original: • testset-skeletons : In principle, we can train NSODE on all possible expressions (using only the specified operators and number ranges) up to a specified number of operators.
Revised: • testset-skeletons : In principle, we can train NSODE on all possible expressions (using only the specified operators and number ranges) up to a specified number of operators.

Original: However, even with the millions examples in our dataset, we have by far not exhausted the huge space of possible  skeletons (especially for larger numbers of operators).
Revised: However, even with the millions examples in our dataset, we have by far not exhausted the huge space of possible [Figure] skeletons (especially for larger numbers of operators).

Original: Hence, our third test set contains 100 novel random ODEs with skeletons that were never seen during training.
Revised: Hence, our third test set contains 100 novel random ODEs with skeletons that were never seen during training.

Original: • testset-iv-163 : This is a subset of testset-iv motivated by the fact that most symbolic regression models we want to compare to require a separate optimization for every individual example, which was computationally infeasible for our testset-iv.
Revised: • testset-iv-163 : This is a subset of testset-iv motivated by the fact that most symbolic regression models we want to compare to require a separate optimization for every individual example, which was computationally infeasible for our testset-iv.

Original: For a fair comparison, we therefore subsampled up to 10 ODEs per complexity uniformly at random, yielding 163 examples.
Revised: For a fair comparison, we therefore subsampled up to 10 ODEs per complexity uniformly at random, yielding 163 examples.

Original: • testset-textbook : To assess how NSODE performs on “real problems”, we manually curatedscalar, non-linear ODEs from Wikipedia pages, physics textbooks, and lecture note from university courses on ODEs.
Revised: • testset-textbook : To assess how NSODE performs on “real problems”, we manually curatedscalar, non-linear ODEs from Wikipedia pages, physics textbooks, and lecture note from university courses on ODEs.

Original: These equations are listed in Table 7 in Appendix C.
Revised: These equations are listed in Table 7 in Appendix C.

Original: We note that they are all extremely simple compared to the expressions in our generated dataset in that they are ultimately mostly low order polynomials, some of which with one fractional exponent.
Revised: We note that they are all extremely simple compared to the expressions in our generated dataset in that they are ultimately mostly low order polynomials, some of which with one fractional exponent.

Original: • testset-classic : To validate our approach on existing datasets we turn to benchmarks in the classic symbolic regression literature (inferring just the functional relationship between input-ouput pairs) and simply interpret functions as ODEs.
Revised: • testset-classic : To validate our approach on existing datasets we turn to benchmarks in the classic symbolic regression literature (inferring just the functional relationship between input-ouput pairs) and simply interpret functions as ODEs.

Original: In particular we include all scalar function listed in the overview in McDermott et al.
Revised: In particular we include all scalar function listed in the overview in McDermott et al.

Original: (2012) which includes equations from many different benchmarks Keijzer (2003); Koza (1993; 1994); Uy et al.
Revised: (2012) which includes equations from many different benchmarks Keijzer (2003); Koza (1993; 1994); Uy et al.

Original: ; Vladislavleva et al.
Revised: ; Vladislavleva et al.

Original: For example, we interpret the function f
Revised: For example, we interpret the function f

Original: ( y ) =
Revised: ( y ) =

Original: y 

Revised: y

Original: + y 2 + y from Uy et al.
Revised: + y 2 + y from Uy et al.

Original: (2011) as an autonomous ODE ˙ y ( t ) =
Revised: (2011) as an autonomous ODE ˙ y ( t ) =

Original: f ( y ( t ))= y ( t )
Revised: f ( y ( t ))

Original: +
Revised: =

Original: y ( t )  2 + y , which we solve numerically for a randomly sampled initial value as described before.
Revised: y ( t ) 3 + y ( t ) 2 + y , which we solve numerically for a randomly sampled initial value as described before.

Original: [-testset-iv testset-constants testset-skeletons 222222    % allclose 
-]
Revised: [-testset-iv testset-constants testset-skeletons 222222    % allclose 
-]

Original: R 2 0.

Original: 222222topk    %

Original: [Figure]

Original: recovery 222222topk    skel. recov.

Original: & allclose

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare our method to recent popular baselines from the literature (see Section 2).
Revised: We compare our method to recent popular baselines from the literature (see Section 2).

Original: We briefly describe them including some limitations here and defer all details to Appendix E. First, no baseline is suited directly to infer dynamical laws, but only to infer functional relationships. Therefore, all baselines fit a separate regression function mapping y ( t ) (cid:55)→ ˆ˙ y ( t ) per individual ODE, using the coefficient of determination R 2 as optimization objective.
Revised: We briefly describe them including some limitations here and defer all details to Appendix E. All baselines work by explicitly fitting a separate regression function per individual ODE to map the observed samples of the solution trajectory to their temporal derivatives y ( t ) (cid:55)→ ˆ ˙ y ( t ) , using the coefficient of determination R 2 as optimization objective.

Original: Since derivatives ˆ˙ y ( t ) are typically not observed, we approximate them via finite differences using  PySindy (de Silva et al., 2020).
Revised: Since derivatives ˆ˙ y ( t ) are typically not observed, we approximate them via finite differences using the implementation available in PySindy (de Silva et al., 2020).

Original: Hence, all these methods crucially rely on regularly sampled and noise-free observations, whereas our approach can easily be extended to take those into account (see Appendix A).
Revised: Hence, all these methods crucially rely on regularly sampled and noise-free observations, whereas our approach can easily be extended to take those into account (see Appendix A).

Original: • Sindy (Brunton et al., 2016):
Revised: • Sindy (Brunton et al., 2016):

Original: Sindy builds a (sparse) linear combination of a fixed set of (non-linear) basis functions.
Revised: Sindy builds a (sparse) linear combination of a fixed set of (non-linear) basis functions.

Original: The resulting Lasso regression is efficient, but suffers from limited expressiveness. In particular, Sindy cannot easly represent nestedfunctions or non-integer powers as all non-linear expressions have to beadded explicitly  to the set of basis functions.
Revised: The resulting Lasso regression is efficient however complex expressions such as nested or parameterized functions can only be approximated but not easily be represented in their precise symbolic form as all non-linear expressions have to be explicitly added to the set of basis functions.

Original: We cross-validate Sindy over a fairly extensive hyperparameter grid of 800 different combinations for each individual trajectory.
Revised: We cross-validate Sindy over a fairly extensive hyperparameter grid of 800 different combinations for each individual trajectory.

Original: • GPL 5 (genetic programming): GPL(earn) maintains a population of programs each representing a mathematical expression.
Revised: • GPL 5 (genetic programming): GPL(earn) maintains a population of programs each representing a mathematical expression.

Original: The programs are mutated for several generations to heuristically optimize a user defined fitness function.
Revised: The programs are mutated for several generations to heuristically optimize a user defined fitness function.

Original: While not originally developed for ODEs, we can apply GPLearn on our datasets by leveraging the finite difference approximation.
Revised: While not originally developed for ODEs, we can apply GPLearn on our datasets by leveraging the finite difference approximation.

Original: We use a population  size of 1000 and report the best performance across all final programs.
Revised: We use a population [Table] size of 1000 and report the best performance across all final programs.

Original: Compared to sindy, GPLearn is more expressive yet much slower to fit.
Revised: Compared to sindy, GPLearn is more expressive yet much slower to fit.

Original: • AIFeynman (Udrescu & Tegmark, 2020; Udrescu et al., 2020): AIFeynman is a physics-inspired approach to symbolic regression that exploits the insight that many famous equations in natural sciences exhibit well-understood functional properties such as symmetries, compositionality, or smoothness.
Revised: • AIFeynman Udrescu & Tegmark, 2020; Udrescu et al., 2020): AIFeynman is a physics-inspired approach to symbolic regression that exploits the insight that many famous equations in natural sciences exhibit well-understood functional properties such as symmetries, compositionality, or smoothness.

Original: AIFeynman implements a neural network based heuristic search that tests for such properties in order to identify a symbolic expression that fits the data.
Revised: AIFeynman implements a neural network based heuristic search that tests for such properties in order to identify a symbolic expression that fits the data.

Original: For every test sample AIFeynman computes a pareto front of solutions that trade off complexity versus accuracy.
Revised: For every test sample AIFeynman computes a pareto front of solutions that trade off complexity versus accuracy.

Original: We report the best performance across all functions on the pareto front.
Revised: We report the best performance across all functions on the pareto front.

Original: Notably, AIFeynman performs quite an exhaustive search procedure such that running it even on a single equation took on the order of tens of minutes.[Table] [Table]
Revised: Notably, AIFeynman performs quite an exhaustive search procedure such that running it even on a single equation took on the order of tens of minutes.

Original:  R ESULTS
Revised: 4. R ESULTS

Original: Model Performance.
Revised: Model Performance.

Original: Figure 2 shows NSODE’s performance on our testset-iv, testset-constants, and testset-skeletons according to our numerical and symbolic metrics as well as combined skeleton recovery and allclose as we vary k in the top-k considered candidates of the beam search.
Revised: Figure 2 shows NSODE’s performance on our testset-iv, testset-constants, and testset-skeletons according to our numerical and symbolic metrics as well as combined skeleton recovery and allclose as we vary k in the top-k considered candidates of the beam search.

Original: Investing sufficient test-time-compute (i.e., considering many candidates) continuously improves performance.
Revised: Investing sufficient test-time-compute (i.e., considering many candidates) continuously improves performance.

Original: While we capped k at 1536 due to memory limitations, we did not observe a stagnation of the roughly logarithmic scaling of all performance metrics with k .
Revised: While we capped k at 1536 due to memory limitations, we did not observe a stagnation of the roughly logarithmic scaling of all performance metrics with k .

Original: This cannot be attributed to “exhaustion effects”, where one may assume that all possible ODEs will eventually be among the candidates, because (a) the space of possible skeletons grows much faster than exponentially, and (b) the numerical metrics are extremely sensitive also to the predicted constant values in continuous domains. 
Revised: This cannot be attributed to “exhaustion effects”, where one may assume that all possible ODEs will eventually be among the candidates, because (a) the space of possible skeletons grows much faster than exponentially, and (b) the numerical metrics are extremely sensitive also to the predicted constant values in continuous domains. We verified this in an ablation experiment in Appendix F.

Original: As one may expect, performance decreases as we move from only new initial conditions, to also sampling new constants, to finally sampling entirely unseen skeletons.
Revised: As one may expect, performance decreases as we move from only new initial conditions, to also sampling new constants, to finally sampling entirely unseen skeletons.

Original: On testset-iv with k =we achieve about 50% skeleton recovery and still successfully recover more than a third skeletons of testset-skeletons with similar numbers for allclose.
Revised: On testset-iv with k =we achieve about 50% skeleton recovery and still successfully recover more than a third skeletons of testset-skeletons with similar numbers for allclose.

Original: The fact that the combined metric (symbolic + numerical) is only about half of that indicates that numerical and symbolic fit are indeed two separate measures, none of which need to imply the other.
Revised: The fact that the combined metric (symbolic + numerical) is only about half of that indicates that numerical and symbolic fit are indeed two separate measures, none of which need to imply the other.

Original: Hence, a thorough evaluation of both is crucial to understand model performance in symbolic regression tasks.
Revised: Hence, a thorough evaluation of both is crucial to understand model performance in symbolic regression tasks.

Original: Comparison to baselines.
Revised: Comparison to baselines

Original: In Table 2 we compare NSODE to all baselines (see Section 4.2) on all our metrics (see Section 3.3) on all test sets (see Section 4.1) using k = 1536 in our beam search.
Revised: In Table 2 we compare NSODE to all baselines (see Section 4.2) on all our metrics (see Section 3.3) on all test sets (see Section 4.1) using k = 1536 in our beam search.

Original: We also include the average wallclock runtime per expression for each of the datasets.
Revised: We also include the average wallclock runtime per expression for each of the datasets.

Original: Since GPLearn  often generates extremely long string expressions, it often takes SymPy up to half a minute to parse these expressions.
Revised: Since GPLearn [Figure] often generates extremely long string expressions, it often takes SymPy up to half a minute to parse these expressions.

Original: We denote this extra time in grayin the table and provide a more detailed version in Table 8.
Revised: We denote this extra time in gray and provide a more detailed version in Table 9.

Original: First, we note that on our subsampled testset-iv-163, NSODE outperforms competing approaches in terms of skeleton recovery by a wide margin and also performs best in terms of joint skeleton recovery and numerical measures, which is a strong indication of actually having recovered the governing
Revised: First, we note that on our subsampled testset-iv-163, NSODE outperforms competing approaches in terms of skeleton recovery by a wide margin and also performs best in terms of joint skeleton recovery and numerical measures, which is a strong indication of actually having recovered the governing

Original: ODE accurately.
Revised: ODE accurately.

Original: By spending over 200x more time on its exhaustive heuristic search, AIFeynman manages to outperform NSODE in terms of numerical accuracy ( R 2 and allclose), which again does not necessarily correspond to correct identification of the ODE structure.
Revised: By spending over 200x more time on its exhaustive heuristic search, AIFeynman manages to outperform NSODE in terms of numerical accuracy ( R 2 and allclose), which again does not necessarily correspond to correct identification of the ODE structure.

Original: Figure 3 shows the number of skeletons recovered by each method given the complexity of equations, results for dataset-classic and dataset-textbook can be found in Appendix F. 6 While AIFeynman and Sindy recover some of the [Figure] AIFeynman [Figure]
Revised: Figure 3 shows the number of skeletons recovered by each method given the complexity of equations, results for dataset-classic and dataset-textbook can be found in Appendix G. 6 While AIFeynman and Sindy recover some of the low complexity expressions, NSODE is the only method to also recover some of the more complex skeletons.

Original: complexity

Original: [Figure]

Original: complexity

Original: [Figure]

Original: [Figure]

Original: un t

Original: Sindy

Original: GPLearn

Original: [Figure]

Original: complexity

Original: [Figure]

Original: un t low complexity expressions, NSODE is the only method to also recover some of the more complex skeletons.

Original: This also explains the strong performance on AIFeynman on the overwhelmingly simple expressions in the testset-classic and testset-textbook.
Revised: This also explains the strong performance on AIFeynman on the overwhelmingly simple expressions in the testset-classic and testset-textbook.

Original: The heuristics deployed by AIFeynman are tuned to incorporate properties of known equations in the sciences.
Revised: The heuristics deployed by AIFeynman are tuned to incorporate properties of known equations in the sciences.

Original: In particular, the algorithm explicitly attempts to fit a polynomial to the data which immediately corresponds to the correct ground truth expression in 8 / 12 ≈ 66 .
Revised: In particular, the algorithm explicitly attempts to fit a polynomial to the data which immediately corresponds to the correct ground truth expression in 8 / 12 ≈ 66 .

Original: 6% equations in testset-textbook and 12 / 26 ≈ 46 .
Revised: 6% equations in testset-textbook and 12 / 26 ≈ 46 .

Original: 2% equations in testsettextbook.
Revised: 2% equations in testsettextbook.

Original: Together with the fact that it is solving a somewhat simpler task (identifying functional relationships with manually provided derivatives from noise-free regularly sampled time-series), its strong performance on these particular test cases is to be expected.
Revised: Together with the fact that it is solving a somewhat simpler task (identifying functional relationships with manually provided derivatives from noise-free regularly sampled time-series), its strong performance on these particular test cases is to be expected.

Original: However, even on these simple examples AIFeynman takes over 200x longer than our method, which in turn clearly outperforms
Revised: However, even on these simple examples AIFeynman takes over 200x longer than our method, which in turn clearly outperforms

Original: Sindy and GPLearn in terms of skeleton recovery.
Revised: Sindy and GPLearn in terms of skeleton recovery.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have developed a flexible and scalable method to infer ordinary differential equations ˙ y
Revised: We have developed a flexible and scalable method to infer ordinary differential equations ˙ y

Original: = f ( y ) from a single observed solution trajectory.
Revised: = f ( y ) from a single observed solution trajectory.

Original: Our method follows the successful paradigm of large-scale pretraining of attention-based sequence-to-sequence models on essentially unlimited amounts of simulated data, where the inputs are the observed solution { ( t i , y i ) } ni =1 and the output is a symbolic expression for f as a string.
Revised: Our method follows the successful paradigm of large-scale pretraining of attention-based sequence-to-sequence models on essentially unlimited amounts of simulated data, where the inputs are the observed solution { ( t i , y i ) } ni =1 and the output is a symbolic expression for f as a string.

Original: Once trained, our method is orders of magnitude more efficient than similarly expressive existing symbolic regression techniques that require a separate optimization for each instance and achieves strong performance in terms of skeleton recovery especially for complex expressions on various benchmarks.
Revised: Once trained, our method is orders of magnitude more efficient than similarly expressive existing symbolic regression techniques that require a separate optimization for each instance and achieves strong performance in terms of skeleton recovery especially for complex expressions on various benchmarks.

Original: In addition to all code (including baselines) and final model checkpoints, we open-source a dataset of ODEs together with numerical solutions.

Original: This dataset is unprecedented in size and diversity.

Original: Current limitations of our method mostly concern extensions to systems of high-order nonautonomous equations f , which we discuss in detail in Appendix A. While these extensions are possible in principle with few to no modifications of NSODE, it is an interesting direction for future to assess empirically how far this approach can be pushed.
Revised: Current limitations of our method mostly concern extensions to systems of high-order non-autonomous equations f , which we discuss in detail in Appendix A. While these extensions are possible in principle with few to no modifications of NSODE, it is an interesting direction for future to assess empirically how far this approach can be pushed.

Original: Besides the potential usefulness of our model in real-world scientific discovery and hypothesis generation, we also hope that our released dataset, code, and checkpoints will serve as a useful starting point for further research in this direction.
Revised: Besides the potential usefulness of our model in real-world scientific discovery and hypothesis generation, we also hope that our released dataset, code, and checkpoints will serve as a useful starting point for further research in this direction.

Original:  E THICS STATEMENT
Revised:  E THICS STATEMENT

Original: This paper addresses fundamental research and as such its implications are difficult to forsee.
Revised: This paper addresses fundamental research and as such its implications are difficult to forsee.

Original: However, all of our experiments are based on non-personalized data so that we believe that datasets and model checkpoints can be publicly released without compromising private information.
Revised: However, all of our experiments are based on non-personalized data so that we believe that datasets and model checkpoints can be publicly released without compromising private information.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: To facilitate reproducibility we will release all datasets, source code to our model and baseline implementations as well as our final model checkpoint.
Revised: To facilitate reproducibility we will release all datasets, source code to our model and baseline implementations as well as our final model checkpoint.

Original: We also report all hyperparameter choices for our model in Section 3.2 and Appendix B.3 as well as for the baseline methods Appendix E.
Revised: We also report all hyperparameter choices for our model in Section 3.2 and Appendix B 3 as well as for the baseline methods Appendix E.

Original: Hyper-parameters for data generation are described in Section 3.1 and concrete choices are listed in
Revised: Hyper-parameters for data generation are described in Section 3.1 and concrete choices are listed in

Original: Appendix B.2.
Revised: Appendix B.2.

Original: A P OSSIBLE EXTENSIONS
Revised: A P OSSIBLE EXTENSIONS

Original: While we have focused exclusively on the huge class of scalar, autonomous, first-order ODEs, we believe that our approach can scale also to non-autonomous, higher-order, systems of ODEs.
Revised: While we have focused exclusively on the huge class of explicit scalar autonomous first-order ODEs we believe that our approach can scale also to non-autonomous, implicit, higher-order systems of ODEs.

Original: Non-autonomous equations.
Revised: Non-autonomous equations.

Original: Since our model is provided with time t as inputs, it is capable of learning functions f
Revised: Since our model is provided with time t as inputs, it is capable of learning functions f

Original: ( y, t ) depending on y and t explicitly.
Revised: ( y, t ) depending on y and t explicitly.

Original: Hence, extending our approach to non-autonomous ODEs is simply a matter of adding a symbol for t in our data generation.
Revised: Hence, extending our approach to non-autonomous ODEs is simply a matter of adding a symbol for t in our data generation.

Original: Irregular samples and noise.
Revised: Implicit equations. A generalization from purely explicit ODEs to all ODEs of the form f ( ˙ y, y ) =are possible but rely on the availability of appropriate ODE solvers for implicit equations. Sampling f of this general form using the sampling algorithm described in Section 3.1 is straightforward as we only need to include the additional variable ˙

Revised: y in the set of candidate symbols and its associated distribution p sym .

Revised: If we have an appropriate implicit numerical solver at hand, we can supply the numerical solution trajectory to NSODE as before and use f for teacher forcing during training.

Original:  Due to the separate data generation and training phase, it is straight forward to train NSODE on corrupted input sequences, where we could for example add observation noise (need not be additive) to the { y i } ni =1 or randomly drop some of the observations on the regular observation grid to simulate irregularly sampled observations.
Revised: Irregular samples and noise. Due to the separate data generation and training phase, it is straight forward to train NSODE on corrupted input sequences, where we could for example add observation noise (need not be additive) to the { y i } ni =1 or randomly drop some of the observations on the regular observation grid to simulate irregularly sampled observations.

Original: We expect these models to still perform well and in particular much better than the baselines, since their targets are approximated derivatives, which are highly sensitive to noise and irregular samples.
Revised: We expect these models to still perform well and in particular much better than the baselines, since their targets are approximated derivatives, which are highly sensitive to noise and irregular samples.

Original: Systems of equations.
Revised: Systems of equations.

Original: For a system of K first-order ODEs, the data generation requires two updates: first, we have to generate K equations ( f k ) k ∈{ 1 ,...,K } , one for each component.
Revised: For a system of K first-order ODEs, the data generation requires two updates: first, we have to generate K equations ( f k ) k ∈{ 1 ,...,K } , one for each component.

Original: In addition, each of those functions can depend not only on y (and t in the non-autonomous case), but K different components { y ( k ) } k ∈{ 1 ,...,K } .
Revised: In addition, each of those functions can depend not only on y (and t in the non-autonomous case), but K different components { y ( k ) } k ∈{ 1 ,...,K } .

Original: This is easily achieved by allowing for K tokens y ( k ) in the data generation.
Revised: This is easily achieved by allowing for K tokens y ( k ) in the data generation.

Original: We can then simply augment the input sequence to the transformer to contain not only ( t, y ) as for the scalar case, but ( t, y (1) , y (2) , . . .
Revised: We can then simply augment the input sequence to the transformer to contain not only ( t, y ) as for the scalar case, but ( t, y (1) , y (2) , . . .

Original: , y ( K ) ) for a system of K variables.
Revised: , y ( K ) ) for a system of K variables.

Original: Finally, we have decide on a way to set the target for the transformer, i.e., how to represent the system of equations in symbolic form as a single sequence .
Revised: Finally, we have decide on a way to set the target for the transformer, i.e., how to represent the system of equations in symbolic form as a single sequence .

Original: A straight forward way to achieve this is to introduce a special delimitation character, e.g., "|" to separate the different components.
Revised: A straight forward way to achieve this is to introduce a special delimitation character, e.g., "|" to separate the different components.

Original: Note that the order in which to predict the K equations is dictated by the order in which they are stacked for the input sequence, hence this information is in principle available to the model.
Revised: Note that the order in which to predict the K equations is dictated by the order in which they are stacked for the input sequence, hence this information is in principle available to the model.

Original: For example, in the case of a two-dimensional system, where y =
Revised: For example, in the case of a two-dimensional system, where y =

Original: ( y (1) , y (2) )
Revised: ( y (1) , y (2) )

Original: ∈ R 2 , we could have the model perform the following mapping
Revised: ∈ R 2 , we could have the model perform the following mapping

Original: [Equation]
Revised: [Equation]

Original:  where the separator | delimits the two components of f ( y, t ) corresponding to ˙ y (1) and ˙ y (2) .
Revised: " , where the separator | delimits the two components of f ( y, t ) corresponding to ˙ y (1) and ˙ y (2) .

Revised: Although the extension to systems is conceptually straightforward, we remark that the performance of the approach needs to be carefully evaluated in future work as systems of ODEs show a much larger variety of qualitative behaviors - including chaos.

Revised: One of the defining characteristics of chaos is the sensitivity of solution trajectories to initial values (Strogatz, 2018), that is, even similar initial values result in drastically different trajectories.

Revised: Chaotic behavior may pose a fundamental limitation for

Revised: NSODE, which relies on supervised optimization on a representative training dataset, a challenging requirement in case of chaotic systems.

Original: Higher-order equations.
Revised: Higher-order equations.

Original: Finally, it is well known that any higher order (system of) ODEs can be reduced to a first-order system of ODEs.
Revised: Finally, it is well known that any higher order (system of) ODEs can be reduced to a first-order system of ODEs.

Original: Specifically, a d -th order system of K variables can be reduced to an equivalent first-order system of d · K variables.
Revised: Specifically, a d -th order system of K variables can be reduced to an equivalent first-order system of d · K variables.

Original: Hence, one can handle higher-order systems analogously as before with multiple separator tokens.
Revised: Hence, one can handle higher-order systems analogously as before with multiple separator tokens.

Original: One obstacle in this case is that when only observations of y ( t ) are given, one first needs to obtain observed derivatives to reduce a higher-order system to a first-order system.
Revised: One obstacle in this case is that when only observations of y ( t ) are given, one first needs to obtain observed derivatives to reduce a higher-order system to a first-order system.

Original: These would in turn have to be estimated from data, which suffers from the same challenges we have mentioned previously (instability under noise and irregular samples)
Revised: These would in turn have to be estimated from data, which suffers from the same challenges we have mentioned previously (instability under noise and irregular samples).

Original: Finally, when we want to have a single model deal with higher-order equations of unknown order, or systems with differing numbers of variables, it remains on open question how to have the model automatically adjust to the potentially vastly differing input dimensions or how to automatically detect the order of an ODE.
Revised: Finally, when we want to have a single model deal with higher-order equations of unknown order, or systems with differing numbers of variables, it remains on open question how to have the model automatically adjust to the potentially vastly differing input dimensions or how to automatically detect the order of an ODE.

Original: B I MPLEMENTATION D ETAILS B. R ULES TO R ESAMPLE C ONSTANTS
Revised: B I MPLEMENTATION D ETAILS B. R ULES TO R ESAMPLE C ONSTANTS

Original: As described in Section 3.1, we generate ODEs as unary-binary trees, convert them into infix notation and parse them into a canonical form using sympy .
Revised: As described in Section 3.1, we generate ODEs as unary-binary trees, convert them into infix notation and parse them into a canonical form using sympy .

Original: From each skeleton we then create up to
Revised: From each skeleton we then create up to

Original: ODEs by sampling different values for the constants.
Revised: ODEs by sampling different values for the constants.

Original: When resampling constants we want to ensure that we do not accidentally modify the skeleton as this would additionally burden our model with resolving potential ambiguities in the grammar of ODE expressions.
Revised: When resampling constants we want to ensure that we do not accidentally modify the skeleton as this would additionally burden our model with resolving potential ambiguities in the grammar of ODE expressions.

Original: Furthermore, we do not want to reintroduce duplicate samples on the skeleton level after carefully filtering them out previously.
Revised: Furthermore, we do not want to reintroduce duplicate samples on the skeleton level after carefully filtering them out previously.

Original: We therefore introduce the following sampling rules for constants:
Revised: We therefore introduce the following sampling rules for constants:

Original: Do not sample constants of value 0.
Revised: Do not sample constants of value 0.

Original: When the original constant in the skeleton is negative, sample a negative constant, otherwise sample a positive constant.
Revised: When the original constant in the skeleton is negative, sample a negative constant, otherwise sample a positive constant.

Original: Avoid base of 1 in power operations as 1 x = 1 . 4.
Revised: Avoid base of 1 in power operations as 1 x = 1 . 4.

Original: Avoid exponent of 1 and -1 in power operations as x 1 = x and x − 1 = 1 /x .
Revised: Avoid exponent of 1 and -1 in power operations as x 1 = x and x − 1 = 1 /x .

Original: Avoid coefficients of value 1 and -1 as 1 · x = x and − 1 · x = − x 6.
Revised: Avoid coefficients of value 1 and -1 as 1 · x = x and − 1 · x = − x 6.

Original: Avoid divisions by 1 and -1 as x/ 1 = x and x/ − 1 = − x
Revised: Avoid divisions by 1 and -1 as x/ 1 = x and x/ − 1 = − x

Original: B
Revised: B. D ATA GENERATION

Original: D ATA GENERATION

Original: As discussed in the main text, the choices of the maximum number of internal nodes per tree K , the choice and distribution over N bin binary operators, the choice and distribution over N una unary operators, the probability with which to decorate a leaf with a symbol p sym (versus a constant with 1 − p sym ), and the distribution p c over constants uniquely determine the training distribution over
Revised: As discussed in the main text, the choices of the maximum number of internal nodes per tree K , the choice and distribution over N bin binary operators, the choice and distribution over N una unary operators, the probability with which to decorate a leaf with a symbol p sym (versus a constant with 1 − p sym ), and the distribution p c over constants uniquely determine the training distribution over

Original: ODEs f .
Revised: ODEs f .

Original: These choices can be viewed as flexible and semantically interpretable tuning knobs to choose a prior over ODEs.
Revised: These choices can be viewed as flexible and semantically interpretable tuning knobs to choose a prior over ODEs.

Original: For example, it may be known in a given context, that the system follows a “simple” law (small K ) and does not contain exponential rates of change (do not include exp in the unary operators), and so on.
Revised: For example, it may be known in a given context, that the system follows a “simple” law (small K ) and does not contain exponential rates of change (do not include exp in the unary operators), and so on.

Original: The choice of the maximum number of operators per tree, how to sample the operators, and how to fill in the leaf nodes define the training distribution, providing us with flexible and semantically meaningful tuning knobs to choose a prior over ODE systems for our model.
Revised: The choice of the maximum number of operators per tree, how to sample the operators, and how to fill in the leaf nodes define the training distribution, providing us with flexible and semantically meaningful tuning knobs to choose a prior over ODE systems for our model.

Original: We summarize our choices in Tables 3 to 5, where U denotes the uniform distribution.
Revised: We summarize our choices in Tables 3 to 5, where U denotes the uniform distribution.

Original: Whenever a leaf node is decorated with a constant, the distribution over constants is determined by first determining whether to use an integer or a real value with equal probablity.
Revised: Whenever a leaf node is decorated with a constant, the distribution over constants is determined by first determining whether to use an integer or a real value with equal probablity.

Original: In case of an integer, we sample it from p int , and in case of a real-valued constant we sample it from p real shown in Table 3.
Revised: In case of an integer, we sample it from p int , and in case of a real-valued constant we sample it from p real shown in Table 3.

Original: Finally, when it comes to the numerical solutions of the sampled ODEs, we fixed the parameters in
Revised: Finally, when it comes to the numerical solutions of the sampled ODEs, we fixed the parameters in

Original: Table 6 for our experiments.
Revised: Table 6 for our experiments.

Original: We highlight that there is no such thing as “a natural distribution over equations” when it comes to
Revised: We highlight that there is no such thing as “a natural distribution over equations” when it comes to

Original: ODEs.
Revised: ODEs.

Original: Hence, ad-hoc choices have to be made in one way or another.
Revised: Hence, ad-hoc choices have to be made in one way or another.

Original: However, it is important to note that neither our chosen range of integers nor the range of real values for constants are in any way restrictive as they can be achieved by appropriate rescaling.
Revised: However, it is important to note that neither our chosen range of integers nor the range of real values for constants are in any way restrictive as they can be achieved by appropriate rescaling.

Original: In particular, the model itself represents these constant values merely be non-numeric tokens and interpolates between those anchor tokens (our two-hot encoding) to represent continuous values.
Revised: In particular, the model itself represents these constant values merely be non-numeric tokens and interpolates between those anchor tokens (our two-hot encoding) to represent continuous values.

Original: Hence, the model is entirely agnostic to the actual numerical range spanned by these fixed grid tokens, but the relative accuracy in recovering interpolated values will be constant and thus scale with the absolute chosen range.
Revised: Hence, the model is entirely agnostic to the actual numerical range spanned by these fixed grid tokens, but the relative accuracy in recovering interpolated values will be constant and thus scale with the absolute chosen range.

Original: Therefore, scaling p int and p real by essentially any power of 10 does not affect our findings.
Revised: Therefore, scaling p int and p real by essentially any power of 10 does not affect our findings.

Original: Similarly, the chosen range of initial values ( y min0 , y max0 ) is non-restrictive as one could simply scale each observed trajectory to have its starting value lie within this range.
Revised: Similarly, the chosen range of initial values ( y min0 , y max0 ) is non-restrictive as one could simply scale each observed trajectory to have its starting value lie within this range.

Original: [Table] [Table] [Table] [Table]
Revised: B. M ODEL

Original: For our Transformer model we choose the implementation of BigBird (Zaheer et al., 2020) available in HuggingFace.
Revised: For our Transformer model we choose the implementation of BigBird (Zaheer et al., 2020) available in HuggingFace.

Original: The model is trained on 4 Nvidia A100 GPUs for 18 epochs after which we evaluate the best model based on the validation loss.
Revised: The model is trained on 4 Nvidia A100 GPUs for 18 epochs after which we evaluate the best model based on the validation loss.

Original: We choose a batchsize of 600 samples and use a linear learning rate warm-up over 10,000 optimization step after which we keep the learning rate constant at 10 − 4 .
Revised: We choose a batchsize of 600 samples and use a linear learning rate warm-up over 10,000 optimization step after which we keep the learning rate constant at 10 − 4 .

Original: For the fixed tokens that are used to decode constants, we choose an equidistant grid − 10 =
Revised: For the fixed tokens that are used to decode constants, we choose an equidistant grid − 10 =

Original: x 1 < x 2 < . . .
Revised: x 1 < x 2 < . . .

Original: < x m = 10 with m = 21 .
Revised: < x m = 10 with m = 21 .

Original: This worked well empirically and using fewer or more tokens did not seem to improve model performance substantially.
Revised: This worked well empirically and using fewer or more tokens did not seem to improve model performance substantially.

Original: Finally, we use N eval =for the evaluation of our numerical metrics.
Revised: Finally, we use N eval = 100 for the evaluation of our numerical metrics.

Revised: We did not optimize architectural choices or hyperparameters for maximum performance and use the same choices in all experiments.

Original: While not relevant for our dataset as we check for convergence of the ODE solvers, we remark that the input-encoding via IEEE-754 binary representations also graciously represents special values such as nan or inf without causing errors.
Revised: While not relevant for our dataset as we check for convergence of the ODE solvers, we remark that the input-encoding via IEEE-754 binary representations also graciously represents special values such as nan or inf without causing errors.

Original: Those are thus valid inputs that may still provide useful training signal, e.g., “the solution of the ODE of interest goes to inf quickly”.
Revised: Those are thus valid inputs that may still provide useful training signal, e.g., “the solution of the ODE of interest goes to inf quickly”.

Original: C T EXTBOOK EQUATIONS DATASET
Revised: C T EXTBOOK EQUATIONS DATASET

Original: Table 7 list the equations we collected from wikipedia, textbooks and lecture notes together with the initial values that we solved them for.
Revised: Table 7 list the equations we collected from wikipedia, textbooks and lecture notes together with the initial values that we solved them for.

Original: We can also see that almost all of these equations simplify to low-order polynomials.
Revised: We can also see that almost all of these equations simplify to low-order polynomials.

Original: D D ATASET STATISTICS
Revised: D D ATASET STATISTICS

Original: We provide an overview over the complexity distribution and the absolute frequency of all operators (after simplification) for all datasets in Figure 4.
Revised: We provide an overview over the complexity distribution and the absolute frequency of all operators (after simplification) for all datasets in Figure 4.

Original: We can see that our self-generated dataset covers by far the larges complexity whereas both complexities and operator diversity are much lower for equations in the classic and textbook ODEs.
Revised: We can see that our self-generated dataset covers by far the larges complexity whereas both complexities and operator diversity are much lower for equations in the classic and textbook ODEs.

Original: E B ASELINES
Revised: E B ASELINES

Original: We here describe more detail on the optimization of the baseline comparison models.
Revised: We here describe more detail on the optimization of the baseline comparison models.

Original: Sindy.
Revised: Sindy.

Original: We use the implementation available in PySindy (de Silva et al., 2020) and instantiate the basis functions with polynomials up to degree 10 as well as all unary operators listed in Table 5.
Revised: We use the implementation available in PySindy (de Silva et al., 2020) and instantiate the basis functions with polynomials up to degree 10 as well as all unary operators listed in Table 5.

Original: When fitting sindy to data we often encountered numerical issues especially when using high-degree polynomial or the exponential function.
Revised: When fitting sindy to data we often encountered numerical issues especially when using high-degree polynomial or the exponential function.

Original: To attenuate such issues we set the highest degree of thepolynomials per sample to the highest degree present in the ground truth.
Revised: To attenuate such issues we set the highest degree of the

Original: Secondly, when numerical issues are caused by a particular basis function, we discard this basis function for the current sample

Original: [Table]
Revised: [Table]

Original: Logistic equation with harvesting
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Logistic equation with harvesting
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Tank draining −√ 2 · 9 . 81 · (2 / 9) 2 ·√ y −
Revised: 5 − 5 . 5 · y 0 . Tank draining −√ [Equation]

Original: [Table]
Revised: [Equation]

Original:  Draining water through a funnel − (0 .
Revised: y − 0 . 21 · y 0 . Draining water through a funnel − (

Revised: [Equation]

Original: √ 2 · 9 . 81 · (sin 1 / cos 1) 2 · y − 1 .− 0 .
Revised: [Equation] [Equation] (sin [Equation]

Original: 67 /y
Revised: cos

Original: [Table]
Revised: [Equation]

Revised: 1 .− 0 .

Revised: 67 /y 1 .

Original: velocity of a body thrown vertically upwards − 9 .
Revised: velocity of a body thrown vertically upwards − 9 .

Original: 81 − 0 .
Revised: 81 − 0 .

Original: 9 · y/ 8 .− 0 .
Revised: 9 · y/ 8 .− 0 .

Original: 10 · y − 9 .0 .and
Revised: 10 · y − 9 .0 .polynomials per sample to the highest degree present in the ground truth.

Original:  restart the fitting process.
Revised: Secondly, when numerical issues are caused by a particular basis function, we discard this basis function for the current sample and restart the fitting process.

Original: We run a separate full grid search for every ODE over the following hyper-parameters and respective values (these all include the default values):
Revised: We run a separate full grid search for every ODE over the following hyper-parameters and respective values (these all include the default values):

Original: • optimizer-threshold ( np.logspace(-5, 0, 10) ): Minimum magnitude for a coefficient in the weight vector to not be zeroed out.
Revised: • optimizer-threshold ( np.logspace(-5, 0, 10) ): Minimum magnitude for a coefficient in the weight vector to not be zeroed out.

Original: • optimizer-alpha ( [0 . 001 , 0 . 0025 , 0 .
Revised: • optimizer-alpha ( [0 . 001 , 0 . 0025 , 0 .

Original: 05 , 0 . 1 , 0 . 2] ): L2 regularizer on parameters.
Revised: 05 , 0 . 1 , 0 . 2] ): L2 regularizer on parameters.

Original: • finite differences order ( [2 , 3 , 5 , 7 , 9] ): Order of finite difference approximation.
Revised: • finite differences order ( [2 , 3 , 5 , 7 , 9] ): Order of finite difference approximation.

Original: • maximum number of optimization iterations ([20, 100]): Maximum number of optimization steps.
Revised: • maximum number of optimization iterations ([20, 100]): Maximum number of optimization steps.

Original: For every ODE, sindy is fit using solution trajectory in the initial interval [0 , 2] and validated on the interval (2 , 4] .
Revised: For every ODE, sindy is fit using solution trajectory in the initial interval [0 , 2] and validated on the interval (2 , 4] .

Original: The grid search thus results in a ranking of models with different hyper-parameter configurations.
Revised: The grid search thus results in a ranking of models with different hyper-parameter configurations.

Original: Instead of evaluating only the performance of the best model, we report top-k performance across the ranked hyper parameter configurations.
Revised: Instead of evaluating only the performance of the best model, we report top-k performance across the ranked hyper parameter configurations.

Original: Sindy is computationally highly efficient yet limited in its expressiveness, in particular it can not represent nestedfunctions or non-integer powers. GPLearn.
Revised: Sindy is computationally highly efficient yet complex expressions such as nested or parameterized functions can only be approximated (e.g. by a polynomial basis) but not represented in their exact symbolic form.

Original: We instantiate GPLearn with a constant range of ( − 10 , 10) and all binary operators listed in Table 4 and all unary operators listed Table 5 except for the exponential function which caused numerical issues.

Original: We keep the default hyper-parameters but run a grid search across the parsimony coefficient ( { 0.0005, 0.001, 0.01, 0.05, 0.1, 0.2, 0.5, ”auto” } ) which trades of fitness versus program length.

Original: We choose R 2 as fitness function.

Original: AIFeynman.

Original: We use the AIFeynman implementation from [] and run the algorithm with the following (default) hyper-parameters:

Original: • Brute force try time: 60 seconds • Number of epochs for the training :• Operator set:• Maximum degree of polynomial tried by the polynomial fit routine:

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u
Revised: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u

Original: [Table]
Revised: [Table]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: dd
Revised: dd

Original: [Table]
Revised: [Table]

Original: d i v e x p l o g m u l n e g p
Revised: d i v e x p l o g m u l n e g p

Original: o
Revised: o

Original: w s i n s q r t s u
Revised: w s i n s q r t s u

Original: [Table]
Revised: [Table]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y
Revised: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y
Revised: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 0246810 c o un t
Revised: 0246810 c o un t

Original: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y
Revised: C a dd c o s d i v e x p l o g m u l n e g p o w s i n s q r t s u b y

Original: [Figure]
Revised: [Figure]

Original: c o un
Revised: c o un

Original: [Table]
Revised: [Table]

Original: testset
Revised: testset

Original: [Figure]
Revised: [Figure]

Original: classic
Revised: classic

Original: [Figure]
Revised: [Figure]

Original: c o
Revised: c o

Original: [Figure]
Revised: [Figure]

Original: C a dd c o s e x p l o g m u l n e g p o w s i n s q r t s u b yc o un
Revised: C a dd c o s e x p l o g m u l n e g p o w s i n s q r t s u b y

Original: [Figure]
Revised: [Figure]

Original: testset-textbook       complexity

Original: [Figure]
Revised: [Figure]

Original: c o
Revised: GPLearn.

Original: [Figure]
Revised: We instantiate GPLearn with a constant range of ( − 10 , 10) and all binary operators listed in Table 4 and all unary operators listed Table 5 except for the exponential function which caused numerical issues.

Original: t
Revised: We keep the default hyper-parameters but run a grid search across the parsimony coefficient ( { 0.0005, 0.001, 0.01, 0.05, 0.1, 0.2, 0.5, ”auto” } ) which trades of fitness versus program length.

Original: [Figure]
Revised: We choose R 2 as fitness function.

Original:  F D ETAILED RESULTS
Revised: AIFeynman. We use the AIFeynman implementation from https://github.com/SJ001/AI-Feynman and run the algorithm with the following (default) hyper-parameters: • Brute force try time: 60 seconds • Number of epochs for the training :• Operator set:• Maximum degree of polynomial tried by the polynomial fit routine: F A BLATION

Revised: The experimental results presented in Section 4.3 show that the performance measured by all metrics improves with increases in hyperparameter k of the top-k sampling algorithm.

Revised: While this result is intuitive, it also raises the question whether the model simply generates a diverse set of random guesses that cover the full diversity of the testset.

Revised: We investigated this question via the following ablation experiment: If we assume that the model only generates a diverse set of random guesses, then we should be able to permute predictions across samples without loss in performance, e.g.

Revised: [Equation]

Revised: where f n is the ground truth symbolic expression for sample n and { ˆ f n, 1 , ..., ˆ f n,K } correspond to the generated predictions for sample n with top-k= K .

Original: We provide a comprehensive summary of performances of all models on all datasets in Table 8.
Revised: We implemented this experiment using the results obtained on testset-iv for k = 1536 . Contrary to our assumption Table 8 shows that the resulting performance drops substantially in comparison to the original performance on all metrics. We can therefore conclude that our model is not simply generating diverse random guesses but actually learned a systematic mapping from input observations to symbolic expressions. Similarly, we can conclude that samples in the dataset are sufficiently distinct from each other,

Original: Figures 5 and 6 further show again the number of correctly recovered skeletons by each method per complexity.
Revised: i [Table] e. that the data generation process presented in Section 3.1 resulted a diverse dataset.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: G D ETAILED RESULTS

Original: [Table] [Table] [s] 5. 29 + 1203.
Revised: We provide a comprehensive summary of performances of all models on all datasets in Table

Original: classic

Original: [Table]
Revised: [Table]

Original: [Table] skel [Table] [Table]
Revised: Figures 5 and 6 further show again the number of correctly recovered skeletons by each method per complexity.

Original: [Table]
Revised: [Figure]

Original: runtime [s] 5. 23 + 1291. textbook allclose 58. skel
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: skel

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [s]  23 + 1267.

Original: [Figure] [Figure]
Revised: [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]