Paper: ODEFormer: Symbolic Regression of Dynamical Systems with Transformers (ICLR-2024)

Reviewer comment (Novelty):
The idea in this paper is not novel enough. The idea of transformer-based symbolic regression on tokenized sequences was previously reported [1]. The proposed model shares the same network structure but is applied to ordinary differential equation data. However, no analysis could be found in the paper on how the network is adapted to this new type of data.

Author response:
Thank you for bringing this up. There are two key differences between our proposed model and the model by D’Ascoli et al..
The model by D'Ascoli et al.'s model is designed for recurrent sequence prediction, focusing entirely on inherently discrete sequences. An example, taken from Table 1 of D’Ascoli et al., is to find the underlying principle of the following (Josephus) sequence “0,1,1,3,1,3,5,7,1,3“, which is given by “u_n = (u_n−1 +n)%(n−1)−1”. In contrast, ODEFormer is developed to predict ordinary differential equations (ODEs), which represents a continuous function. Although input sequences for both models are discrete, ODEFormer assumes this discreteness to arise from sampling a continuous process at discrete measurement times.

Furthermore, the model by D’Ascoli only considers univariate input sequences whereas our model allows for multivariate observations, i.e., it predicts the dynamic interactions between different system components. This important difference makes our model applicable to many more modeling scenarios and also manifests itself in the model architecture, in particular in the embedding layer which in our case needs to combine multiple trajectories. We have improved the description of this aspect of our model in Section 4 in order to clarify this aspect.

In the revised manuscript we have added a new section in Appendix H which emphasizes the differences between our approach and the model by D’Ascoli et al..

Reviewer comment (Presentation):
Figures and explanations of the proposed model are way too rough and lack the necessary details.

Author response:
Thank you for this feedback! In the revised manuscript we have 
- **Modified the overview figure** (Figure 2) to include more details of the proposed approach,
- **improved the description** of the model in Section 4, and extended the section on architecture details and training hyper-parameters in Appendix C.
We believe these adjustments indeed make the paper easier to follow and we are thus grateful for this comment. Should there still be sections that are too rough or lacking details, we would kindly ask the reviewer to point us more concretely towards them so that we can improve these.

Reviewer comment (Experiments):
This paper lacks crucial detail on dataset separation and the definition of tasks in the experiment section.

Author response:
Thank you for this feedback! To clarify: the datasets used for the evaluation were not used for training. Indeed, this was not explicitly stated in the original submission - **we have hence updated the experimental section (Section 5)** to make this more clear. The revised manuscript also includes a concise statement of the experimental tasks.

Reviewer comment (Reproducibility):
Could the authors give a detailed description and specific form of the loss function used during training? Is it symbolic or numerical? If the loss function is symbolic, does it measure the similarity between the symbolic tree structure or simply between the sequences? And as the ground-truth equations can be organized in different orders under both forms, which one of them is chosen in practice, and what are the reasons for refusing other possible sequences/trees?

Author response:
Thank you for your question. We use the established encoder-decoder transformer architecture where the decoder is an autoregressive model that iteratively predicts an output sequence. In our case this output sequence is supposed to be a function (the right-hand side of an ODE), i.e., a sequence of mathematical symbols. The model is trained with the standard cross-entropy loss which, at each autoregressive prediction step, compares the predicted softmax distribution over all possible mathematical symbols vs the 1-hot distribution of the true target symbol. The loss is hence on the symbolic level and measures the similarity between the predicted vs target sequence. 
Indeed, different symbolic sequences may correspond to the same function (e.g. 1 + x = x + 1). We do not attempt to find a canonical representation to resolve such ambiguity during training but instead use the sequence that was originally sampled. Our results in figures 4 and 5 in the manuscript show that this ambiguity does not pose a problem for training or inference in practice. Note that this ambiguity also exists in most sequence to sequence tasks: for example, there are many possible ways to translate any given sentence from one language to another, yet this does not prevent Transformers to excel at translation.
**To prevent future confusion, we have updated the overview figure (Figure 2) and the description of the training procedure in Appendix C.**

Reviewer comment (Experiments):
As the authors claimed in section 2, most existing approaches of symbolic regression require a separate optimization for each new observed system. However, it is unclear in the paper whether or not the proposed model needs separate optimization on unseen systems as previous models do. Could you add some analysis on this problem and provide reasons and results to support your arguments?

Author response:
Thank you for your question. A separate optimization for each new observed system is *not* necessary for ODEFormer. In the submitted manuscript we allude to this in section 2 and mention it explicitly in Appendix D (“In contrast to all baseline models, ODEFormer is a pretrained model and predicted ODEs are not explicitly fit to the data observed at inference time.”). Since this is an important detail of our approach, we appreciate your feedback and agree that this aspect should be mentioned more prominently in the main text. **Therefore we have adjusted the paragraph “Model evaluation” in Section 5 of the revised manuscript**, which now includes the following:

“All baseline models are fitted to each equation separately, including a separate hyperparameter optimization per equation. In contrast, we evaluate a single, pre-trained ODEFormer model on all equations in all benchmarks.”

Importantly, although our model does not require any optimization per newly observed system, it is optionally possible to finetune the constants of a predicted ODE to the particular observed system. This step is described in section 4 “Optional parameter optimization”. In all of our experiments (section 6) we show results of the fully end-to-end ODEFormer as well as ODEFormer with post-hoc constant fine-tuning, dubbed ODEFormer (opt). While on Strogatz, the additional tuning considerably increases accuracy, the effect is less pronounced on ODEBench and for the generalization results (Figure 5).  Despite the additional constant tuning, ODEFormer (opt) remains one of the fastest models in the comparison. In the revised manuscript we added an explicit statement of these results.

Reviewer comment (Experiments):
The authors conduct subsampling in the experiment part and compare model performances under different subsampling rates. Could you explain further what condition in practice the subsampling process corresponds to? And how does the proposed model deal with the resulting irregular time-interval as most sequence transformers are built to encode trajectories with regular time intervals?

Author response:
Thank you for this question. Subsampling simulates measurement processes that do not follow a precise sampling grid, which is a common property of time series data. Examples can be found in natural science but also in many other domains:

- Biology: The Lotka-Volterra equations are a famous ODE system from ecology and describe the interaction of two populations of species, typically predator and prey. Fitting this model to data requires empirical observations of the two populations (e.g. counting how many species there are on a given day). In practice such observations may often not follow a perfect sampling grid e.g. due to weather conditions, logistical constraints on the field study, or perhaps simply because species were only counted on weekdays but not on the weekend. Instead there will be missing observations, i.e. the sampling grid is irregular.
- Physics: A second example of empirical observations obtained on an irregular sampling grid is given by particle collision data in high-energy physics experiments, like those conducted at the Large Hadron Collider. The detection of subatomic particles resulting from high-energy collisions occurs at irregular intervals, dictated by the random nature of the particle collisions and the intervals at which these collisions are recorded by detectors. This results in datasets that are inherently irregular.
- Healthcare Monitoring: In patient health monitoring, data points might be recorded irregularly based on the patient's condition. For instance, heart rate or blood sugar levels might be monitored more frequently during a medical event or change in condition, but less frequently otherwise. 
- Financial Markets: In finance, transactions or market events do not occur at regular intervals. Stock prices, for example, are recorded at the time of each trade, which can vary substantially throughout the trading day.

Irregular sampling grids (in particular in combination with additional measurement noise) pose a severe problem for models that rely on finite difference estimates of temporal derivatives, which become inaccurate under signal corruptions. However, in contrast to almost all baseline models (see table 1), ODEFormer does not require finite difference estimates. Instead we include the time information itself as part of the input that is supplied to the transformer (see section 4: “Embedding numerical trajectories” ). Since we train the model on corrupted (noisy and randomly subsampled) data, the model learns to become invariant with respect to irregular sampling grids. **We have updated the manuscript to make this aspect of our approach more clear.**

Reviewer comment (Novelty):
While authors mention the presence of a very related work "Becker, Sören, et al. 'Predicting Ordinary Differential Equations with Transformers.' (2023).", I am not sure why they do not elucidate the difference between the paper mentioned and their proposed method, why is this method not used as a baseline? There is no attempt to illustrate the difference at all.

Author response:
Thank you for raising this issue. We noted the major difference in the related works (Section 2): the model proposed by Becker et al. (2023) can only be applied to univariate ODEs, while our model can handle multivariate ODEs. However, we agree that this comparison deserves more details and **added a full section (“DETAILED COMPARISON WITH NSODE (Becker et al. (2023)”) in Appendix H** in the revised manuscript, describing the similarities and differences w.r.t. Becker et al. (2023).

Since the model by Becker et al. (2023) can only be applied to univariate ODEs, it can not be benchmarked on the multivariate datasets used in our paper (Strogatz, ODEBench). Instead **we added an evaluation and performance comparison between ODEFormer and Becker et al. (2023)** using the original 1D datasets suggested by Becker et al. (2023). Our results show that ODEFormer shows competitive performances on those datasets as well, for details please refer to Appendix H.2 in the revised manuscript.

Reviewer comment (Experiments):
Regarding the generation of the data, specifically the way it is being integrated: as mentioned, the authors use a fixed homogeneous grid to integrate, and it is claimed that the number of points does not matter during inference (Figure 3). Is this indeed an artifact of the way the data is generated? How do the results change when the integration procedure is altered? It would be great to have the authors comment on this issue.

Author response:
Thank you for bringing up this point, which reveals that our formulation was misleading: to clarify, we do not integrate on a homogeneous grid. Instead, we use the Runge-Kutta method of order 5(4) (often called “rk45”) which is a classic adaptive step size solver. At every integration step, adaptive step size solvers choose the maximally allowed step size, which still keeps the integration error below a user-defined tolerance threshold. 
In practice, this usually results in small steps within areas of high variation and larger steps in more “flat” areas, yielding an irregular sampling grid. While adapting the step size greatly increases integration speed,  the irregular sampling is often undesirable for modeling real-world data. 

 In our particular case, the irregularities would be informative of the signal variation - an unreasonable general assumption for naturally observed data. Fortunately, most numerical solvers (in particular the rk45 implementation underlying scipy.integrate.solve_ivp, which we use) accept an additional parameter ‘time’ which allows the user to define a time grid along which one would like to obtain the numerical solution. In this case, the final solution returned by the solver corresponds to an interpolation of the adaptive step size solution, evaluated on the user-defined time grid. 

In our data generation procedure (Section 3), we supply a fixed homogeneous grid of N points, where N is sampled independently for each generated sample from a discrete uniform distribution, i.e. N ~ U{50, 200}. Since a homogenous grid would yet again correspond to an unreasonable assumption for naturally observed data, we simulate irregular sampling by dropping a fraction of p time points at random (as described in Section 5 “Corruptions”). 
So while the internal adaptive solver uses irregular steps, the numerical solution is an interpolation on a  homogeneous grid from which we randomly remove points to simulate imperfect real-world sampling. We believe this sampling procedure to be sound.

We have clarified this sampling procedure in the revised manuscript to avoid potential confusion.

Nevertheless, you bring up an interesting question regarding the robustness of ODEFormer w.r.t. the number of time points in an observation. The datasets in the additional comparison with Becker et al. (2023) (Appendix H in the revised manuscript, see our previous comment) include trajectories with 256 timepoints, which exceeds what ODEFormer has seen during training (trajectories during training include between 25 to 200 timepoints). Across all three datasets, results on trajectories with 256 time points are similar to results (frequently even exceeding those) with 128 time points, giving evidence that the robustness with respect to the number of time points observed in Figure 3 is not strictly limited to the training distribution.

In our original manuscript we write “[...] ODEFormer is surprisingly insensitive to the number of points in the trajectory” (section 5, “Results on synthetic data”). We did not intend to “claim that the number of points don't matter during inference” but instead that one intuitively might expect better results when more data is available (hence the surprise). However, **we can see how our statement can lead to your interpretation and have hence rephrased this result in the revised manuscript to avoid confusion.**

Reviewer comment (Novelty):
Please compare and contrast the proposed method with the work "Becker, Sören, et al. 'Predicting Ordinary Differential Equations with Transformers.' (2023).". As acknowledged that this is closely related, it should be clearly articulated what are the differences. The mentioning of difference between univariate and multivariate is okay, but this needs more explanation. I strongly believe this should be a baseline for comparison in some setting.

Author response:
Thank you for raising this issue. We noted the major difference in the related works (Section 2): the model proposed by Becker et al. (2023) can only be applied to univariate ODEs, while our model can handle multivariate ODEs. However, we agree that this comparison deserves more details and **added a full section (“DETAILED COMPARISON WITH NSODE (Becker et al. (2023)”) in Appendix H** in the revised manuscript, describing the similarities and differences w.r.t. Becker et al. (2023).

Since the model by Becker et al. (2023) can only be applied to univariate ODEs, it can not be benchmarked on the multivariate datasets used in our paper (Strogatz, ODEBench). Instead **we added an evaluation and performance comparison between ODEFormer and Becker et al. (2023)** using the original 1D datasets suggested by Becker et al. (2023). Our results show that ODEFormer shows competitive performances on those datasets as well, for details please refer to Appendix H.2 in the revised manuscript.

Reviewer comment (Theory):
How should one think about inferring PDE with a variant of this framework? Can we have a discussion around this as part of the limitations if that is the case?

Author response:
Thank you for this suggestion! We agree that extensions to other classes of differential equations, e.g. partial differential equations (PDEs) or stochastic differential equations (SDEs), seem like an exciting direction for future work. A potentially limiting factor may, however, be the difficulty of integrating PDEs numerically with sufficient accuracy and speed to generate a large-scale training dataset. Since we propose a model for ordinary differential equations (ODEs), we do not see potential problems on PDEs as a limitation of our current model. Inspired by your suggestion, we instead **updated the manuscript and added a brief discussion of potential extensions to PDEs and associated concerns** in the conclusion in section 6 (“Conclusion”).