Paper: Let's do the time-warp-attend: Learning topological invariants of dynamical systems (ICLR-2024)

Reviewer comment (Presentation):
The methodology section appears disorganized. The addition of more subtitles would greatly enhance readability. Moreover, the frequent transition between the main content and the appendix disrupts the reading experience.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

1. We have clarified the text and divided the Methodology section into subjects. To improve clarity and flow, we moved some of the model details from the Appendix to the Methodology and kept the more technical details in the Appendix. We framed and centered tables’ content and adjusted font size where plausible.

Reviewer comment (Presentation):
The model setup details being located in the appendix can make it less accessible, as it's separated from the primary content.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

1. We have clarified the text and divided the Methodology section into subjects. To improve clarity and flow, we moved some of the model details from the Appendix to the Methodology and kept the more technical details in the Appendix. We framed and centered tables’ content and adjusted font size where plausible.

Reviewer comment (Presentation):
While the wording and sentences in the experiments section are well-constructed, there's room for improvement in the organization of tables and figures. Consider enlarging figures and addressing font size issues, particularly in tables. For instance, Table 3's column headers could be better aligned with their respective columns.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

1. We have clarified the text and divided the Methodology section into subjects. To improve clarity and flow, we moved some of the model details from the Appendix to the Methodology and kept the more technical details in the Appendix. We framed and centered tables’ content and adjusted font size where plausible.

Reviewer comment (Reproducibility):
The process for reproducing the results remains unclear. It would be beneficial to include a dedicated "Reproducibility Statement" section, providing readers with the necessary guidance to replicate the findings.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

2. We now provide an anonymous link to our preliminary repository as a footnote at the end of the Introduction section.

Reviewer comment (Experiments):
Regarding the model setup, it would be valuable to understand why the current configuration is deemed the most suitable for this problem. An ablation study, exploring architectural variations such as layers with different sizes, would provide insights into the model's design choices.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

3. We added results for varying hyperparameters (learning rate and training epochs) and architectural choices (dropout rate, kernel size, MLP latent dimension size, number of convolutional channels, and number of convolutional layers), see Tables A4 and A8. We found no significant effect on performance for these hyperparameters and ablations, except for a reduced performance with learning rate = 1e-5.

Reviewer comment (Reproducibility):
Are there specific hyperparameters within this method that require tuning, and if so, what is the recommended approach for hyperparameter optimization?

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

3. We added results for varying hyperparameters (learning rate and training epochs) and architectural choices (dropout rate, kernel size, MLP latent dimension size, number of convolutional channels, and number of convolutional layers), see Tables A4 and A8. We found no significant effect on performance for these hyperparameters and ablations, except for a reduced performance with learning rate = 1e-5.

Reviewer comment (Presentation):
A dedicated 'Limitations' section would be a useful addition to provide a comprehensive perspective on the constraints and challenges of the proposed approach.

Author response:
We thank the reviewer for the helpful and practical comments. We have adjusted the manuscript accordingly, specifically:

4. We added a dedicated limitations section, touching upon the challenges of combinatorial annotations and scaling to higher-dimensional systems.

Reviewer comment (Writing):
The paper is written poorly and hard to read. I cannot find the problem definition. What's the meaning of prototypical system?

Author response:
We thank the reviewer for their remarks. 

Consequently, we have extensively revised the introduction and methods to clarify our problem setting. In brief, our aim is to learn a limit cycle detector which is robust both to nuisance features (cycle shape, frequency, etc) and noise. We achieve this with the topological data augmentation method described in the manuscript, which ensures that those nuisances are ignored in favor of the underlying topological structure which defines the bifurcation classes. Our system is trained on augmented versions of a simple system (the “prototype system”, in this case the simple oscillator system whose equations can be found in “Dynamical equivalence and topological augmentation” subpart in Methodology) which concisely captures the main features of these classes. We have adjusted the text to emphasize these details. 

Further, in the Related Works and in the Results sections, we now argue and demonstrate that existing topological methods are either very sensitive to noise, or require system-specific prior knowledge, like the explicit form of the governing equation or the true bifurcation parameter. 

To our knowledge, current data-driven methods for bifurcation detection deal largely with the problem of predicting the onset of a bifurcation from time series data measured from a single system. Our work is the first to tackle a much more general problem: a data-driven approach to automatically delineate dynamical classes across a huge number of real and synthetic systems varying both in their functional and parametric form. And, while the reviewer is correct that convolutional architectures are not uncommon in data-driven dynamical systems, they are certainly rare in this novel problem setting. 

Our paper makes several key contributions in service of this goal. First, it introduces a novel approach for learning topologically invariant features (further clarified in the revised text) in dynamical systems by leveraging warped vector field data. Second, it achieves invariant classification of complex synthetic data. Third, the paper demonstrates the use of classifier confidence to effectively identify bifurcation boundaries. Last, the proposed method is successfully applied to analyze biological systems, specifically uncovering distinct dynamics in the pancreatic differentiation trajectory using single-cell gene expression data where alternative baselines fail (see updated Results section 4.4).

Last, veloVI (Gayoso et al. 2023), the method the reviewer mentioned, is a generative framework of RNA dynamics. It is an alternative to the upstream steps prior to our method. In our analysis we use the current convention in the field, applying standard preprocessing steps (e.g. filtering genes, normalizing expression, etc.) and extraction of velocities with scVelo. For specifically analyzing single-cell RNA velocity data, the coupling of veloVI and our framework in the future can be very interesting. It would require introducing confidence across phase space but will also better mitigate its partial coverage.

Reviewer comment (Novelty):
The main method is based on convolutional attention method and data augmentation, which is widely used in dynamical system modeling.

Author response:
We thank the reviewer for their remarks. 

Consequently, we have extensively revised the introduction and methods to clarify our problem setting. In brief, our aim is to learn a limit cycle detector which is robust both to nuisance features (cycle shape, frequency, etc) and noise. We achieve this with the topological data augmentation method described in the manuscript, which ensures that those nuisances are ignored in favor of the underlying topological structure which defines the bifurcation classes. Our system is trained on augmented versions of a simple system (the “prototype system”, in this case the simple oscillator system whose equations can be found in “Dynamical equivalence and topological augmentation” subpart in Methodology) which concisely captures the main features of these classes. We have adjusted the text to emphasize these details. 

Further, in the Related Works and in the Results sections, we now argue and demonstrate that existing topological methods are either very sensitive to noise, or require system-specific prior knowledge, like the explicit form of the governing equation or the true bifurcation parameter. 

To our knowledge, current data-driven methods for bifurcation detection deal largely with the problem of predicting the onset of a bifurcation from time series data measured from a single system. Our work is the first to tackle a much more general problem: a data-driven approach to automatically delineate dynamical classes across a huge number of real and synthetic systems varying both in their functional and parametric form. And, while the reviewer is correct that convolutional architectures are not uncommon in data-driven dynamical systems, they are certainly rare in this novel problem setting. 

Our paper makes several key contributions in service of this goal. First, it introduces a novel approach for learning topologically invariant features (further clarified in the revised text) in dynamical systems by leveraging warped vector field data. Second, it achieves invariant classification of complex synthetic data. Third, the paper demonstrates the use of classifier confidence to effectively identify bifurcation boundaries. Last, the proposed method is successfully applied to analyze biological systems, specifically uncovering distinct dynamics in the pancreatic differentiation trajectory using single-cell gene expression data where alternative baselines fail (see updated Results section 4.4).

Last, veloVI (Gayoso et al. 2023), the method the reviewer mentioned, is a generative framework of RNA dynamics. It is an alternative to the upstream steps prior to our method. In our analysis we use the current convention in the field, applying standard preprocessing steps (e.g. filtering genes, normalizing expression, etc.) and extraction of velocities with scVelo. For specifically analyzing single-cell RNA velocity data, the coupling of veloVI and our framework in the future can be very interesting. It would require introducing confidence across phase space but will also better mitigate its partial coverage.

Reviewer comment (Experiments):
More SOTA methods should be compared. For example, there are extensive RNA velocity methods [1]. 

[1] Deep generative modeling of transcriptional dynamics for RNA velocity analysis in single cells, 2023.

Author response:
We thank the reviewer for their remarks. 

Consequently, we have extensively revised the introduction and methods to clarify our problem setting. In brief, our aim is to learn a limit cycle detector which is robust both to nuisance features (cycle shape, frequency, etc) and noise. We achieve this with the topological data augmentation method described in the manuscript, which ensures that those nuisances are ignored in favor of the underlying topological structure which defines the bifurcation classes. Our system is trained on augmented versions of a simple system (the “prototype system”, in this case the simple oscillator system whose equations can be found in “Dynamical equivalence and topological augmentation” subpart in Methodology) which concisely captures the main features of these classes. We have adjusted the text to emphasize these details. 

Further, in the Related Works and in the Results sections, we now argue and demonstrate that existing topological methods are either very sensitive to noise, or require system-specific prior knowledge, like the explicit form of the governing equation or the true bifurcation parameter. 

To our knowledge, current data-driven methods for bifurcation detection deal largely with the problem of predicting the onset of a bifurcation from time series data measured from a single system. Our work is the first to tackle a much more general problem: a data-driven approach to automatically delineate dynamical classes across a huge number of real and synthetic systems varying both in their functional and parametric form. And, while the reviewer is correct that convolutional architectures are not uncommon in data-driven dynamical systems, they are certainly rare in this novel problem setting. 

Our paper makes several key contributions in service of this goal. First, it introduces a novel approach for learning topologically invariant features (further clarified in the revised text) in dynamical systems by leveraging warped vector field data. Second, it achieves invariant classification of complex synthetic data. Third, the paper demonstrates the use of classifier confidence to effectively identify bifurcation boundaries. Last, the proposed method is successfully applied to analyze biological systems, specifically uncovering distinct dynamics in the pancreatic differentiation trajectory using single-cell gene expression data where alternative baselines fail (see updated Results section 4.4).

Last, veloVI (Gayoso et al. 2023), the method the reviewer mentioned, is a generative framework of RNA dynamics. It is an alternative to the upstream steps prior to our method. In our analysis we use the current convention in the field, applying standard preprocessing steps (e.g. filtering genes, normalizing expression, etc.) and extraction of velocities with scVelo. For specifically analyzing single-cell RNA velocity data, the coupling of veloVI and our framework in the future can be very interesting. It would require introducing confidence across phase space but will also better mitigate its partial coverage.

Reviewer comment (Presentation):
"In the left portion of Fig. ??" should have been "In the left portion of Fig. A7"?

Author response:
1. Minor comments - These are now corrected in the manuscript.

Reviewer comment (Presentation):
The reference "ML Cartwbight. Balthazar van der Pol, 1960." seems erroneous.

Author response:
1. Minor comments - These are now corrected in the manuscript.

Reviewer comment (Experiments):
It would greatly benefit the demonstration of the usefulness of the method, if there would be an assessment of the performance on systems that have both a fixed point and a limit cycle.

Author response:
2. We added a new experiment in the appendix regarding systems that have mixed limit cycle / fixed point behavior. Note that we use a supervised, multi-class setup so that the classifier can predict both the existence of a limit cycle and a fixed point. Empirically, we find that our method learns that the point and periodic classes are exclusive. For emphasis, we tested our trained model on samples of a subcritical Hopf bifurcation (see Appendix 7.6 having regimes of point attractor, point and periodic attractors, and periodic attractor). We find that, even though our system has never seen this bifurcation, it can detect the monotonic relationship between class score and the order parameter. In the future, we aim to extend to combinatorial annotations of invariant sets. We report this as a limitation and in the discussion converse about how we intend to address this in follow-up work.

Reviewer comment (Evaluation):
It is surprising that the correlation coefficient for the BZ reaction is higher than Supercritical Hopf, while the boundary seems quite off.

Author response:
3. Figure 2 - Pearson correlation assesses linear relationships.  Since the confidence of the BZ reaction oscillator gradually increases with the distance from the bifurcation, its Pearson correlation is very high, as opposed to systems like Supercritical Hopf whose confidence quickly plateaus. We now plot the confidence as a function of the distance from the bifurcation boundary in Figure A11 alongside Spearman’s correlation to quantify the monotonic relationship.

Reviewer comment (Experiments):
The comparison of the method proposed in the paper to phase2vec and the Vector Field Learned Representation seem unfair considering that the models have been trained on a different number of training samples. Why is your model trained on 10K training samples while the others are trained on only 7.5K? This could explain why phase2vec is performing worse.

Author response:
4. Use of different training set sizes for the different methods - We adopted the reviewer’s suggestion of training baselines on train data of equal size of 10K. This had little effect on the performance of the baselines (see Table A4).

Reviewer comment (Experiments):
It would further clarify the differences between the methods if you could furthermore show how the inference of the bifurcation boundaries look like for the other methods.

Author response:
5. Effect of noise and bifurcation boundary diagrams across methods - In Tables 1 and 2 we now benchmark the performance of baselines with noise and their bifurcation boundary diagrams in Figure A12.

Reviewer comment (Experiments):
To fully assess the contribution of this work a more extensive set of methods to compare to should be considered. First of all, Lyapunov exponents could be used to track bifurcations and the existence of limit cycles (Witte, 2014). This method works for any dimension in principle, not just 2.

Author response:
7. Comparison to other methods - To motivate our approach to learning and generalization, we have added new numerical experiments which compare to methods making use of data topology, namely, via (1) heuristics based on critical points, and (2) Lyapunov exponent thresholding, see Appendix 7.3 for details and Tables A4 and 1 for performance on noiseless and noisy data. We observe that the Lyapunov exponent method is more sensitive to noise than our approach (see Table 2). The Conley Morse Graph Database proposed in (Arai et al. 2009) assumes knowledge of the governing equations, so we did not consider it. The Conlety-Morse graph computation from vector representation developed in (Chen et al. 2007; Chen et al. 2008) was not accessible as the software only works in Windows 10. Finally, numerical continuation algorithms operate by iteratively constructing intersecting solution curves in conjoined phase-parameter space. In most practical settings, we do not have access to parameter space and must contend with a dataset of isolated measurements, for example, time series or vector fields. We do not know how those measurements relate to one another in parameter space, and we certainly lack the ability to parametrically interpolate between those samples in a continuous way. The practical data scientific question then becomes ”how does one detect invariant sets in individual examples?”, which is what our paper strives to address.

Reviewer comment (Experiments):
The Conley-Morse graph (Arai, 2009) has more information, works in a more general setting (other dimensions than 2), and should be considered. Is this method faster? More accurate than constructing the Conley-Morse graph or the method proposed in the paper?

Author response:
7. Comparison to other methods - To motivate our approach to learning and generalization, we have added new numerical experiments which compare to methods making use of data topology, namely, via (1) heuristics based on critical points, and (2) Lyapunov exponent thresholding, see Appendix 7.3 for details and Tables A4 and 1 for performance on noiseless and noisy data. We observe that the Lyapunov exponent method is more sensitive to noise than our approach (see Table 2). The Conley Morse Graph Database proposed in (Arai et al. 2009) assumes knowledge of the governing equations, so we did not consider it. The Conlety-Morse graph computation from vector representation developed in (Chen et al. 2007; Chen et al. 2008) was not accessible as the software only works in Windows 10. Finally, numerical continuation algorithms operate by iteratively constructing intersecting solution curves in conjoined phase-parameter space. In most practical settings, we do not have access to parameter space and must contend with a dataset of isolated measurements, for example, time series or vector fields. We do not know how those measurements relate to one another in parameter space, and we certainly lack the ability to parametrically interpolate between those samples in a continuous way. The practical data scientific question then becomes ”how does one detect invariant sets in individual examples?”, which is what our paper strives to address.

Reviewer comment (Experiments):
Why was a learning rate of $10^{-4}$ chosen for training? Why 20 epochs? How was it "confirmed" that that "was enough to fit the training data"? Why were 20 coefficients used for the Fitted Parameter Coefficients?

Author response:
8. As we now note in the revised manuscript, we chose polynomial representation of degree 3 following the earlier work of (Iben and Wagner 2021) which showed this value was sufficient for dynamical systems similar to those we investigate here.

Reviewer comment (Theory):
What is the discontinuity of the theoretical boundary in Figure 3 and Figure A12 relating to?

Author response:
9. The discontinuity of the theoretical boundary of the repressilator system is due to our discretized computation of the boundary. We take a grid of alpha values and solve for their upper and lower boundaries. We mention this now in Appendix 7.7.