Paper: Benchmarking Structural Inference Methods for Interacting Dynamical Systems with Synthetic Data (ICLR-2024)

Reviewer comment (Reproducibility):
The package is mainly based on Python and R. It is unclear whether the implementation is efficient.

Author response:
We thank you for highlighting the efficiency of our Python and R-based packages. We have adhered closely to the original scripts provided by authors with minimal modifications. This approach allows our benchmark to include methods with diverse computational requirements and implementations across various programming languages, including Python, R, Julia, and C++ (with Python or R wrappers). Given the variety of methods optimized for GPU and CPU, with varying degrees of parallelizability, a direct comparison of implementation efficiency is not feasible within our study's scope. Our primary goal is to evaluate these methods' applicability, with efficiency optimization being a secondary concern. For large-scale applications, we provide the flexibility to optimize or rewrite the code in more efficient languages like C++ or CUDA, catering to diverse research needs.

Reviewer comment (Experiments):
How to ensure the synthetic datasets align with the real-world dynamic systems?

Author response:
Your question about aligning our synthetic datasets with real-world dynamic systems is crucial for our study's validity and applicability. To ensure this alignment:

1. **Realistic Parameter Selection**: Parameters for generating underlying interaction graphs are based on real-world graphs from eleven disciplines, incorporating realistic variable ranges and special structure biases like self-loops from empirical studies.
2. **Incorporation of Real-World Characteristics**: Our datasets are designed to include key characteristics of real-world systems, such as trajectories with noise, variability in initialization, and different dynamics types. We model these dynamics using models of first and second order ODE and models with quadratic dependencies, capturing real-world complexity and unpredictability.
3. **Transparency and Limitations**: We have documented our dataset creation methods and assumptions, allowing other researchers to understand the limitations and potential biases, ensuring informed application of our datasets.

In conclusion, while achieving perfect real-world alignment is challenging, our steps ensure that our synthetic datasets are as close a representation as possible, making them valuable for dynamic system modeling research.

Reviewer comment (Experiments):
Why Gaussian noise. Can the authors consider other types of noises? Also, can the authors test the models' performance under Gaussian noise with different conditions.

Author response:
We thank you for raising the question about our choice of Gaussian noise and the exploration of alternative noise models in our study. Our initial use of Gaussian noise was guided by its fundamental role in scientific modeling, given its well-characterized bell-shaped distribution and its definition by two simple parameters: mean and variance. And most measurement errors can be modeled with Gaussian noise. This choice aligns with the standard practices in structural inference research, as evidenced by its frequent application in seminal works [1, 2]. The linear and additive nature of Gaussian noise simplifies both analytical and computational modeling, making it an ideal candidate for initial model assessments.

However, we fully acknowledge the diversity of noise types encountered in real-world data, which often deviate from the idealized Gaussian model. Variants such as Poisson, uniform, and salt-and-pepper noise present unique challenges and characteristics that are critical to consider for a comprehensive evaluation of our methods.

In line with your valuable suggestion, we plan to extend our study to include these various noise types. This broader approach will allow us to assess the robustness and adaptability of our models under a wider range of conditions, enhancing the applicability and relevance of our findings.

Regarding the specific request to evaluate our models under differing Gaussian noise conditions, we aim to conduct additional experiments varying both the mean and variance of the noise. This will enable us to better understand the resilience and precision of our models under a spectrum of noise intensities and distributions. We believe that these further analyses will greatly enrich our comprehension of the practical limits and strengths of our approaches in real-world scenarios, where noise characteristics can be highly unpredictable.

We regret to inform that due to resource constraints, it might not be feasible to complete these additional experiments before the rebuttal deadline. However, we are committed to pursuing this extended analysis and will update our findings on our website and in future publications as soon as they are available. We encourage interested readers and fellow researchers to stay connected for these upcoming developments.

Reviewer comment (Evaluation):
Complexity/running time is missing.

Author response:
We appreciate your insightful comment on the necessity of including running time information for the structural inference methods we investigated. In response, we have added a detailed table summarizing the average running times, calculated over ten runs. The times are reported in minutes unless otherwise specified:

| Methods \ Node size | 15          | 30          | 50          | 100          |
| ------------------- | ----------- | ----------- | ----------- | ------------ |
| ppcor               | <1          | <1          | <1          | <1           |
| TIGRESS             | 7.91        | 15.66       | 31.66       | 113.92       |
| ARACNe              | <1          | <1          | <1          | <1           |
| CLR                 | <1          | <1          | <1          | <1           |
| PIDC                | <1          | <1          | <1          | 1.00         |
| scribe              | 13.67       | 46.51       | 130.32      | 548.60       |
| dynGENIE3           | 3.47        | 12.43       | 1.57        | 2.29         |
| XGBGRN              | <1          | <1          | 1.50        | 4.70         |
| NRI                 | 22.35 hours | 31.03 hours | 39.65 hours | 45.91 hours  |
| ACD                 | 40.14 hours | 52.90 hours | 69.37 hours | 83.15 hours  |
| MPM                 | 44.20 hours | 59.02 hours | 80.43 hours | 95.72 hours  |
| iSIDG               | 43.80 hours | 67.44 hours | 91.25 hours | 106.51 hours |

These running times were recorded using BN\_NS trajectories, chosen for their one-dimensional feature representation. This approach allows us to evaluate both VAE-based methods and other methods under uniform conditions. As indicated in the table, VAE-based methods generally require more time due to the necessity of initial training. In contrast, other methods can directly infer structure without this training phase, making them more efficient in terms of computation time. However, it's important to note that VAE-based methods offer greater versatility, as they are applicable to both multi-dimensional and one-dimensional trajectories. This broader application scope might justify the longer running times for certain use cases.

This comprehensive evaluation of running times has been included in Appendix D.6 of our revised manuscript.

Reviewer comment (Experiments):
I wonder can the authors consider robust testing for this paper?

Author response:
We thank you for the insightful suggestion regarding the implementation of robust testing in our research. Recognizing the critical nature of validating our findings across diverse conditions, we have undertaken the following measures:

1. **Testing Under Varied Conditions:** Our synthetic dataset, comprising over 213,000 trajectories and 231 distinct underlying interaction graphs, provides a broad and diverse testing ground. This diversity in data inherently encompasses a range of conditions, thereby addressing robustness through varied input parameters within the scope of our benchmarking study.
2. **Sensitivity Analysis:** The influence of varying input data on our model's performance is a vital aspect of our research. While a dedicated sensitivity analysis is beneficial, we believe our study partially addresses this through an examination of how different graph properties can impact the effectiveness of structural inference methods. This investigation indirectly contributes to our understanding of model sensitivity to input variations.
3. **Error and Exception Handling:** We have rigorously evaluated the response of the structural inference methods to varying levels of Gaussian noise. This approach is instrumental in assessing how our methods handle common anomalies and errors in input data, thereby providing insights into their reliability under less-than-ideal conditions.
4. **Replicability and Generalizability:** To foster a culture of transparency and replicability in our field, we have made available the implementations of the structural inference methods, along with their hyper-parameter configurations and datasets. This initiative is aimed at enabling fellow researchers to replicate our study seamlessly, further reinforcing the robustness and applicability of our findings.

We are confident that these measures collectively fortify the robustness of our research. They contribute significantly to a comprehensive understanding of the applicability and reliability of our findings in real-world scenarios. Furthermore, we remain open to and welcome any additional suggestions for enhancing the robust testing of our paper. Your guidance in this regard is highly valued and will be considered earnestly in our ongoing and future research endeavors.