Paper: HyperSINDy: Deep Generative Modeling of Nonlinear Stochastic Governing Equations (ICLR-2024)

Reviewer comment (Evaluation):
Problem setting about the stochastic system. As the author mentioned, they used an alternative definition of the stochastic dynamics, e.g., random ODE. However, the reviewer is confused by the benefit of such definition compared to the deterministic setting. In result part 4.1, both the mean and the STD of the underlying system are shown. However, the STD form doesn't correspond to any dynamics and the mean is close to the true mean of the system. In such circumstances, it seems only the mean estimation is important and the STD cannot be leveraged to judge the performance of the proposed model. For part 4.2, the STD result is also confusing. It is compared with the diffusion terms but it is totally different from the true diffusion term. Therefore, the reviewer wants to ask why we need to include the STD and how it can help the proposed model.

Author response:
We apologize for this confusion, which we believe stems from the interpretation of the RDE used to simulate the dynamics. In each case, mean and STD refer to the parameters of the (Gaussian) distribution associated with each coefficient; trajectories are simulated by constructing sample paths from these distributions. In other words, each coefficient is effectively coupled to an independent Wiener process; the system is parameterized by an n-dimensional Wiener  process. Accordingly, each ground truth trajectory is associated with only one particular sample path for the n-dimensional Wiener process.

The STDs factor into the dynamics by specifying the level of dynamical noise that drives each coefficient. The relevance of the STD to the dynamics is most clear in the rows of Fig. 2 corresponding to $\sigma=10$. In these cases, the ground truth trajectory (“Original”) is highly stochastic. HyperSINDy succeeds in recovering the ground truth equation along with the correct coefficient means. However, the mean (middle column) is a poor representation of the original system, which behaves much more randomly. Thus, by discovering appropriately large STD values, HyperSINDy is able to accurately represent the level of uncertainty in the system’s evolution; this also enables HyperSINDy to faithfully recapitulate the dynamical behavior of the original system. E-SINDy fails to capture the appropriate level of uncertainty (as it assumes no dynamical noise); as such, it cannot faithfully recapitulate the dynamical behavior of the original system (Fig. S4).

Reviewer comment (Evaluation):
The result needs to compare with more SOTA models and include more comprehensive metrics. There are several models combining learning-based methods and SINDy-like algorithms for the equation discovery tasks [1][2]. Also, by checking Figure 2, we could find that the discovery form is different under different $\sigma$. For an equation discovery problem, it is important to get a consistent and correct form. Therefore, the reviewer suggests adding more metrics on evaluating if the proposed model can get the correct form, e.g., precision and recall metrics.   
              
[1] Bayesian Spline Learning for Equation Discovery of Nonlinear Dynamics with Quantified Uncertainty. 
        
[2] Physics-informed learning of governing equations from scarce data

Author response:
We believe that this point is partially addressed by our clarification of the current problem setting as stochastic rather than merely noisy, deterministic dynamics. Nonetheless, we completely agree that comparison against another state-of-the-art method for probabilistic equation discovery, such as Bayesian Spline Learning, would improve the manuscript. We have updated the manuscript to include comparisons with this method along with additional quantitative metrics – thank you for these suggestions.
Refer to Table S4 in the appendix for precision and recall results that complement the experiments ran for Table 1 in the main text (on Lorenz and Rossler systems generated with 10 seeds). Refer to Table S5 in the appendix for RMSE, precision, and recall comparisons between HyperSINDy and E-SINDy on multiple 10D Lorenz-96 trajectories, each generated with a different noise level. Refer to Table S6 in the appendix for RMSE, precision, and recall comparisons between HyperSINDy, E-SINDy, and Bayesian Spline Learning on multiple Lotka-Voltera trajectories simulated as RDEs with Gaussian noise on every coefficient. (response continued below)  

We agree that the manuscript can be further improved by including metrics to evaluate recovery of the proper equation form. We have added these metrics as suggested. We find that recall is generally high for all methods, while HyperSINDy generally yields significant improvement on the precision metrics. However, we note that both precision and recall may be more sensitive to specific parameter choices (especially concerning sparsity thresholds, as it helps determine how many terms get thresholded out).
Note that we were unsure how to adapt the publicly available code to cases other than 2-dimensional systems, so we did not evaluate Bayesian Spline Learning on our 10D Lorenz96 datasets. Instead, we report results for Bayesian Spline Learning on a 2D Lotka-Volterra system, simulated with the RDE formulation with random gaussian noise on every coefficient. Please refer to appendix Table S6 for these results.

Reviewer comment (Experiments):
The high-dimensional 10d Lorenz 96 is not compared with any baselines. Moreover, the analytical form is not listed in the main manuscript. Figure 4's caption says check equation 9 but that's for Lorenz 63. Lorenz 96 should be like equation 12 but with the concrete forcing terms. Equation 12 indicates that all the coefficients except the forcing terms should be close to 1 or -1. However, the discovered coefficients for $x_i$ are not close to these values. Again, the reviewer doesn't understand what is the gain of reporting the STD form of the equation here.

Author response:
To the first point, we have run simulations comparing to E-SINDy; please refer to appendix Table S5 for these results. To the second point, we thank you for pointing out the error. Indeed, figure 4 should point to equation 12, not equation 9. We have updated the manuscript to reflect this change. To the third point, we agree that the coefficients should be close to 1 or -1. Most of the coefficients are close, except for the strictly x_i coefficients, as you point out. The main point of note is that the learned STD is appropriately scaled. The discovered STD reflects that of the underlying distribution, and notably only the learned forcing term has a high variance.

Reviewer comment (Writing):
The methodology part is confusing. Figures 2 says $\theta$ has 3 terms but page 8 says $\theta$ has 2 terms. Moreover, in the lower part of Figure 2, $z$ is firstly sampled from $p_{\theta}(z)$ then was fed into decoder $H$. However, the definition of $\theta$ has already included $H$, making the $H$ applied to $z$ 2 times. The term "inference model" is commonly used for test time, but the author uses it to indicate the training procedure. These are all the confusing parts that need to be clarified.

Author response:
We thank you for pointing out this confusion. In the case of the Lorenz and Rossler experiments, theta has 19 and 20 terms, respectively. What we display in figure 2 are the coefficients of the discovered equations, many of which only have 3 terms left due to sparsification. As for Page 8, we show RMSE over the mean and STD of all discovered coefficients, as compared to the ground truth (Table 1). Thus, for both mean and STD, we have one single mean and STD value (so 4 values in total). We have updated the manuscript to reflect changes to increase clarity – please let us know if anything remains unclear. 
For our model, $\Theta$ (i.e., the function library) does not include H (the hypernetwork). $\Theta$ is a library constructed entirely using the state variables (“x” in our notation). Specifically, we construct $\Theta(x)$. Then, H takes in a latent vector (z) and outputs the coefficients, $\Xi_{z}$. We multiply $\Xi_{z}$ elementwise with the sparse mask M, then perform a matrix multiplication of the result with $\Theta(x)$ to predict the derivative. 

With regards to the term “inference model”, we use this term in accordance with standard VAE terminology to refer to a model that infers the latent vector from data observations (in our case, ($x,\dot{x}$). For HyperSINDy’s use case, this is exclusive to training, when (state, derivative) pairs are fed into the encoder to infer the latent vector, z. We do not use the term “inference model” to refer to the training procedure per se, but rather the model that is used to perform inference, which is utilized only during the training phase.

Thank you for giving us the opportunity to clarify these points; please let us know if anything remains unclear.

Reviewer comment (Theory):
The capacity of $\Theta(\mathbf{x})$ still holds as a constraint for the performance, especially in the high-dimensional setup. It would be great if the authors could discuss the impact of the $\Theta(\mathbf{x})$.

Author response:
**Response:** We agree that the model is constrained by the capacity of the function library. This is true of most equation discovery methods. The polynomial basis has broad applicability; moreover, there is often a priori knowledge motivating choice of specific classes of basis functions. The SINDy framework has had widespread success with alternative libraries (e.g., Kaptanoglu et al., 2022). Our method is amenable to such alternative libraries, and thus retains the flexibility of the SINDy framework.

Reviewer comment (Experiments):
What would the performance be if certain symbolic terms (shown up in the true equations) are missing in the dictionary in $\Theta(\mathbf{x})$.

Author response:
**Response:** We agree that the model is constrained by the capacity of the function library. This is true of most equation discovery methods. The polynomial basis has broad applicability; moreover, there is often a priori knowledge motivating choice of specific classes of basis functions. The SINDy framework has had widespread success with alternative libraries (e.g., Kaptanoglu et al., 2022). Our method is amenable to such alternative libraries, and thus retains the flexibility of the SINDy framework.

Reviewer comment (Presentation):
What is the column of ''STD'' in Figure 2 showing? Are they showing the standard deviation of the estimates? If that is the case, plugging in the standard deviation as the coefficients in the equations is confusing.

Author response:
**Response:** Thank you for pointing out this confusion. Correct, this figure is showing the standard deviation of the estimated coefficients, although these standard deviations correspond to the noise processes that contribute to the dynamics. We have updated the figure to clarify the interpretation of the standard deviations.

Reviewer comment (Evaluation):
It would be great if the authors could provide more evaluation metrics for generated trajectories. Metrics like Lyapunov exponents would be helpful to see how good the performance is.

Author response:
**Response:** Thank you for this excellent suggestion. Although we were unable to carry out a systematic analysis of the Lyupanov exponents for generated trajectories, we have incorporated two additional metrics, precision and recall (Tables S4-S6). These additional metrics enable assessment of the ability to recover the correct terms of the original equation.

Reviewer comment (Experiments):
How robust the performance would be across different choice of the dimension of $\mathbf{z}$?

Author response:
**Response:** Thank you for raising this important question. We have carried out additional experiments to assess performance as a function of latent dimension. Specifically, we train HyperSINDy models with varying dimensions of z on each of the 3D Stochastic Lorenz trajectories ($\sigma = 1$, $\sigma = 5$, $\sigma = 10$).

Results from this procedure are shown in Fig. S6. We plot the RMSE of the mean and standard deviation of discovered coefficients, as compared to ground truth. It is true RMSE can vary as a function of z dimension. It is noteworthy, however, that for each trajectory, RMSE is always lower than E-SINDy, regardless of z dimension (refer to Table 1 for E-SINDy RMSE). These results help validate our choice of z in the paper: we use 2x the state dimension for z, and this seems to have good performance at balancing RMSE of both the mean and standard deviation of discovered coefficients.

Reviewer comment (Experiments):
Experiments are conducted in simulated environments where the simulation parameters match to the modeling assumptions (mainly around Gaussianity). I would love to see more experiments confirming the applicability of the approach to broader problems, especially with real data.

Author response:
The insight exploited in this paper is that the standard VAE gaussian prior is well-matched to the well-studied case of stochastic dynamics involving white noise. We completely agree that extension to other kinds of noise will be an important direction for future extensions. As an initial investigation into the feasibility for such extensions, we have included a new experiment (Fig. S5) that supports the potential for HyperSINDy to model more complicated noise types (here, noise drawn from a half-Normal distribution).

Reviewer comment (Theory):
The approach aims to learn both the functional form and parameters of the differential equations. Even though I agree that this might help with interpretability, I worry that the identifiability issues might be prominent. Do the authors expect any identifiability problems?

Author response:
We agree that there can be identifiability issues in equation learning; addressing these issues generally comes at the cost of reduced flexibility. This is precisely the tradeoff reflected in these questions. Thus, specifying a function library and imposing sparsity are significant aids toward identifying a parsimonious model. Moreover, by assuming the structure of the noise, we avoid the major identifiability issues that arise when attempting to learn the form of both the deterministic and stochastic terms. These constraints each necessitate a loss of generality, though they are appealing as they retain very broad applicability. The primary limitation of a function library is that it requires a priori knowledge about the dynamics, i.e. some inductive bias is necessary. However, this inductive bias is also the primary motivation behind equation discovery methods: we want to discover a sparse equation from a finite set of possible terms that could describe the system’s dynamics. One alternative might be to enable a more fully data-driven search for the basis functions (e.g., use via a genetic algorithm (Schmidt and Lipson, 2009)), although such approaches are generally incur much greater computational cost.

Reviewer comment (Theory):
The promise of learning functional form is achieved through the function library. Are there any limitations of using such an approach?

Author response:
We agree that there can be identifiability issues in equation learning; addressing these issues generally comes at the cost of reduced flexibility. This is precisely the tradeoff reflected in these questions. Thus, specifying a function library and imposing sparsity are significant aids toward identifying a parsimonious model. Moreover, by assuming the structure of the noise, we avoid the major identifiability issues that arise when attempting to learn the form of both the deterministic and stochastic terms. These constraints each necessitate a loss of generality, though they are appealing as they retain very broad applicability. The primary limitation of a function library is that it requires a priori knowledge about the dynamics, i.e. some inductive bias is necessary. However, this inductive bias is also the primary motivation behind equation discovery methods: we want to discover a sparse equation from a finite set of possible terms that could describe the system’s dynamics. One alternative might be to enable a more fully data-driven search for the basis functions (e.g., use via a genetic algorithm (Schmidt and Lipson, 2009)), although such approaches are generally incur much greater computational cost.

Reviewer comment (Theory):
What are the limitations of using a Gaussian prior with diagonal covariance for the generative model?

Author response:
One limitation is that such a prior assumes the true latent vector is built of multiple  i.i.d. random variables (diagonal covariance). However, one could certainly imagine noise processes for which this assumption does not hold. Even though the hypernetwork contains nonlinearities (and thus can model dependencies between coefficients), some systems may still be challenging to learn. Future work could explore using alternate priors, such as Gaussian Process Variational Autoencoder (Casale et al., 2018), which may better account for such dependencies.

Despite this limitation, we note that the network is able to learn more complex distributions that include dependencies between coefficients. In particular, refer to Figure S5 in the appendix for results on a Lotka-Volterra system simulated with coefficients drawn from a Half-Normal distribution. Even though HyperSINDy uses a standard gaussian prior with diagonal covariance, it is able to approximate the shape of the distribution of the true non-Gaussian coefficients much better than E-SINDy. We view this experiment as a proof of principle for the flexibility provided by hypernetwork’s nonlinearities: it is able to learn a mapping from latent space (a simple standard gaussian) to a much more complex coefficient space.

Reviewer comment (Experiments):
**Experimental results on higher dimensional datasets might be a bit lacking.** One of the important claims of the advantage of HyperSINDy is that it circumvents the curse of dimensionality which hinders the performance of other methods. However, only the HyperSINDy results for one 10D system is given. It might be better if the authors can clarify how the other methods perform on this system, and/or give other examples of high dimensional systems.

Author response:
Thank you for raising this point. Indeed, the ability to scale to higher dimensions is a distinguishing feature of HyperSINDy. To clarify, the limited scalability of existing SDE discovery methods precludes an assessment of their performance on the 10D system. However, we agree that it would still be informative to benchmark HyperSINDy’s performance against other methods for probabilistic model discovery (of deterministic ODEs) in this high-dimensional setting.

To this end, we have updated the manuscript with new experiments applying E-SINDy to the 10D Lorenz system.
In particular, we simulated the 10D Lorenz system with 3 different levels of noise on the forcing term ($\sigma=0$, $\sigma=5$, $\sigma=10$) and trained HyperSINDy and E-SINDy on each trajectory. We evaluated the RMSE of the mean and standard deviation of the discovered coefficients, as well as precision and recall. Refer to Table S5 in the appendix for the full results. Although E-SINDy achieves slightly improved performance in the noise-free case ($\sigma=0$), HyperSINDy outperforms E-SINDy across all metrics on  both of the higher noise levels.

Reviewer comment (Theory):
In section 3, in "$H$ implements the implicit distribution $p_\theta(\mathbf{\Xi}|\mathbf{z})$", why is it the "implicit distribution?" From my understanding, shouldn't $\Xi_z$ just be a delta distribution (deterministic) on $H(z)$?

Author response:
We apologize for the lack of clarity – we believe this reflects the distinction between z as a random sample vs. a random variable. For a specific sample $\hat{z}, Xi_{z} = H(z)$ is indeed deterministic. For the more general case corresponding to z as a random variable, p(Xi | z) refers to the posterior distribution of SINDy coefficients, conditioned on z. In this setting, the hypernetwork can be interpreted as an implicit distribution for p(Xi | z) (i.e. the neural network weights, biases, and architecture parameterize the distribution p(Xi | z), where z is a random variable). We use “implicit distribution” to mean that we do not actually have full access to the distribution of p(Xi | z), but can still sample from it. For example, if we sample an ensemble of z vectors, then feed that ensemble into H, we are effectively sampling from p(Xi | z). We have revised this line for clarity, including a pointer to (Pawlowski et al., 2018).

Reviewer comment (Theory):
$p_\theta(z)$ is modeled to be a standard Gaussian with diagonal covariance. Would the independence between different $z_t$ allow sudden jumps in the parameters of the system? Would it be better to model it as something like a Gaussian process?

Author response:
Yes, this independence could allow sudden jumps in the parameters of the system. However, because HyperSINDy learns to approximate the (potentially complex) distribution of states and derivatives (i.e., q_\phi(z|x,\dot{x} is kept close to the prior p_theta(z)), independent samples z_t are transformed into realizations of system coefficients that are similar to those observed in the original data. In other words, over the course of training, HyperSINDy learns to utilize i.i.d. dynamical noise to mimic the dynamical behavior of the observed data – including tendency toward sudden jumps. This is an important feature distinguishing HyperSINDy from other probabilistic methods that do not explicitly model the role of dynamical noise in the generation of the observed time series.
We agree that Gaussian processes would provide an ideal framework for modeling the noise, particularly when moving beyond the i.i.d. noise case (see also (Casale et al., 2018; Mishra et al., 2022)). For our work, we assumed that the noise was i.i.d. in time, meaning that the independence of z_ts is a desirable modeling feature (as per the standard SDE framework). However, we note that extension to temporally correlated noise will be an important direction for future inquiry.

Reviewer comment (Experiments):
Related to the last question: does the discretization step size influence the model learning result?

Author response:
The discretization step size can influence results, although this is not an issue connected to HyperSINDy per se – rather, this relates to the more general and well-studied phenomenon of “finite-time effects” emerging in the numerical analysis of continuous stochastic dynamics based upon discretely sampled time series. Algorithmically correcting for such effects has been the topic of numerous works (e.g., (Ragwitz and Kantz, 2001; Lade, 2009; Honisch, 2011; Boujo and Cadot, 2019)), including one such approach recently introduced for stochastic SINDy (Callaham et al. 2021). Recent works also introduce promising approaches toward finite-time correction in the high-dimensional setting (e.g., (Brückner, Ronceray and Broedersz, 2020; Frishman and Ronceray, 2020)), suggesting a number of promising future directions for HyperSINDy to be extended to the theoretically continuous case.

Reviewer comment (Writing):
I don't think what "E-SINDy" stands for is ever introduced in the paper.

Author response:
Thank you for pointing out this oversight! We have updated the Introduction to define E-SINDy.

Reviewer comment (Theory):
Random differential equations are conjugate to stochastic differential equations. It is unclear how to convert a general SDEs into its random ODEs representations, for example, the Langevin type dynamics.

Author response:
Although this conjugacy is well-established, explicit transformations between SDEs and RDEs are not always straightforward. The standard procedure involves substitution via an Ornstein-Uhlenbeck process, the solution to a linear SDE. Following Han & Kloeden (2017), we may demonstrate this transformation for a scalar SDE with additive noise:

$$
\begin{align}
	dX_t = f(X_t)dt + dWt
\end{align} $$
becomes
$$\begin{align}
	\dot{Z_t} = f(Z_t + O_t) + O_t,
\end{align} $$
where $Z_t \coloneqq X_t - O_t$ and $O_t$ is the stationary Ornstein-Uhlenbeck process satisfying the SDE $dO_t = -O_tdt + dW_t$.

A similar conjugation can work for many cases of additive or multiplicative SDEs. In general, though, this conversion is not straightforward to implement. Thus, although we appeal to RDE-SDE conjugacy in order to validate results obtained in the RDE framework, the ability to convert between these representations in practice is more challenging.

On the other hand, this challenge of converting between SDE and RDE formulations highlights unique advantages of our problem formulation. Indeed, much of the interest in RDEs is motivated from the practical advantages afforded by this formulation in certain scenarios in comparison to SDEs (e.g., (Bauer et al., 2017; Solin, Tamir and Verma, 2021)).

We have added this discussion to the Supplementary Appendix.

Reviewer comment (Experiments):
This manuscript lacks a comparison with other methods.

Author response:
Our original submission included comparisons with the leading “ensemble SINDy” method for robust equation discovery in the presence of noise, and with “stochastic SINDy” for a two-dimensional SDE discovery model. We have updated the manuscript to also include comparisons with Bayesian Spline Learning. Refer to Table S6 in the appendix for comparisons between HyperSINDy, E-SINDy, and Bayesian Spline Learning on a Lotka-Volterra RDE simulation.

Reviewer comment (Experiments):
Although the authors have commented in the manuscript, it is still unclear if this HyperSINDy framework can handle complex noise terms as well as the robustness of noises.

Author response:
We have performed new experiments to address the question of how our method performs under different scenarios for the noise. Refer to Figure S5 in the appendix for results comparing HyperSINDy and E-SINDy on a Lotka-Volterra system simulated as an RDE, but where each coefficient is drawn from a Half-Normal distribution. In this case, the distribution of coefficients does not match the Gaussian assumption of the latent space; however, HyperSINDy is still able model the complex coefficient distributions due to the expressivity provided by its neural network. This result establishes that the method has potential to generalize beyond the iid noise case, while leaving for subsequent work a more systematic validation in different noise settings.

Reviewer comment (Experiments):
This manuscript could have been enhanced if it can provide examples of learning underdamped Langevin systems, for example, learn the harmonic oscillator under thermal bath.

Author response:
Thank you for raising these possibilities for further extensions of our approach. Our updated Appendix clarifies that explicit RDE-SDE transformation is not always trivial, meaning the recasting of these canonical physics SDE problems may not be entirely straightforward; however, we agree that it represents an important future direction for broadening the scope of our framework.

Reviewer comment (Experiments):
The manuscript could have been enhanced if it can provide numerical examples when different types of noises are added to the observation data.

Author response:
Please see response to Weaknesses #3 above.

Reviewer comment (Novelty):
It is a typical A+B type of paper. Each part is well studied and author glue them together and demonstrate it in several simple examples. I don't think there is enough novelty here.

Author response:
**Response:** We believe the present contribution goes well beyond a “typical A+B paper”, in the sense that the two methods claimed to be simply glued together (hypernetworks (Ha, Dai and Le, 2016; Pawlowski et al., 2018) and SINDy (Brunton, Proctor and Kutz, 2016; Kaptanoglu et al., 2022) are just two pieces of an overarching conceptual and technical framework being put forward. Moreover, these two methods refer to high-level frameworks/architectures with many possible implementations (e.g., a hypernetwork is simply a neural network that predicts the weights of another network – a concept we reimagine for predicting the coefficients of an analytical expression). Thus, the integration of the two methods (which have arisen in different academic communities for very different purposes) is far from straightforward.

At a conceptual level, integration of these methods is entirely contingent on our innovative problem formulation in terms of random differential equations. This alternative formulation for stochastic dynamics enabled us to leverage the unique advantages of numerous algorithms/techniques (including SINDy and hypernetworks, but also, e.g., variational autoencoders and backpropagation-compatible L0 loss) toward our overarching goal of stochastic dynamical modeling.

At a technical level, unifying all these approaches within a common framework is a highly non-trivial task, requiring strategies to appropriately balance multiple losses relating to equation sparsity, overall model complexity, reconstruction error, and proximity to a Gaussian prior (that is, promoting discovery of a model approximating a canonical SDE).

All in all, while it can be heuristically useful to summarize the proposed approach as combining hypernetworks and SINDy, this intuitive picture belies a much more sophisticated integration of techniques and ideas. We believe that this claim is consistent with evaluations from the other reviews, who each note the novelty/innovation of the proposed framework.

Reviewer comment (Experiments):
All three examples are artificially made for this algorithm. All examples are corrected identified but I am not impressed unless authors are able to demonstrate some non-trivial RODE. The second example equation (11) is not even a valid example of stochastic Lotka-Volterra. I don't know what is N(0,1) on the Right hand side means here.

Author response:
**Response:** We believe that this impression stems in part from notation differences between fields. We have modeled systems subject to white noise (thus, a general setting for stochastic dynamics), though we adopt notation more familiar in statistics/machine learning. Thus, N(0,1) denotes a Gaussian distribution with mean=0, standard deviation=1; iteratively sampling from this distribution amounts to simulating a specific realization of a driving white noise process (which is defined by random variables distributed according to N(0,1) at each time point t (Han and Kloeden, 2017). As such, our selected problems are consistent with widely used stochastic modeling problems (namely, classic SDEs) in which the source of stochasticity is taken to be a Gaussian white noise. Nonetheless, we agree that this notation can obscure connections to standard SDE representations; as such, we have rewritten equation 11 in canonical SDE matrix notation involving a 2D Wiener process (which we simulate via the Euler-Maruyama integration scheme), thus clarifying that it is indeed a valid Lotka-Volterra SDE.

Reviewer comment (Writing):
If x' is not available (e.g., after training), z is sampled from the prior z ∼ p_θ(z) to produce \Xi. I don't understand this part. Please elaborate more or give an example.

Author response:
**Response:** This statement refers to the (post-training) use of HyperSINDy to generate stochastic dynamics. Thus, new trajectories are generated by simply sampling z from the prior p_θ(z) (a standard gaussian with diagonal covariance, in the present setting), then feed this z into the hypernetwork to produce a prediction of \Xi. We appreciate you bringing this potentially unclear text to our attention; we have updated this text (Fig. 1 caption) to clarify the above.