Paper: MetaPhysiCa: Improving OOD Robustness in Physics-informed Machine Learning (ICLR-2024)

Reviewer comment (Experiments):
How many basis functions are used in the experiments? If there is no prior knowledge, the number of basis functions will be very large.

Author response:
**A1:** That is a good question. In our experiments, we use the same set of 14 basis functions for all 3 dynamical systems that include sine, cosine and polynomial terms up to 3rd power. At the reviewer’s suggestion, we repeated our damped pendulum experiments with increasing numbers of basis functions ranging from 7 basis terms (sinusoidal terms, polynomial terms up to power 1) to 32 basis terms (sinusoidal terms, polynomial terms up to power 6) per output dimension in the damped pendulum system ($\theta_t$ and $\omega_t$). 

Figure 9 in Appendix D.7 shows the training loss and $||\hat{\Phi}||_1$ over epoch. MetaPhysiCa converges to the true dynamics of the damped pendulum system with 3 basis terms for all the different values of $m$ tested. However, as $m$ increases, model requires a higher number of epochs to reach convergence.

Reviewer comment (Theory):
The proposed method still adopts the SINDy-like approach in which all key terms should be included in a set of basis functions. It may not work on complex equations like $dx/dt = 1/(x_1^2 + sin (x_2^3+x_3))$.

Author response:
**A2:** Thank you for the question. In Section 3.1 we discuss the importance of having appropriate basis functions in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

At reviewer’s suggestion, we show two cases where algorithmic alignment is more challenging to achieve:

1. Appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.  
    In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms:  
    $\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$.  
    Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.  
    
2. More expressive SCM. In Appendix D.4, we extend MetaPhysiCa to a more expressive SCM by composing the given basis functions (e.g., to obtain terms such as $sin(x^2)$, etc.). MetaPhysiCa with such an expressive SCM shows OOD performance gains on a complex ODE task (Appendix D.4), but sometimes suffers from learning stiff ODEs during optimization due to the complexity of such a 2-layer composition procedure. Better optimization techniques may help alleviate this problem. We discuss this as a potential future work in Conclusions section.

Reviewer comment (Experiments):
Conduct experiments on more complex systems in the CoDA paper, such as reaction-diffusion system and Navier-Stokes system.

Author response:
**A3:** That is a good suggestion. We have started exploring fluid dynamic applications as a future direction. For now, at least, we can say that this extension is non-trivial but also very interesting. A PDE needs an expanded set of basis functions including differential operators in order to be able to OOD model behaviors like shock waves and turbulence, which pose unique OOD challenges.

Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This seems to entail developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. Moreover, for fluid simulations, it is important to balance the trade-offs between physical accuracy and computational efficiency. We discuss including PDEs as future work in the Conclusions section.

Reviewer comment (Evaluation):
I am curious about the causal structure in Fig. 4. Which two terms in each of the three dynamical systems have causal relationships? Please give an example to explain it. It seems that the authors did not discuss this in the Evaluation section.

Author response:
**A4:** Thank you for the question. First, a note not entirely related to the reviewer’s question but that helps avoid misunderstandings: The causal model is a model of the dynamical system. That is, for the interventions on the initial state $X_{t_0}$ and environment changes ($W^*$ parameters), the resulting output of our model is equivalent to what would happen in the system (up to noise). This is enough to make the model robust OOD but it is not a complete causal model of the real system (interventions on variables that are not $X_{t_0}$ and $W^*$ parameters are not considered).

Now to the reviewer’s question: All terms on the RHS of the true dynamical system affect the future dynamics $dx/dt$. For example, in the damped pendulum system, the change in angular velocity is causally affected by both $sin(\theta_t)$ and the damping term. To be OOD robust in our scenarios we must be robust to changes in initial conditions $X_{t_0}$ and interventions on the parameters $W^*$.

Reviewer comment (Experiments):
Please conduct experiments on noisy data.

Author response:
**A5:** Thank you for the suggestion. We had evaluated the effect of noise on MetaPhysiCa’s ability to find the true dynamical system in Appendix D.3. We have now added a pointer to this section in the main text.

In Appendix D.3., we repeat the Damped pendulum and Predator-prey experiments with increasing amounts of noise. Specifically, we add 1%, 5% and 10% Gaussian noise to all the trajectories, both in training and in test. We report the normalized RMSE for different models trained on the noisy versions of data in Figure 8.

In both tasks, the proposed method is relatively robust to small amounts of noise and outperforms the baselines. With 10% noise, MetaPhysiCa is unable to identify the dynamical system accurately, but performs comparable to the baselines. This is consistent with the impossibility results in Fajardo-Fontiveros et al. [1], where the authors show that there is a fundamental limit to learning the true model from noisy data: after a level of noise, true model is unlearnable by any model.


**References**

[1] Fajardo-Fontiveros, O., Reichardt, I., De Los Ríos, H.R., Duch, J., Sales-Pardo, M. and Guimerà, R., 2023. Fundamental limits to learning closed-form mathematical models from data. Nature Communications, 14(1), p.1043.

Reviewer comment (Experiments):
While MetaPhysiCa showcases success in the datasets mentioned, it is usually the case that the the correctly specified basis function is included in the search space of $f$. MetaPhsysiCa’s performances in the set of incorrectly specified basis function is not known.

Author response:
**A1:**  That is a good point that we now address in the revised submission. We have added clarifying statements in the updated submission. Specifically, we emphasize in Section 3.1 that appropriate basis functions must be present in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

In the updated submission we perform an interesting experiment (thank you for the suggestion!). We repeated the damped pendulum experiment without algorithmic alignment, i.e., appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.
In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms: 
$\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$. 
Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.

Reviewer comment (Experiments):
There should be an ablation study that shows MetaPhysiCa’s performances for a relatively small pool of $f$ basis functions and a relatively large pool of $f$ basis functions. For example, for the large pool of $f$, does learning the causal structure more difficult?

Author response:
**A2:** Thank you for the great suggestion! In Appendix D.7 of the revised submission, we repeat our damped pendulum experiments with increasing numbers of basis functions ranging from 7 basis terms (sinusoidal terms, polynomial terms up to power 1) to 32 basis terms (sinusoidal terms, polynomial terms up to power 6) per output dimension in the damped pendulum system ($\theta_t$ and $\omega_t$). 

Figure 9 in Appendix D.7 shows the training loss and $||\hat{\Phi}||_1$ over epoch. MetaPhysiCa converges to the true dynamics of the damped pendulum system with 3 basis terms for all the different values of $m$ tested. However, as $m$ increases, model requires a higher number of epochs to reach convergence.

Reviewer comment (Experiments):
The paper mentions that joining optimizing all parameters result in comparable experiments than bi-level optimization. But the experiments in the main body of the paper forego showing this. The authors should consider adding experiment results that shows bi-level optimization results as well as the jointing optimization.

Author response:
**A3:** We have now added a section D.2.3 in the Appendix discussing bi-level optimization vs joint optimization in more detail. We observe that bi-level optimization and joint optimization result in $\hat{\Phi}$ learning the true dynamics for all 3 tested dynamical systems. 
However, joint optimization is $8.2\times$ faster per epoch than the bi-level optimization (taking 90ms vs 744ms on average on one Intel(R) Xeon(R) CPU core). The final NRMSE values are the same for both as test-time adaptation is an independent step and learns the same task-specific parameters once the true structure is learnt by either optimization method.
We have added a pointer to this section in the main text.

Reviewer comment (Experiments):
The authors have described the MetaPhysiCa primarily for ODE tasks. However, could the MetaPhysiCa be adapted to PDE setting, for example, for finding the parameters for Burger’s Equation? How could MetaPhysiCa be adapted to methods such as PINN?

Author response:
**A4:** That is an interesting question. We have started exploring fluid dynamic applications as a future direction. For now, at least, we can say that this extension is non-trivial but also very interesting. A PDE needs an expanded set of basis functions including differential operators in order to be able to OOD model behaviors like shock waves and turbulence, which pose unique OOD challenges.

Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This seems to entail developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. Moreover, for fluid simulations, it is important to balance the trade-offs between physical accuracy and computational efficiency. We discuss including PDEs as future work in the Conclusions section.

Reviewer comment (Theory):
Why are only the task-specific parameters updated during the test-time? Why would causal structure not needed to be updated? For fraudulent detection system, it is possible that a ODE function shifts during the test time.

Author response:
**A5:** Thank you for the interesting question. In our OOD settings, we assume we will not see enough of the test-time series in order to learn a new basis (or a modification of a basis). SINDy, for instance, tries to learn the basis under this setting and generally fails. Hence, our OOD shift is with respect to the interventions on the initial conditions $X_{t_0}$  and parameters $W^*$, while the underlying structure of the dynamical system remains the same. Thus, it is enough to update the task-specific parameters. Small OOD changes to the causal structure is an interesting future work direction, possibly requiring additional regularization terms to restrict the amount of changes to the learnt structure.

Reviewer comment (Theory):
Theorem 1 is only guaranteeing that MetaPhysiCa identified the true causal structure. What is the theoretical guarantee that MetaPhysiCa discovers the good task specific parameters, especially for few-shot updates during the test time?

Author response:
**A6:** This is another interesting challenge. Unfortunately, establishing theoretical bounds for task-specific parameters under arbitrary basis assignments seems quite difficult. These bounds appear intrinsically linked to the basis used in the causal-equivalent model, leading to a scenario where each application demands its unique set of parameter bounds. Although there might be potential methods to circumvent these task-dependent bounds, we are yet to find a universally applicable solution that could extend across all tasks.

Reviewer comment (Theory):
Could the authors provide more insights into the computational complexity introduced by MetaPhysiCa, especially for more larger pools of $f$ basis functions and increasing number of observations?

Author response:
**A7:** For a single task $i$ and time $t$, computing the predicted derivatives $d\hat{X}^{(i)}_t/dt$ in Equation 3 has $O(md)$ complexity to evaluate the Hadamard product and the matrix-vector product, where $m$ is the number of basis functions and $d$ is the state dimension of the dynamical system. Since there are $M$ tasks/trajectories of maximum length $T$, the overall complexity per epoch is $O(mdMT)$. However, as Figure 9 shows, MetaPhysiCa could require a higher number of epochs to converge for higher values of $m$. 
We have added a paragraph in Appendix C.1 discussing the time complexity.

Reviewer comment (Theory):
Would MetaPhysicCa be adapted or be integrated into PIML methods that contain different loss functions? Is that as simple as adding the additional loss terms into Equation 4?

Author response:
**A8:** Depending on the application, other PIML loss functions could be further incorporated (e.g., energy conservation, monotonicity, etc.) while preserving the basic principles of MetaPhysiCa: Orthogonal basis functions, identifiability of the equivalent causal model for OOD robustness. However, these PIML constraints should also be enforced during test-time adaptation of task-specific parameters by adding these loss terms in Equation 5.

Reviewer comment (Experiments):
Testing is required on more challenging settings (e.g., 1D convection, convection diffusion other “stiff” PDE / ODE settings where physics informed approaches are known to fail).

Author response:
**A1:** We agree that extending MetaPhysiCa to forecasting PDEs (instead of ODEs) such as 1D convection, diffusion, etc under OOD scenarios is an important future research direction; this would include an expanded set of basis functions including differential operators. Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This entails developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. 

We discuss including PDEs as future work in the Conclusions section.

Reviewer comment (Theory):
The assumption that the collection of m possible (appropriate) basis functions are always available seems too strong to the reviewer. It would be helpful if more clarity about this can be provided by the authors. Basically, this strong assumption might significantly reduce the impact of the proposed method as the full set of basis functions might not always be available to select from.

Author response:
**A3:** That is a good point that we now address in the revised submission. We have added clarifying statements in the updated submission. Specifically, we emphasize in Section 3.1 that appropriate basis functions must be present in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

In the updated submission we perform an interesting experiment (thank you for the suggestion!). We repeat the damped pendulum experiment without algorithmic alignment, i.e., appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.

In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms: 

$\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$. 

Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.

Reviewer comment (Experiments):
Have authors tested on more challenging (stiff PDE, ODE settings and on settings like 1D convection, convection-diffusion where traditional physics-informed models fail)?

Author response:
**A2:** Traditional PIML methods including SINDy typically struggle for stiff ODEs because the dynamical system contains both slow and fast varying states [2,3]. Unfortunately, MetaPhysiCa does not solve this PIML issue with stiff ODEs, rather inherits this weakness. For instance, in our experiments, it is unable to identify the stiff ODE describing Robertson's chemical reaction [1] with fast and slow dynamics. One proposed solution in the literature is to learn the ODE only for slow-varying dynamics and use a neural network for the fast dynamics [2]; however this is not guaranteed to be OOD robust due to the neural network component (as discussed in Section 3.1). We believe solving this issue while preserving OOD robustness is an important future work.

Reviewer comment (Theory):
Could you please expound on the assumption of the m possible basis functions? Are there contexts where all sub-parts of the full applicable basis might not be present in the pre-trained network? How can the proposed method adapt to this OOD scenario? Has this been tested?

Author response:
**A3:** That is a good point that we now address in the revised submission. We have added clarifying statements in the updated submission. Specifically, we emphasize in Section 3.1 that appropriate basis functions must be present in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

In the updated submission we perform an interesting experiment (thank you for the suggestion!). We repeat the damped pendulum experiment without algorithmic alignment, i.e., appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.

In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms: 

$\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$. 

Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.

Reviewer comment (Experiments):
My main concern with this paper is the robustness and completeness of the empirical results. Authors developed two OOD scenarios and picked one for each dynamical system. The selection of OOD scenarios for each dynamical system requires further clarification, as the current rationale is not provided. In addition, the authors use limited in- and out-distribution "pairs": only one single distribution for training, and another single distribution for testing. This limits the assessment of method generalizability. Expanding the experimental design to include both OOD scenarios across all dynamical systems and a broader range of distribution pairs would likely enhance the validity of the findings.

Author response:
**A1:** Thank you for the question. We wish to clarify that both the OOD scenarios are tested for all 3 dynamical systems, presented in Appendix due to lack of space. Table 1 (in the main text) and Tables 2&3 (in Appendix, Pg 17) show both OOD scenarios for epidemic modeling, damped pendulum and predator-prey respectively. In both OOD scenarios for all 3 dynamical systems, MetaPhysiCa is able to obtain significantly more robust predictions than baselines.

**A2:** Thanks for the interesting suggestion! In the revised submission we have added results for multiple in- and out-of-distribution pairs in Appendix D.5. Table 9 shows that under a variety of training ranges for initial conditions and parameters $W^*$, MetaPhysiCa is able to learn the true structure of the damped pendulum dynamical system. Given the learnt structure, Table 10 shows OOD NRMSE for different OOD ranges for initial conditions and parameters.

Reviewer comment (Theory):
The methodological scope is limited by not accounting for interactions between basis functions, potentially restricting the model's capability to learn more intricate dynamics. Additionally, by confining the approach to ODE-style dynamics, it may not accurately reflect the complexity of real-world systems, such as those found in real world epidemics.

Author response:
**A3:** We thank the reviewer for the comment. The review will find Appendix D.4 interesting, where we extend MetaPhysiCa to a more expressive SCM by composing the given basis functions (e.g., to obtain terms such as $sin(x^2)$, etc.). MetaPhysiCa with such an expressive SCM shows OOD performance gains on a complex ODE task (Appendix D.4), but sometimes suffers from learning stiff ODEs during optimization due to the complexity of such a 2-layer composition procedure. Better optimization techniques may help alleviate this problem

We agree that extending MetaPhysiCa to forecasting PDEs (instead of ODEs) under OOD scenarios is an important future research direction; this would include an expanded set of basis functions including differential operators, and require careful consideration of OOD boundary conditions. In our revised submission we discuss both these limitations in the Conclusions section.

Reviewer comment (Novelty):
While the method can integrate prior knowledge of dynamical systems into the creation of basis functions, it appears that this is the extent to which such insights can be utilized. Expanding the method to incorporate prior knowledge more deeply could further improve model performance and applicability.

Author response:
**A4:** We agree with the reviewer. In application-specific scenarios, prior physics knowledge could be further incorporated (e.g., known invariances, dimensionless variables, mesh structures for fluid simulations, etc.). The basic principles of MetaPhysiCa would still be preserved: Orthogonal basis functions, identifiability of the equivalent causal model for OOD robustness. We envision MetaPhysiCa as a starting point for OOD robustness in various application-specific scenarios.

Reviewer comment (Experiments):
Is it possible for you to run extra experiments for multiple in- and out-distribution "pairs" and show that the results are consistent? I'd be happy to increase my score after seeing a more robust evaluation of the method.

Author response:
**A2:** Thanks for the interesting suggestion! In the revised submission we have added results for multiple in- and out-of-distribution pairs in Appendix D.5. Table 9 shows that under a variety of training ranges for initial conditions and parameters $W^*$, MetaPhysiCa is able to learn the true structure of the damped pendulum dynamical system. Given the learnt structure, Table 10 shows OOD NRMSE for different OOD ranges for initial conditions and parameters.

Reviewer comment (Theory):
A real epidemic does not follow the SIR model but there are some characteristics of the SIR model that are useful. How can we adapt your work to handle such real-world dynamical system?

Author response:
**A5:** That is an interesting suggestion. In order to verify MetaPhysiCa we focused on models where we were sure there would be a unique causal structure for the train and test data (i.e., the basis functions were nearly orthogonal under the support of the training data). For more complex models, we recommend adding more complex basis functions. There is an interesting set of future works that could design task-specific basis functions and, potentially, design their associated regularization objectives for better learning the equivalent causal model for basis functions that are not orthogonal over the support of the training data.