Paper: CONFIDE: CONtextual FInite DifferencE modelling of PDEs (ICLR-2024)

Reviewer comment (Novelty):
Novelty: Both the task definition and network architecture are not novel. The task of estimating PDE coefficients from observation data is previously known as the PDE inverse problem. And the AE architecture and finite difference scheme are not new.

Author response:
While we agree that estimating PDE coefficients from observations is not a new task, the task we aim to solve is quite different. In our scenario, each signal in the datasets (both train and test) are generated from different dynamics. This means that there are dynamics in the test set never seen before. Having different coefficients per signal makes our task much more challenging than other related works, which mainly focus on the scenario where the entire dataset is created from the same dynamics but with different initial conditions. 
The novelty in our approach is that it is capable of creating reliable predictions even in this scenario, while also providing a specific estimation of the PDE coefficients for the given signal. This estimation holds high value in many scenarios as also discussed in many other papers in the area (for example papers [1] and [2].

Reviewer comment (Evaluation):
Significance: the main contribution is claimed as 'Proposing a DL encoding scheme ... enabling generalization for prediction of unseen samples based on minimal input'. However such generalization was only tested on in-distribution settings. Also, the expensive time cost of incorporating a finite-difference solver is not addressed or discussed.

Author response:
We agree that using a PDE-solver has an expensive time cost. However, the main goal of our paper is not to create a fast replacement to a solver (such as in neural-operator related works). Our algorithm’s main goal is to utilize given mechanistic knowledge in order to estimate the coefficients for each signal. We then show at inference time how these coefficients could be used to provide reliable predictions.

Reviewer comment (Experiments):
Solidness: the authors claimed in the abstract that 'We include results of extensive experimentation, comparing our method to SOTA approaches...'. however, the PDEs in experiments are relatively simple, and the baselines are not strong, compared with [1].

Author response:
As we show throughout our experiments section, a relatively simple PDE is already a hard task when considering every signal in the dataset comes from a different context. This task gets even harder when the test set signals are generated from coefficients that were never seen in the train set.
Regarding the benchmarks suggested in the paper you provided, all datasets are generated from using the same coefficients. For example, in the Navier-Stokes experiment, the viscosity coefficient is constant (10^-5), and in the Darcy-Flow experiment, the external force equals 1 for all signals.

Reviewer comment (Experiments):
For params estimation, is the generalization or time cost of CONFINDE better than PINN and the classical adjoint method? For PDE solving, is the time cost of CONFINDE smaller than pure neural models such as FNO?

Author response:
For params estimation, our method provides a very fast alternative to the “integral” approach used in the adjoint method. Adding a PINN loss does not improve the time cost of this method.
To demonstrate this, we added a new demonstration to the appendix (appendix E), where we demonstrate how CONFIDE works in comparison to an adjoint+PINN method.
In this experiment, we evaluate these methods on a pendulum equation (which is formulated as an ODE), and demonstrate that CONFIDE’s results are not only better in terms of prediction and coefficient estimation, but also trains ~44 times faster.

Reviewer comment (Experiments):
The experiments are all about smooth solutions (Fig3 and Fig5). But in more practical settings, the PDE solution includes a discontinuity, also known as a 'shock wave' in Burgers and N-S. In such cases, the finite difference is more likely to fail or require more computations than in the smooth solution. Can you add some experiments on the shock wave data?

Author response:
As requested, we added a new shockwave experiment based on the Burgers’ equation in appendix C1.
For this experiment, we created a new test set of signals that have a discontinuity at t=0, and compared the algorithms on this data. We observed that CONFIDE still outperformed other baselines (as can be seen in table 3). We believe that the reason is that CONFIDE evaluates the derivatives at different patches of the observed context $u^c$ at each iteration, and most of them are smooth. 
Note that in this case, CONFIDE could be modified to handle discontinuity, by evaluating the spatial and temporal derivatives only on the parts in the signal that are smooth. For instance, if we observe that there is a discontinuity in the initial conditions of the signal, at x=x_0, then we can evaluate finite differences for x<x0, and x>x0 separately and enforce the PDE in these regions. For the sake of evaluation we did not add this modification to our algorithm in this

Reviewer comment (Experiments):
In the ablation study, network size (number of learnable parameters) varies in different settings, which may affect the validity of the performance comparison.

Author response:
In the ablation study where we change the size of the context (the amount of time steps in each signal in the train set), the only change of size is in the first layer. The network size does change but not drastically:
rho=0.05: #params = 4.7M,
rho=0.2: #params = 5M, 
rho=0.5: #params = 5.6M.
As can be seen in figure 11, CONFIDE’s results are almost not impaired even when the context size is very small, and the network size is a bit smaller as a result.

Reviewer comment (Theory):
The choice of finite difference methods over finite volume (suited for hyperbolic conservation laws) and finite element methods for unstructured meshes.

Author response:
We use finite-differences method only to obtain numerical derivatives which can be substituted into the PDE. Using other methods, as you mentioned, is completely valid as long as the derivatives can be calculated. The choice of finite-differences in our case, was enough to result in reliable and fast estimation of the coefficients. However, a user can change this easily and plug-in any choice of derivatives calculations.  
Regarding boundary-conditions: when using finite differences to compute the derivatives, we do not use the boundary condition (we only need them when attempting to solve the PDE). Therefore, estimating the coefficients does not depend on boundary-conditions. However, during inference time, when a user wants to use the estimated coefficients to solve the PDE, one would need to determine what boundary conditions are needed.

Reviewer comment (Experiments):
Since the authors are comparing to a variant of NeuralODE, they should compare to DINO ("Continuous PDE Dynamics Forecasting with Implicit Neural Representations", ICLR 2023), which is designed for PDEs instead of ODEs for NeuralODEs and uses neural representation for the spatial discretization and then NeuralODEs to advance the continuous ODE forward in time.

Author response:
In addition to the last comment, we have added the DINO baseline as you suggested.

Reviewer comment (Evaluation):
How does learning the PDE parameters couple with and affect the accuracy of the numerical solver?

Author response:
The process of learning the PDE parameters does not depend on the numerical solver at all. During the training phase, our model consumes the observed signal and outputs an estimation for the coefficients. These coefficients are then optimized to best fit the PDE alongside the derivatives which are numerically computed. It is only in the inference phase where we might use a numerical solver in order to provide predictions to a given signal.

Reviewer comment (Reproducibility):
The numerical solver is a critical component and several solvers are better for different PDEs. Why was PyPDE chosen and which numerical schemes is it using?

Author response:
The main goal of our paper is not to create a new efficient PDE solver (such as in neural-operator related works), but to utilize given mechanistic knowledge in order to estimate the coefficients of the PDE (which are different for each signal). We then demonstrate how one can use the estimated coefficients to create reliable predictions, even using a rather simple solver such as PyPDE (specifically the explicit solver).  
Our training scheme does not use any solver, but directly infers the PDE coefficients.

Reviewer comment (Theory):
Why are finite difference methods used over finite volume and finite element methods? The choice of numerical method may vary for the different class of PDEs. Also, I don't understand why using finite differences eliminates the need to explicitly state the boundary conditions.

Author response:
We use finite-differences method only to obtain numerical derivatives which can be substituted into the PDE. Using other methods, as you mentioned, is completely valid as long as the derivatives can be calculated. The choice of finite-differences in our case, was enough to result in reliable and fast estimation of the coefficients. However, a user can change this easily and plug-in any choice of derivatives calculations.  
Regarding boundary-conditions: when using finite differences to compute the derivatives, we do not use the boundary condition (we only need them when attempting to solve the PDE). Therefore, estimating the coefficients does not depend on boundary-conditions. However, during inference time, when a user wants to use the estimated coefficients to solve the PDE, one would need to determine what boundary conditions are needed.

Reviewer comment (Experiments):
Why is an autoencoder architecture selected? Were there ablation studies to motivate this choice?

Author response:
We address this question in ablation appendix C.3 (figure 10) (which in the new version became D3, and figure 12), where we compare between an architecture without a decoder part, one with a decoder part and one with a decoder part which is also “initial-conditions aware”.

Reviewer comment (Theory):
Based on the notation in Eqn. (1), does the method only work on linear PDEs? For instance, I see nonlinear Burgers $u_t + uu_x = 0 \iff u_t + \frac{1}{2}\frac{\partial u^2}{\partial x} = 0$ in the experiments but I don't see how this equation fits in the form of Eq (1). I later found in Section 4.2 that it is only quasi-linear Burgers that is concerned so I think the p in Eqn. (1) could not be constant and need to depend on $u$ even in this case.

Author response:
We are sorry for the confusion. The coefficients in equation (1) are not necessarily constants, but could be functions. In the Burgers’ experiment, the coefficient b is a function of u, and in the FN2D experiment, the coefficients which are the local reaction terms depend on both solution signals u and v.

Reviewer comment (Evaluation):
Why is FNO and U-Net beating the proposed method CONFIDE in Figure 2a at early time steps?

Author response:
Our method learns a mapping between the observed signal and the coefficients. Estimating the coefficients correctly results in good long-horizon prediction as shown in the figure (and as usually desired). FNO and UNET train to predict one-step into the future, and then auto-regressively iterate to produce the next predictions. This is why these methods tend to do well in the very near future predictions, while CONFIDE tends to be more robust and reliable.

Reviewer comment (Evaluation):
Why is FNO performing so poorly on the Fitz-Hugh-Nagumo equations? This seems odd.

Author response:
As can be seen in Figure 6, FNO behaves very similarly to UNET through most of the prediction time until at some point it diverges from the true solution. The prediction figures we added should help to understand how each algorithm scored at each time point.

Reviewer comment (Theory):
Since CONFIDE is not an operator, is it resolution invariant like Neural Operator methods are?

Author response:
CONFIDE could be used as long as numerical derivatives can be computed (e.g., by finite-differences). This notion does not change when resolution changes, therefore there is no limit to using CONFIDE on different resolutions.

Reviewer comment (Experiments):
Is the extrapolation task tested in the experiments?

Author response:
We stress that each signal in our test set is generated not only from different initial conditions, but also from coefficients not seen in the train set. As can be seen in our experiments section, this task alone is already a hard task, as modern baselines struggle to generate reliable predictions.

Reviewer comment (Evaluation):
How is CONFIDE an "explainable" method?

Author response:
We note that the term “explainable” here is not the same as standard “explainable-ML” methods.  
Our meaning was that since CONFIDE provides coefficient-estimation to a given signal, it provides the user some knowledge (or explanation) regarding the observed signal. For example, In the electric vehicle batteries case ( last paragraph of the introduction), the coefficients hold valuable information regarding the battery’s capacity, how old it is, how fast it could discharge and so on. In healthcare for instance, there has been a great effort in designing a differential system of the cardio-vascular system, and inferring the parameter of cardiac contractility, or the parameter of arterial resistance, holds great value for clinicians as they decide on their treatments.

Reviewer comment (Theory):
the method is limited to PDEs of the form (1)

Author response:
The CONFIDE algorithm is not necessarily limited to the form of equation (1). 
For comparison and demonstration purposes we assumed the form in (1), but in the general case CONFIDE could handle any PDE form, as long as numerical derivatives could be computed through finite-differences. For example, CONFIDE has the ability to evaluate the coefficient in the wave equation (d^2u/dt^2 = c^2 * d^2u/dx^2)  by evaluating the second derivatives of time and space on the observed signal u(t=0..T, x=0…L).

Reviewer comment (Experiments):
no comparison with baselines such as Brandstetter et al., 2022 on prediction accuracy

Author response:
For comparison we chose the SOTA algorithms currently existing (FNO and UNET), and added a baseline that demonstrates an integral approach vs differential approach.