Paper: Learning semilinear neural operators: A unified recursive framework for prediction and data assimilation. (ICLR-2024)

Reviewer comment (Theory):
While the adoption of the FNO architecture is reasonable for the prediction term, it is not clear why the observer gain $K$ should/ are sufficient to be parametrized as in (15). A further explanation of the intuition here might be necessary.

Author response:
**Intuition for the parametrization of observer gain:**
To design the parametrization of the gain $K(t)$ as in Equation (15) we modified the gating intuition discussed in [Guen2020]. There, the gain works as a gate controlling the influence of the measurements in the updated states. When $K(t) = 0$ the measurement does not influence the updated states. Conversely, when $K(t)=I$,
the latent dynamics is only driven by the measurements. However, this strategy does not accommodate the measurement  operators $C$ and so, we modified their design.
In our scenario, $K$ corrects the predicted state based on the error in the predicted measurements, thus, better approximating the observer design. Note, however, that when $C = I$, the same gating interpretation holds for the gain design in (15) for $K\equiv I$.
To clarify this intuition we added the comment "To parametrize the observer gain $K$, we generalized the gating architecture used in (Guen \& Thome, 2020) to cope with general measurement operators $C$ and the observer design:" when introducing (15) in the revised paper.

Reviewer comment (Experiments):
As for the experiments, it seems only the proposed NODA method needs a warmup. While I totally understand the methodology here and the need for the warmup, I just wonder if this would cause unfair comparison with other methods that have no access to this part of the data and thus less strong performance.

Author response:
**Effect of warmup:** 
    We understand the Reviewer's point. Indeed, the proposed strategy uses the warmup data to improve its internal estimated states. Nevertheless, the presented experiments highlight the proposed strategy's benefits when compared with current state-of-the-art NO techniques. Also note that we performed experiments for different lengths of warmup periods $t_H$ (shown in Figure 3 of the main paper) which show that as the warmup time reduces the results of the proposed approach approach those of the standard FNO method. Nonetheless, our contribution is an observer-design-inspired framework that can be coped with different NO strategies besides FNO.

Reviewer comment (Presentation):
It might be better to include colorbars for all the figures, not only for one in Figure 5.

Author response:
**Including colorbars on all figures:** We have included colorbars on all figures in the revised paper.

Reviewer comment (Theory):
Unclear how Eq. 12 was derived. I tried going through the effort of deriving it myself but couldn't get the same results, so it's critical that this step is justified (you can include the proof in the appendix if it burdens the main text) but a proof must be provided since the whole method is based on it.

Author response:
We thank the Reviewer for the constructive suggestions to improve our paper. Regarding each of your suggestions:
+  **Derivation of equation 12:** First, let us note that as explained in [Afshar2023] equation (11) is a solution to the observer system in equation (10) when $K(t)$ is strongly continuous, due to proposition 2.4 in [Afshar2023]. To improve clarity, we have made precise the need for the strong continuity of $K(t)$ for the solution in equation (11) to be valid right before equation (11) in the revised paper. Equation (12) consists of a time-discretization of equation (11). It can be derived by using the semigroup property of $T(t)$ [Curtain2020], which satisfies 
 \begin{align} 
T(t+t')=T(t)T(t')
\end{align}
 for any pair $t,t'$. This way, identifying $t$ in equation (11) with the discrete time $t_k$ we can express $\hat{z}(t_k)$ as:
  \begin{align}
    \hat{z}(t_k) ={}& T(t_k)\hat{z_0} + \int_{0}^{t_k}T(t_k-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
    \\\\
    ={}&  T(t_k-t_{k-1}+t_{k-1})\hat{z_0} + \int_{0}^{t_{k-1}}T(t_k-t_{k-1}+t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
    \\\\
    +{}& \int_{t_{k-1}}^{t_k}T(t_k-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
    \\\\
    ={}& T(t_k-t_{k-1})T(t_{k-1})\hat{z_0} + T(t_k-t_{k-1})\int_{0}^{t_{k-1}}T(t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
     \\\\
    +{}& \int_{t_{k-1}}^{t_k}T(t_k-s)G(\hat{z}(s),s)ds + \int_{t_{k-1}}^{t_k}T(t_k-s)K(s)[y(s) - C\hat{z}(s)]ds \,
  \end{align}
where we used the linearity of $T(t)$. By noting that the term inside the large brackets highlighted in the previous equation is equal to
\begin{align}
  T(t_{k-1})\hat{z_0} + \int_{0}^{t_{k-1}}T(t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds=\hat{z}(t_{k-1})
\end{align}
it can be seen that this equation is now in the same form as equation (12). We have included this derivation as an appendix in the revised paper, and added a remark pointing to this derivation right before equation (12) in the main body of the paper.

Reviewer comment (Theory):
In the main text and the paper from Afshar et al. (2022), The *observer gain* $K(t)$ needs to satisfy the condition that in the absence of noise from the system dynamics $z(t)$ and noise from the measurements $y(t)$, $\hat{z}(t)\rightarrow z(t)$, yet nothing is said about that when the architecture for it was introduced in Eq. 15. Perhaps a pretraining period where no noise is introduced in the samples could be used to make sure that the learned $K$ satisfies the aforementioned condition (at least approximately) and then finetuning further using your training scheme.

Author response:
We thank the Reviewer for the constructive suggestions to improve our paper. Regarding each of your suggestions:
+ **Conditions for observer design in Afshar's paper:** We agree with the reviewer that the observer design could be used to inform the architecture of the observer gain $K(t)$. However, we emphasize that the proposed approach is fully data-driven, and thus, the conditions for it to work are not directly comparable to those in [Afshar2022]. Moreover, during our experiments, we were able to obtain significant improvements in the experimental results compared to the competing techniques when using the parametrization of the observer gain in (15), without need for a noiseless pre-training period. Thus, although a pretraining step seems like an interesting way of tightening the connection between the optimal observer design and the proposed data-driven strategy, this will be the focus of followup work. We modified the text in the paper above equation (11) in the paper, where we discuss the conditions under which the optimal observer design is expected to work, which now reads:

    "By solving a nonlinear infinite-dimensional Riccati equation, one can design an operator $K(t)$ based on the PDE in equation 10 that can compute the solution $\hat{z}(t)$ over time, such that under the absence of noise and disturbances, and under mild additional conditions, the solution satisfies $||z(t)-\hat{z}(t)||_{\mathcal{Z}_x}\to0$ as $t\to\infty$ for sufficiently small initial errors (see Theorem 5.1 in [Afshar2023]). Furthermore, the authors show that under bounded disturbances $\omega(t)$ and $\eta(t)$, the estimation error can also be bounded for all $t$ (see Corollary 5.2 in [Afshar2023]). Similarly to the previous case, when $K(t)$ is strongly continuous the analytical form of a solution $\hat{z}(t)$ to (10) is given by:''

Reviewer comment (Theory):
Is it possible to have a clear demonstration of why Navier-Stokes and KdV equations are semi-linear PDEs?

Author response:
+ **NS and KdV as semilinear PDEs:** For better convenience, let us recall here the expression of these PDEs, for the NS equation:
    \begin{align}
        \frac{\partial z(x,t)}{\partial t} + u(x,t)\nabla z(x,t) = v\Delta z(x,t) +f(x)
    \end{align}
    where $\nabla \cdot u(x,t)=0$, $z=\nabla \times u$;
    and for the KdV equation, which we have write it into an easier-to-interpret form (compared to its form in the original paper) as:
    \begin{equation}
        \frac{\partial z}{\partial t} = -z\frac{\partial z}{\partial x}-\frac{\partial^3 z}{\partial^3 x} 
    \end{equation}
In short, both the NS and KdV equations can be viewed as semilinear evolution PDEs as they contain 1) only a liner first order differential term involving the time variable, 2) linear differential terms that do not depend on the time derivatives of the solution $z$, and 3) nonlinear terms which also do not depend on the time derivatives of the solution $z$ and satisfy some regularity conditions.
However, formally showing that this is the case is more laborious, thus we spare us both the work and point to some relevant references. For the NS equation, this can be found, for instance, in Section 3.3 of [Temam1982]. We have included this reference in the revised manuscript. For the KdV equation, which is known to be a third order semilinear PDE [Sonner2022], we have included a reference and also rewritten it using a clearer notation, as in the equation shown just above in this response, which we believe should make the connection to the general semilinear evolution PDE form easier for the reader to visualize.

Reviewer comment (Reproducibility):
**Computational Overhead**: Although the paper suggests that the proposed model has little computational overhead, it would be valuable to provide more specific information regarding the computational resources required for practical implementation.

Author response:
We would like to point the reviewer to Appendix B where we have provided the details of the computational resources used for conducting all the experiments. 
    To compare the NODA with the competing methods with respect to the computational overhead required, we are running simulations to evaluate the timings of the different methods, and will include them in the revised version of the paper by the rebuttal deadline.

Reviewer comment (Evaluation):
The abstract claims that the proposed method makes fast predictions, and has “little computational overhead”. Yet no experimental evidence is provided to support this statement. It’s important the authors rectify this by including quantitative timing metrics for the experiments, and compare it against the other methods presented.

Author response:
+ **Including timings for the experiments** We are running the simulations to evaluate the timings, and will include them in the revised version of the paper.

Reviewer comment (Experiments):
The selection of experiments is very limited – only looking at one 1D and one 2D PDE. It would be advisable to at least present results for more than one Reynolds number.

Author response:
+ **Including more variety in the experiments:** We note that an extra experiment in the Kortweg-De Vries (KdV) equation is present in the appendix of the submitted manuscript. Nonetheless, we are working on additional experiments for NS with $R_e\in$ {500, 5000} for FNO, MNO and NODA that will be included in the appendix of the revised manuscript by the rebuttal deadline.

Reviewer comment (Evaluation):
The experimental results are well presented, but the tables lack uncertainty estimates. The reader is therefore left unable to gauge whether the performance differences between different methods are statistically significant.

Author response:
+  **Including uncertainties in the experimental results:**  
    We thank the Reviewer for pointing that out. We added the uncertainties to our tables, and they are small compared with the performance gain obtained with NODA.

Reviewer comment (Experiments):
While there are a good number of alternative methods in the benchmarks, I’m not sure they were the most appropriate choices. The two LSTM-based models are taken from papers that are over eight years old. Meanwhile the MNO is specifically designed for dissipative dynamics, yet the Navier-Stokes experiment is chosen to be in the non-dissipative regime. Furthermore, none of the other methods selected for benchmarking are able to make use of the noisy data, so we are unable to test the key feature of NODA. Alternative works which may offer more suitable benchmarks include ‘Approximate Bayesian Neural Operators’ and ‘Multiwavelet-based Operator Learning for Differential Equations’.

Author response:
+ **Justification for the benchmark methods:** To the best of our knowledge, NODA is the first method to integrate NO learning, data assimilation, and a flexible system modeling that can also account for, e.g., measurement noise. Thus, it is indeed difficult to find baseline methods that would allow for a perfectly fair comparison.  
We have selected MNO and LSTM-based methods due to their recursive structure, which resembles one of the main aspects of our NODA, while FNO is used as a basic block in the architecture of NODA.  
We have clarified the justification for the selected baselines in the beginning of Section 4 in the revised paper, which reads:  

  "The MNO and LSTM-based methods were used as baselines due to their recurrent nature, which is also one of the main aspects of NODA, while FNO was selected since it is one of the main building blocks used in the NODA architecture."     

  The baselines proposed by the Reviewer are definitely relevant, and we have included them in the related work section of the revised paper. We are currently working to evaluate them on the considered datasets, but, unfortunately, due to limited time and computational resources we might not be able to include them before the end of the rebuttal period. Nevertheless, in recognizing the benefit of such as comparison, we are working on obtaining results to compare NODA with the Multiwavelet NO approach [Gupta2021].  Regarding the Bayesian approach proposed in [Magnani2022], we could not find its source code in online open repositories. For this reason, we only discuss it in the related work section.

Reviewer comment (Evaluation):
Including timings for the experiments, to show how fast the algorithm is compared to competing methods.

Author response:
+ **Including timings for the experiments** We are running the simulations to evaluate the timings, and will include them in the revised version of the paper.

Reviewer comment (Experiments):
To introduce more variety in the experiments. Would it be impractical to explore higher Reynolds numbers, and do we expect NODA to cope less well with turbulent flow?

Author response:
+ **Including more variety in the experiments:** We note that an extra experiment in the Kortweg-De Vries (KdV) equation is present in the appendix of the submitted manuscript. Nonetheless, we are working on additional experiments for NS with $R_e\in$ {500, 5000} for FNO, MNO and NODA that will be included in the appendix of the revised manuscript by the rebuttal deadline.

Reviewer comment (Experiments):
Either some justification needs to be added for the particular choices of the benchmark methods, or more suitable alternatives should be introduced in their place.

Author response:
+ **Justification for the benchmark methods:** To the best of our knowledge, NODA is the first method to integrate NO learning, data assimilation, and a flexible system modeling that can also account for, e.g., measurement noise. Thus, it is indeed difficult to find baseline methods that would allow for a perfectly fair comparison.  
We have selected MNO and LSTM-based methods due to their recursive structure, which resembles one of the main aspects of our NODA, while FNO is used as a basic block in the architecture of NODA.  
We have clarified the justification for the selected baselines in the beginning of Section 4 in the revised paper, which reads:  

  "The MNO and LSTM-based methods were used as baselines due to their recurrent nature, which is also one of the main aspects of NODA, while FNO was selected since it is one of the main building blocks used in the NODA architecture."     

  The baselines proposed by the Reviewer are definitely relevant, and we have included them in the related work section of the revised paper. We are currently working to evaluate them on the considered datasets, but, unfortunately, due to limited time and computational resources we might not be able to include them before the end of the rebuttal period. Nevertheless, in recognizing the benefit of such as comparison, we are working on obtaining results to compare NODA with the Multiwavelet NO approach [Gupta2021].  Regarding the Bayesian approach proposed in [Magnani2022], we could not find its source code in online open repositories. For this reason, we only discuss it in the related work section.

Reviewer comment (Presentation):
Figures 1 and 2 are lacking a colour scale, making it challenging for the reader to assess the nature of the uncertainties. The reader is left to guess values based upon the Table but it would be preferable to be explicit and quantitative.

Author response:
+ **Including colorbars in the figures:** We have included colorbars in all figures in the revised paper.