Paper: Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws (ICLR-2024)

Reviewer comment (Novelty):
About the novelty. The analysis in Section 3.1 is similar to the following paper, but they didn't cite this paper in this section, making it hard to judge their contribution: Neural Conservation Laws: A Divergence-Free Perspective, NeurIPS 2022.

Author response:
**Comparison with [1] and our novelty**: We would like to point out that we have cited [1] and discussed the differences and limitations in their setting, in the last paragraph of Section 2 in the original manuscript. Although our Section 3.1 has described the differential form representation as in [1], the novelty of our idea is NOT on formulating the conservation law into a divergence-free architecture with a differential form representation. In fact, such a representation is well-known for more than a decade, see, e.g., [2], way before [1]. Therefore, it is not the novelty of [1] either. We have added more references on differential forms in the revised manuscript, to avoid such a confusion.

The novelty of our approach is 1) the combination of neural operator with the mass conservation as a basic physical law, to learn the unknown physical system responses such as the constitutive laws with improved data efficiency; and 2) the design of a pre-calculated numerical differentiation layer in the neural operator architecture, to enable the incorporation of differential form-guaranteed divergence-free output, without increasing the number of trainable parameters and computational cost. To our best knowledge, neither of these two ideas have been discussed in existing neural operator literature. In [1] and all other PINN-based architectures, they are designed based on the knowledge of full governing laws, while we (as the general neural operator papers) consider the hidden physics scenario where the full PDE is not given. In the PINNs setting, derivatives are evaluated via automatic differentiation and hence the evaluation of differential forms is trivial in [1]. However, in neural operators we need to design approaches to evaluate the derivatives. This is also the main focus of section 3.2.


[1] Richter-Powell, Jack, Yaron Lipman, and Ricky TQ Chen. "Neural conservation laws: A divergence-free perspective." Advances in Neural Information Processing Systems 35 (2022): 38075-38088.

[2] Barbarosie, Cristian. "Representation of divergence-free vector fields." Quarterly of applied mathematics 69.2 (2011): 309-316.

Reviewer comment (Experiments):
About the efficiency. Since clawNOs introduce an additional differentiation layer, the comparisons with respect to FNO and GFNO in running time, GPU memory, and model parameters are expected.

Author response:
**Efficiency**: We thank the reviewer's suggestion, and have added table 5 in the appendix of the revised manuscript for detailed performance comparison of selected models. The comparison metrics include the total number of model parameters, the per-epoch runtime, inference time, the peak GPU memory usage, and the L2-norm of the output divergence.

Note that the weights in our additional differential layer is pre-calculated, and it therefore contains no trainable parameters. As shown in table 5 of the revised paper, the number of trainable parameters and the GPU cost of clawNOs and their counterparts are almost the same. During inference, although the number of parameters in iterative layers still dominate, the non-trainable parameters in clawNO will play a role. Therefore, we do observe a minor increase in inference time.

Reviewer comment (Experiments):
More baselines. Actually, FNO and GFNO are both simple baselines. Recently, there are many advanced neural operators, such as LSM [1] and U-NO [2]. It is hard to judge the model performance unless they compare with the above-mentioned baselines.  
[1] Solving High-Dimensional PDEs with Latent Spectral Models, ICML 2023  
[2] U-NO: U-shaped Neural Operators, TMLR 2023

Author response:
**More baselines**: Following the reviewer's suggestion, we have added three state-of-the-art neural operator baselines, i.e., LSM, UNO, and KNO, in our incompressible Navier-Stokes experiment in table 1 of the revised manuscript. Our original conclusion still stands: clawNOs consistently outperform all baselines in all three considered data regimes.

Reviewer comment (Experiments):
More analysis about the divergence-free property of outputs. If the model learns an optimal differentiation layer, the output results will strictly satisfy the divergence-free property. However, the authors adopt an approximation design. How well will the differentiation layer perform? A metric for the output divergence-free property is expected. Does clawNOs really perform better than FNO or GFNO in learning divergence-free results? Experiments are required.

Author response:
**Analysis of the output divergence**: The differentiation layer in clawNOs is not trained through data, but pre-calculated through either FFT or through pre-computed weights, so the divergence-free property of the output is guaranteed. To investigate the effect of this differentiation layer, we compute the averaged L2-norm divergence on the test dataset for all the models trained with ntrain=1000 samples, and report the results in table 5 in the appendix of the revised manuscript. As one can see in the table, the L2-norm divergence is close to zero for both clawGFNO and clawFNO, whereas other non-claw baselines have relative large divergence.

Reviewer comment (Evaluation):
About the long-term prediction. Since the authors claim the long-term performance as an important advancement of clawNOs, they should plot the time-stamp-wise error curves in comparison with other baselines. Concretely, how does the prediction error change along the temporal dimension?

Author response:
**Time-stamp-wise error**: The time-stamp-wise error is added in Fig. 4 in the revised paper. As expected, the prediction error increases as the prediction time step grows, due to the accumulation of error. All models have a similar growth rate, while clawNOs, together with FNO, significantly outperform other baselines in accuracy.

Reviewer comment (Writing):
The authors claim that “ClawNOs significantly outperform the state-of-the-art NOs in both accuracy and generalizability” in the abstract. What does the “generalizability” mean here?

Author response:
**Generalizability**: Our small data regime setting is designed to test the performance of the trained models when the available data is scarce and does not provide a good coverage of the parameter range (i.e., out-of-domain). Taking the incompressible Navier-Stokes experiment as an example, when ntrain is as small as $10$, it is expected that some test samples will be out of the training regime. To validate this, in the paper revision we show in Fig. 3 of the appendix the data distribution in the incompressible Navier-Stokes experiment case 1. In particular, the histogram of the solution L2-norms are plotted in the $x$ and $y$ components. One can see that in the small data regime, the training set does not provide a good coverage of the dataset, and a large number of test samples are outside the training regime. Then, to illustrate the generalizability of clawNOs, we also plot the per-sample test errors in Fig. 3a, where one can see that clawNO's error is much smaller than that of the non-claw counterpart, especially in the out-of-distribution regime. The training datasets of the medium and large data regimes provide good coverage of the entire dataset and aim to test the in-domain predictability of various models.

Reviewer comment (Theory):
Confusion about the physical understanding of clawNOs. Are there any physical meanings to the skew-symmetric matrix-valued function $\mu$, or is this just a mathematical trick? This question is important, since it determines whether clawNOs hold potential physical limitations or not.

Author response:
**Physical meaning to the skew-symmetric matrix-valued function**: There is no unified physical meaning for the skew-symmetric matrix-valued function. Because the clawNOs have no restriction on the applicable physical system, the physical meaning of solution as well as $\mu$ depend on the specific problem. Taking the 2D Navier-Stokes equation as an example, the skew-symmetric matrix has only one degree of freedom, $\mu_{12}$, satisfying $\dfrac{\partial \mu_{12}}{\partial x}=u$ and $\dfrac{\partial \mu_{12}}{\partial y}=v$. Here, $u$ and $v$ are the $x-$ and $y-$ components of the velocity field, respectively. In this context,  $\mu_{12}$ is the potential function defined in classical hydrodynamics literature.

Reviewer comment (Presentation):
The presentation of the paper could have been much better. For some parts such as the last two paragraphs of section 3.1 and section 3.2, the content sounds very unclear. Additionally, the description of different experiments and the result discussions could have been improved. For example: were all tests out-of-domain evaluation or in-domain evaluation? For the cases where FNO outperformed GFNO, the authors could have easily provided numerical investigation instead of describing the reason as their "guess" (referring to the last paragraph of section 4.1).

Author response:
**Clarifications of sections 3.1 and 3.2**: We thank the reviewer's valuable suggestion and have added more clarifications in the appendix (highlighted in red), due to the page limit. In particular, to clarify section 3.1 we have added a section in appendix A for the necessary definition and detailed derivations in differential forms. We have also provided the detailed derivations of section 3.2 in appendix D.

**Reason for FNO outperforming GFNO**: We appreciate the reviewer's insightful suggestion, and have performed an experiment where GFNO's latent dimension is increased to match the latent dimension of FNO. The resulting error is decreased to 1.19\% in the large data regime, which is on a similar scale to FNO (1.47\%). In this setting, the number of trainable parameters in GFNO is increased to 3,412,142. This thus verifies our guess. We have included this discussion in the last paragraph of section 4.1 in the revised manuscript.

**Out-of-domain/in-domain evaluation**: Our small data regime setting is designed to test the performance of the trained models when the available data is scarce and does not provide a good coverage of the parameter range (i.e., out-of-domain). Taking the incompressible Navier-Stokes experiment as an example, when ntrain is as small as $10$, it is expected that some test samples will be out of the training regime. To validate this, in the paper revision we show in Fig. 3 of the appendix the data distribution in the incompressible Navier-Stokes experiment case 1. In particular, the histogram of the solution L2-norms are plotted in the $x$ and $y$ components. One can see that in the small data regime, the training set does not provide a good coverage of the dataset, and a large number of test samples are outside the training regime. Then, to illustrate the generalizability of clawNOs, we also plotted the per-sample test errors in Fig. 3a, where one can see that clawNO's error is much smaller than that of the non-claw counterpart, especially in the out-of-distribution regime. The training dataset of the medium and large datasets provide good coverage of the entire dataset and are aimed to test the in-domain predictability of various models.

Reviewer comment (Novelty):
There are missing related papers that aimed to address the same topic (enforcing conservation laws), which could raise questions about the comprehensiveness of this work, the literature review, and the originality of the idea. For example, one closely related work is "Neural Conservation Laws: A Divergence-Free Perspective" (https://arxiv.org/abs/2210.01741). Since the idea of the present paper is quite similar to that past work, the authors should have at least acknowledged the relationship. It would be more appropriate to have discussions about the similarities, the parts where the methods depart, and why this work was not used as one of the baselines. There are additional works that could have been at least covered in the literature review, such as: "Learning physical models that can respect conservation laws" (https://arxiv.org/abs/2302.11002) which proposes a projection method to enforce the integral form of conservation; "Learning differential solvers for systems with hard constraints" (https://arxiv.org/pdf/2207.08675.pdf), and more.

Author response:
**Related papers and our novelty**: We thank the reviewer's suggestion. We would like to point out that we have cited [1] and discussed the differences and limitations in their setting, in the last paragraph of Section 2 in the original manuscript. Per the reviewer's suggestion, we have also added [2] and [3] and discussed the differences between their approaches and our approach. Generally speaking, existing architectures [1-3] can only be applied on the known PDE-solving scenario where the full governing law is given, while in our framework we consider the physics discovery scenario where the constitutive laws are hidden and the model has to be learned from data. As a result, our approach can handle experimental data and is generalizable to different input functions, e.g., different initial conditions and boundary conditions. Another major difference (which is also the novelty in our architecture) is: in [1] the PINN architecture is used to approximate the solutions of this particular governing equation, i.e., mapping from x to the solution u(x). As such, the differential forms are approximated with automatic differentiation in NNs trivially. In our neural operator architecture, the differential forms can no longer be approximated via automatic differentiation, and therefore we propose a new architecture by attaching a pre-calculated numerical differentiation layer after the projection layer and provide error estimates in Section 3.2. We have added the above discussion, together with more details of [1-3], to the last paragraph of Section 2.

We would also like to point out that the novelty of our idea is NOT on formulating the conservation law into a divergence-free architecture with a differential form representation. In fact, such a representation is well-known for more than a decade, see, e.g., [4], way before [1]. We have added more references on differential forms in the revised manuscript, to avoid such a confusion. The novelty of our approach is 1) the combination of neural operator with the mass conservation as a basic physical law, to learn the unknown physical system responses such as the constitutive laws with improved data efficiency; and 2) the design of a pre-calculated numerical differentiation layer in the neural operator architecture, to enable the incorporation of differential form-guaranteed divergence-free output, without increasing the number of trainable parameters and computational cost. To our best knowledge, neither of these two ideas have been discussed in existing neural operator literature.

[1] Richter-Powell, Jack, Yaron Lipman, and Ricky TQ Chen. "Neural conservation laws: A divergence-free perspective." Advances in Neural Information Processing Systems 35 (2022): 38075-38088.

[2] Négiar, Geoffrey, Michael W. Mahoney, and Aditi Krishnapriyan. "Learning differentiable solvers for systems with hard constraints." The Eleventh International Conference on Learning Representations. 2022.

[3] Hansen, Derek, et al. "Learning Physical Models that Can Respect Conservation Laws." ICLR 2023 Workshop on Physics for Machine Learning. 2023.

[4] Barbarosie, Cristian. "Representation of divergence-free vector fields." Quarterly of applied mathematics 69.2 (2011): 309-316.

Reviewer comment (Experiments):
The authors did not provide sufficient information (preferably numerical experiments) to discuss the computational complexity of their method. As far as can be seen, they only mentioned in one sentence that their method has comparable network size and computational cost. A high-quality work requires exploring different aspects and reporting them properly.

Author response:
**Computational complexity and computational cost**: A standard FNO model consists of two fixed-size MLPs for the lifting layer and the projection layer, and $L$ numbers of Fourier layers between them. The kernel weight matrix in the Fourier domain takes up the majority of the model size, requiring $O(LH^2m^p)$ trainable parameters, where $L$ is the number of Fourier layers, $H$ is the latent dimension after lifting, $m$ is the number of Fourier modes on each dimension after truncation, and $p$ is the problem dimension. Since the differentiation layer is pre-computed and does not require any training, the complexity of clawNOs is the same as that of a standard FNO. To further verify this analysis and compare the computational cost, we have provided detailed performance comparison of all models in table 5 of the revised manuscript. The comparison metrics include the total number of model parameters, the per-epoch runtime, inference time, the peak GPU memory usage, and the L2-norm of the output divergence. One can see that the number of trainable parameters of clawNOs as well as GPU memory usage remain the same as their counterparts. Similar observation also applies to the runtime in large data regime. The inference time has a minor increase, due to the fact that the inferred solution will go through the an additional layer which avoidably increases the computational cost.

Reviewer comment (Experiments):
The paper doesn’t explain the setup of experiments well. For example, different test cases use different Ts, number of samples and other settings. Why different settings were chosen for each test case?

Author response:
**More details on experiments**: The experimental details can be found in the appendix (A1-A4 corresponding to the four experiments) given the page limitation of the main text. We also include more details on the training strategies in Appendix A5. Specifically, all the 2D models (only two spatial dimensions) are trained for a total of 100 epochs whereas all the 3D models (two spatial dimensions plus one temporal dimension) are trained for 500 epochs with an early stop if the validation loss stops improving for consecutive 100 epochs. 2D models are trained with less number of epochs as one training sample in 3D corresponds to $(T-T_{in})$ training samples in 2D. A detailed performance breakdown for all the models, including the total number of model parameters, the per-epoch runtime during training, the inference time, the peak GPU memory usage, and the L2-norm of the output divergence, is provided in table 5 of the appendix in the revised manuscript. In terms of the different settings for each test case, we have followed the settings of previously published and successful neural operator baselines and two PDE benchmark papers (PDEBench and PDEArena) and adopted the same settings as theirs with only two exceptions: The first exception is in the incompressible NS experiment where we decide to add one more challenging setting of $(T_{in}=5, T_{out}=25)$ in addition to the commonly adopted $(T_{in}=10, T_{out}=20)$ to further analyze the model accuracy under a longer time extrapolation. The 2nd exception is in the atmospheric modeling experiment, for which we have explained our choice of the setting in Appendix A3.

Reviewer comment (Reproducibility):
It would be good if the paper could provide more details in the training experiments. For example, how many training iterations it takes for the proposed methods and other methods. How much time it takes for each iteration in training and inference.

Author response:
**More details on experiments**: The experimental details can be found in the appendix (A1-A4 corresponding to the four experiments) given the page limitation of the main text. We also include more details on the training strategies in Appendix A5. Specifically, all the 2D models (only two spatial dimensions) are trained for a total of 100 epochs whereas all the 3D models (two spatial dimensions plus one temporal dimension) are trained for 500 epochs with an early stop if the validation loss stops improving for consecutive 100 epochs. 2D models are trained with less number of epochs as one training sample in 3D corresponds to $(T-T_{in})$ training samples in 2D. A detailed performance breakdown for all the models, including the total number of model parameters, the per-epoch runtime during training, the inference time, the peak GPU memory usage, and the L2-norm of the output divergence, is provided in table 5 of the appendix in the revised manuscript. In terms of the different settings for each test case, we have followed the settings of previously published and successful neural operator baselines and two PDE benchmark papers (PDEBench and PDEArena) and adopted the same settings as theirs with only two exceptions: The first exception is in the incompressible NS experiment where we decide to add one more challenging setting of $(T_{in}=5, T_{out}=25)$ in addition to the commonly adopted $(T_{in}=10, T_{out}=20)$ to further analyze the model accuracy under a longer time extrapolation. The 2nd exception is in the atmospheric modeling experiment, for which we have explained our choice of the setting in Appendix A3.