Paper: Deep PDE Solvers for Subgrid Modelling and Out-of-Distribution Generalization (ICLR-2024)

Reviewer comment (Novelty):
The proposed framework does not show enough novelty. The main adaptation in the architecture is simply embedding a discretized Laplacian structure in the convolution filters in order to reduce the total number of parameters in the model.

Author response:
Thank you for your review and valuable feedback! 

We note that our paper is intended as a proof-of-concept architecture and we show a result which has not been demonstrated by other methods, which is Out-of-Distribution generalization. Furthermore, our paper shows results based on data sampled from different Fourier spectra, which is also an improvement over many other published papers where the distributions are essentially modelled by one parameter.

Reviewer comment (Experiments):
The problem setting is too simplistic to demonstrate significance. The authors exclusively focus on the linear heat equation with nonhomogeneous unknown coefficients, which is nowhere near the complexity of the other problems referenced, including Navier-Stokes and weather/climate. The heat equation has linear dynamics and generally admits smooth solutions, which means errors do not accumulate over time in complicated ways and solution has very little small-scaled features (so that representing the solution/dynamics fields is relatively easy). The claim that it is "complex enough" (page 4 top) is too much of a stretch. It is certainly not evident that the same OOD generalization might be observed on, for example, turbulent flows.

Author response:
What we meant when mentioning that the modelled problem is complex enough is that it is sufficient to highlight the results of our proof-of-concept architecture and shows that the proposed method works and is promising for CWM. Our goal is to show that this is a starting point to solve harder problems (like Navier Stokes) but we are not claiming to have solved them. We are currently working on extending the results to general advection-diffusion equations and the goal is to further extend it to all parabolic PDEs.

Reviewer comment (Experiments):
Baselines: FCN should not be considered a valid baseline if the dynamics is known to be local. In addition, both FCN and ConvN show signs of overfitting, which suggests the architecture/training is sub-optimal. Techniques such as layer/group normalizations and learning rate annealing should at least be considered.

Author response:
Concerning the baselines used, we picked a standard fully connected net and a standard convolutional net as proxies for off-the-shelf ML models. We do agree though that a comparison with more complex baselines would be beneficial, and we aim to add more experiments to compare our model.

Reviewer comment (Writing):
Potential typo second line of page 7: "sample it up in space" - do you mean sample it down in time?

Author response:
As for the typo, it has been fixed, thanks for the catch! (we did mean "sample it down in time")

Reviewer comment (Writing):
Not really clear what the ML part is here or if there even is one

Author response:
The model proposed is a deep neural network that is trained with ML algorithms to learn the parameters of the Model. The model architecture is explained in section 4 (specifically equation 3 and the paragraph above it). We appreciate the feedback and will consider adding a graphic for visualization.

Reviewer comment (Experiments):
Not clear if this method scales to larger scale problem (which any relevant CWM problem would be)

Author response:
The goal of our research is to show that the method does scale to larger-scale problems. In the paper, we show results for a specific dynamic (heat equation with non-constant coefficients) as a proof-of-concept that the proposed method works and is promising for CWM. We are currently working on extending the results to general advection-diffusion equations and the goal is to further extend it to all parabolic PDEs.

Reviewer comment (Writing):
Abstract should not include references

Author response:
The architecture proposed is novel and it is based on expert knowledge of the structure of PDE solution operators, which is what is mentioned in the abstract. We have also removed the citations from the abstract.

Reviewer comment (Writing):
Abstract does not say anything about the type of ML approach that is going to be used, e.g. architecture

Author response:
The architecture proposed is novel and it is based on expert knowledge of the structure of PDE solution operators, which is what is mentioned in the abstract. We have also removed the citations from the abstract.

Reviewer comment (Presentation):
No conclusion section, please add, especially given that there is space left up to the page limit of 9 pages

Author response:
We explain our results throughout the paper and focus on the main takeaways in the introduction. We did not opt for a conclusion section to avoid sounding repetitive.

Reviewer comment (Theory):
Why is OOD in the Fourier space? It would be nice to have an explanation about why this is the relevant OOD case

Author response:
There are many types of OOD generalization relevant to CWM and in our paper, we focus on initial data with a different shape from data previously seen by the models (as explained in section 3.3). This corresponds to the problem of having the same physical dynamics, but a different distribution of the density of particles (e.g. a more oscillatory density profile). Thus, to represent datasets with different shapes, we generate samples from different Fourier Spectra.

Reviewer comment (Reproducibility):
Details of the neural network training were missing, including optimizer, learning rate, stopping criteria, and hyperparameter optimization. In the appendix, the authors comment on the learning dynamics but it is impossible for such a comparison to be meaningful without details of the learning algorithms.

Author response:
We have attached the code as supplementary material including the details of the optimization. More specifically, all models were trained with the Adam optimizer for 1000 epochs and no weight decay. We have ensured that the training parameters are equal for all models to ensure fair comparisons, but the optimization itself is not the objective of our paper.

Reviewer comment (Writing):
In the introduction, what does out of distribution data mean in this context?

Author response:
We explain in the last paragraph of section 3.3 the meaning of Out-of-Distribution data: "Here we consider OOD to be initial data with a different shape (Fourier spectrum) from data previously seen by the models. This corresponds to the problem of having the same physical dynamics, but a different distribution of the density of particles (e.g. a more oscillatory density profile)."

Reviewer comment (Writing):
What is the use case? I understand that this is for situations where the PDE is unknown (Section 3.2), and the training data consists of one (or more?) gridded data example.

Author response:
Correct, the use of our approach is to learn a solution operator for an unknown PDE given observations of solutions in a given coarse grid. What we show is that having some prior knowledge of the physical process behind it (in this case knowing it behaves like a heat equation but we don't know the coefficients) can help us engineer an architecture that produces a better model than standard ML approaches. We are currently working on extending our results to general advection-diffusion equations and the goal is to further extend it to all parabolic PDEs.

Reviewer comment (Experiments):
If we don't know the PDE coefficients anyway, why not just coarsen the training data into the desired grid and learn the PDE coefficients directly?

Author response:
In reality, we will not have access to fine-grid training data, so we are more interested in having a model that learns a solution operator rather than coefficients. We note that while we did observe that the coefficients learnt were a  (non-linear) average of the ground truth coefficients, this might not be the case when we extend our work to more complex dynamics.