Paper: Dilated convolution neural operator for multiscale partial differential equations (ICLR-2024)

Reviewer comment (Evaluation):
The datasets and tasks seem relatively simple, and are presented without much detail. For example, no qualitative evaluations of the Darcy cases are presented, and it is left unclear how the averaged MSE quantities influence the obtained solutions.

Author response:
In the revision, we have provided more detailed information about the experimental datasets and setup.

Reviewer comment (Experiments):
FNO and dil-ResNet by themselves do worse than the DCNO. Potentially, there could be an ingredient in the combined architecture that yields the "best of both worlds", but the evaluations in the paper leave this point open. If the authors could shed light on where the advantages come from, maybe with a careful ablation or additional analysis, the paper would be much more convincing. In combination with the relatively narrow range of experiments and sparse evaluation, there is hesitance to try out the proposed architecture.

Author response:
We are delighted that the reviewer acknowledged the concept of achieving the "best of both worlds". In the following, we will conduct a meticulous ablation study to address this point. To accomplish this, we propose integrating the two Fourier layers within the Decode part into the Process part of the architecture. This consolidation allows us to vary the number of Fourier (F) and convolutional ( C ) layers in the architecture and examine their impact on performance. By comparing results across various configurations, we can ascertain that DCNO represents the optimal combination of F and C layers.

| F layers and C layers| Time per epoch(s)|Darcy rough($\times 10^{-2}$)|
| ------------------------------- | ----------------- | ----------------------- |
| 8 F layers                         |8.44 | 0.719 | 
| 7 F layers and 1 C layers(FCFFFFFF)|8.22 | 0.615 | 
| 6 F layers and 2 C layers(FCFCFFFF)|8.03 |0.563  | 
| 5 F layers and 3 C layers（DCNO)    |7.88 |**0.446** |
| 4 F layers and 4 C layers (FCFCFCFC)      |7.66 |0.571 | 
| 3 F layers and 5 C layers (FCFCCCFC)      |7.47 |0.500 |
| 2 F layers and 6 C layers (FCCCCCFC)      |7.27 |0.508 |
| 1 F layers and 7 C layers (CCCCCCCF)     |6.96 |0.548  | 
| 8 C layers                      |6.52 |3.197 |

It is important to note that we did not explore all possible combinations of F and C layers, and we adopted a notation like (FCFCFFFF) to denote the order of F and C layers.

Reviewer comment (Evaluation):
As the approach targets multi-scale PDEs, and employs a scale separation in the proposed loss, have the authors evaluated frequency-based evaluations and metrics?

Author response:
In Table 1 of the revised paper, we conducted the Darcy rough and multiscale trigonometric experments using both $L^2$ loss and (frequency based) weighted $L^T$ loss. These experiments were also evaluated using two different error metrics: the $L^2$ error and the (frequency based) weighted $L^T$ error.

Reviewer comment (Experiments):
Where do the time ranges used for the different Reynolds numbers come from? The difficulty seems to "jump" up and down while increasing Re, and details are missing about how these tests were set up. Are these single test sequences, or were results averaged over multiple sequences to obtain stable results?

Author response:
We refer to two papers [FNO](https://arxiv.org/abs/2010.08895) and [UNO](http://arxiv.org/abs/2204.11127), for the setup of time ranges with different Reynolds numbers adopted in this work. The general rule is to use a smaller simulation time for flows with higher Reynolds numbers. 

The observed variations in the results, which may appear as "jumping" up and down, could be attributed to different time ranges. We now perform an experiment using the same time range ($T_{in}=6, T=15$) for different Reynolds numbers: 

| Model | v=$1e^{-3}$            | v=$1e^{-4}$            | v=$1e^{-5}$            | v=$1e^{-6}$            |
| ----- | --------- | ------ | -------- | ------- |
| FNO  | $8.969 \times 10^{-4}$ | $9.116 \times 10^{-3}$ | $3.337 \times 10^{-2}$  | $5.280 \times 10^{-2}$ |
| DCNO  | $8.151 \times 10^{-4}$ | $5.483 \times 10^{-3}$ | $1.707 \times 10^{-2}$ | $3.227 \times 10^{-2}$ |

Based on the experiment results, it is consistent that the models exhibit higher accuracy for flows characterized by smaller Reynolds numbers.

For the Navier-Stokes equations, the train-val-test split dataset has 5000, 500, and 500 samples. The results are averaged over 500 test datasets.

Reviewer comment (Novelty):
Apart from modifications to the FNO architecture, this paper has limited novelty and insights.

Author response:
We acknowledge that we may not have sufficiently emphasized the novelty of our method. The observed improvement in accuracy can be attributed to an optimal combination of modules that excel at extracting either low-frequency or high-frequency features. By effectively integrating these modules, our approach achieves superior performance compared to existing methods. Please also refer to the responses in "Common Concerns" and to the responses to the following specific questions.

Reviewer comment (Presentation):
The page formatting, page margins, and/or the aspect ratio of this paper are clearly different from other submissions. This is especially noticeable due to the amount of whitespace under the page number. However, since most other formatting aspects are reasonably consistent, it is unclear if this counts as a template/formatting violation.

Author response:
We are very greatful to the reviewer for pointing out the formatting issue, we have made corrections to address those issues.

Reviewer comment (Evaluation):
**E2:** The claim that DCNO performs better than previous methods in the high-frequency domain due to dilated convolution layers is not well supported. Fig. 1b only compares to FNO, which is known to perform poorly on high frequencies. In Fig. A.1, DCNO performs only on par with other methods. Fig. 1c may be misleading, as DCNO was trained with a weighted loss focusing on high frequencies while FNO in Fig. 1d was not; without this loss, DCNO performs significantly worse (see Fig. C.1). Other baselines (especially MWT or HANO) may even perform better in lower frequencies, as seen in Fig. C.1. Also, a model without dilations achieves similar performance to DCNO (Table 4). Additional evaluations, such as frequency analyses on Navier-Stokes cases using established techniques (e.g., *Turbulent flows*, Pope), would be a necessary improvement to determine if DCNO fundamentally improves high-frequency behavior.

Author response:
Thanks for the suggestion. In the revised paper, we have made a fair comparison by evaluating all models using the $L^2$ loss, as shown in Figure 1.1 and Figure A.1. The results clearly demonstrate that DCNO consistently achieves the highest accuracy among all the models, with or without the weighted loss. 

Furthermore, we agree that it is a little difficult to compare different models from Figure C.1. To provide a clearer representation of the loss dynamics, we have included Figure 1.1 (c,d) in the revision. In these figures, we decompose the error into low-frequency ($\leq 10\pi$, Figure 1.1.c) and high-frequency ($\geq 10\pi$, Figure 1.1.d) components. It is evident that DCNO surpasses most models (only HANO exhibits similair performance but with a training time more than 3 times longer than DCNO) in both the low-frequency and high-frequency components, further highlighting its superior performance. 

We have made updates to Table 4 in Appendix B.2. According to the revised table, for the Darcy rough case, a model with no dilation, denoted as (1) exhibits an $L^2$ error of 7.63%. With two layers (1,3), the error reduces to 5.27%, and with three layers (1,3,9), the error further decreases to 4.46%. This implies that the (1,3,9) configuration reduces the error by over 40%. Additionally, we not only observe an improvement with increasing layers but also note that the (1,1,1) setup has an $L^2$ error of 0.511, which is 15% higher compared to the (1,3,9) setup. This difference accounts for the improvement resulting from increasing dilation rates.

We note that we carry out those studies for the Darcy case as Navier-Stokes cases require more time to train. In-depth investigation of the Navier-Stokes case will be conducted in future studies.

Reviewer comment (Experiments):
**E4 (less crucial):** If Appendix B.1 is interpreted correctly, all results are from test sets randomly split from training data. For PDE simulations, generalizing to new parameters and initial conditions is highly desirable. Applying trained models to new parameter ranges (e.g., different Reynolds numbers for fluid flow) would be interesting.

Author response:
We train the model on the Navier-Stokes equation and test the model under different Reynolds numbers with $T_{in}=6, T=15$ .
| DCNO ($T_{in}=6, T=15$) | test v=$1e^{-3}$ | test v=$1e^{-4}$ | test v=$1e^{-5}$ | test v=$1e^{-6}$ |
| ----------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
| train v=$1e^{-3}$       |  | 0.3531   | 0.4038    | 0.4115    |
| train v=$1e^{-4}$       | 0.4056    |      | 0.0923      | 0.1123   |
| train v=$1e^{-5}$       | 0.4564   | 0.0885      |      |  0.0402     |
| train v=$1e^{-6}$       | 0.4554       | 0.1031      | 0.0262 |       |

It is evident that, in general, models trained with viscosities closer to the test cases exhibit reasonably good performances. An interesting observation is that all models trained with smaller viscosities, corresponding to higher Reynolds numbers, do not perform well for the $v=1e^{-3}$ case.

Reviewer comment (Experiments):
**E5 (less crucial):** In the Navier-Stokes case, the data is downsampled from $256 \times 256$ to $64 \times 64$. This may average out the effects of higher Reynolds numbers. It would be interesting to see visualizations and predictions at different Reynolds numbers from this dataset.

Author response:
In the revision, we have included a comparison between the Navier-Stokes (NS) solutions and DCNO predictions of the vorticity field, with viscosity $\nu=10^{-5}$, in Figure 4.3.

Additionally, we add videos for the NS solutions and DCNO predicitons at different Reynolds numbers in the supplementary materials.

Reviewer comment (Presentation):
There are minor presentation issues and the appendix structure could be improved. Minor issues include:  
- “allows us to selectively skip” → “allows for selectively skipping” (3rd paragraph of Introduction)  
- “convoluted” → “convolved” (several times in C layers description)  
- The statement “which maps the vorticity from time 0 to T0 to the vorticity from time T0 to a later time T” is unclear (Section 4.2)  
- The statement “Each convolution layer includes three convolutional neural networks, each” is unclear (C layers description)  
- Several citations lack publication information (e.g., *Cai et al., 2019*, *Freese et al., 2021*, *Liu et al. 2023*)  
- Citation formatting is inconsistent (e.g., arXiv citations appear as “arXiv e-prints,” “ArXiv,” “CoRR,” etc.)  
- Fig. 4.1 should have log scaling on the colorbar  
- Fig. 4.2 should have log scaling on the plot axis description  
- Subplots are plotted inconsistently across figures

Author response:
Thanks for pointing out those presentation issues and we have addressed them in the revised paper.