Paper: Learning from the Future: Improve Long-term Mesh-based Simulation with Foresight (ICLR-2024)

Reviewer comment (Writing):
The writing of the paper is poor, with many duplicated expressions. For example, in Section 4.1: "which are flexible to produce flexible outputs at any given timestamp".

Author response:
A1. Thanks for your comment. We have corrected the typos and corrected the expressions.

Reviewer comment (Writing):
The paper lacks proper explanations and details. For example, there is no explanation for the "multi-task learning" that is used to train the model.

Author response:
A2. Thanks for your comment. Here, we generate predictions with different prediction lengths, each of which corresponds to a task. These tasks share the same encoder and decoder. We have included the explanation to clarify the introduced concept.

Reviewer comment (Presentation):
The paper has poor organization and missing labels for figures. For example, in Fig. 5, it is unclear what "a, b, c, d" refer to, and the colors in Fig. 5 are not clear enough to distinguish different settings.

Author response:
A3. Thanks for your comment. We have included "a,b,c,d" in Figure 5 and make the figure more clear.

Reviewer comment (Experiments):
Can the coarse-refine framework apply to the baselines? Will the proposed framework benefit them also?

Author response:
A4. Thanks for your comment. It would be out-of-memory for any baselines since they adopt the rollout methods for long-term predictions, which could include a large computation cost to output a range of predictions at the same time. In contrast, our FAIR utilizes a graph ODE to generate **coarse predictions** to release the efficiency issue, which makes our framework feasible.

Reviewer comment (Experiments):
The proposed model is closely related to GMR-GMUS (Han et al., 2022), which similarly aims to enhance long-term prediction in mesh-based simulators. Notably, GMR-GMUS achieves long-term predictions for 400 time steps in the case of Cylinder Flow. Including a comparison between FAIR and this approach in the experiments would be valuable.

Author response:
A1. Thanks for your comment. We are willing to compare the method with our FAIR but their source code is not publicly available. Due to the limited time, we are unable to finish re-implementing their code according to the algorithms. We therefore only cited and discussed these important baselines in Section 2 in our updated paper.

Reviewer comment (Presentation):
In Table 1, for the DeformingPlate experiment with prediction lengths of 50, MS-GNN-Grid has the best result with an RMSE of 2.78. It would be beneficial to highlight this result in bold and provide some discussions hopefully.

Author response:
A2. Thanks for your comment. We have highlighted the results and included the following discussion:  
"Moreover, the performance of our FAIR on MS-GNN-Grid is a little worse than MS-GNN-Grid. The potential reason is the high complexity of DeformingPlate makes it harder to generate accurate foresight, which could deteriorate the model performance."

Reviewer comment (Presentation):
In Figure 5, it would be helpful to annotate the four sub-figures with (a-d) to improve reference.

Author response:
A3. Thanks for your comment. We have included these annotations as you suggested.

Reviewer comment (Writing):
The paper includes duplicate notations for $L$, referring to both the prediction length and the number of stacked layers.

Author response:
A5. Thanks for the comment. We have corrected this by including letters $L^p$ and $L^s$.

Reviewer comment (Writing):
I am a little confused with the equation $t_l = t_0 + rl − r + 1$ in Section 4.1. What does step size mean in the paper?

Author response:
A6. Thanks for the comment. The step size means the interval for coarse predictions, which is $r$ in $t_{l}=t_{0}+r l-r+1$.

Reviewer comment (Theory):
In Section 4.2, it would be beneficial if the authors could provide an explanation for their choice to compute the offset in the observation space, as opposed to calculating an updated latent vector in the latent space.

Author response:
A7. Thanks for the comment. Firstly, utilizing observation space is more efficient due to low dimension. Moreover, the calculation in the observation space is more close to interpolation.

Reviewer comment (Writing):
In Eq.(11), does the notation $\{z_i^{t_1}\}_{i \in \mathcal{V}}$ indicate that the offset of node $i$ is computed using representations from all nodes in the set $\mathcal{V}$?

Author response:
A8. Thanks for the comment. For efficiency consideration, we only introduce the node itself without neighborhood information, which has been utilized for coarse prediction generation.

Reviewer comment (Writing):
The line of argumentation is in some cases weird, unclear, vague, or lacks citations, especially in section 3 and 4:
- “Given these limitations….” (difficult to understand, top of page 2)
- “Our graph ODE model is flexible to generate…” (sound weird, middle of page 2)
- “The model would be evaluated…” (sound weird, end of section 3.1)
- “However, they are inferior in modeling…” (unfounded/missing citation, end of section 3.2)
- “To capture the hierarchy structure among meshes…” (unclear, end of section 3.3)
- “As a foundational step, it is…” (sounds weird, begin of section 4.1)
- “To learn interacting dynamics…” (very unclear, begin of section 4.1)
- “Furthermore, we notice that data-driven…” (unfounded/missing citation, middle of section 4.1)
- “Firstly, our FAIR…, Secondly, our approach…” (unclear, end of section 4.1, very similar at the begin of section 4.3)
- “multi-task learning framework” (unclear, what other task?, frequently across the paper)

Author response:
1. “Given these limitations (on top of page 2)...”; we have refined our writing to clearly reflect that we are tackling the accumulated errors due to iterative rollouts.
2. “Our graph ODE model is flexible to generate…”; we have refined it to “Our model has the flexibility to generate the predictions at any steps ahead…”
3. “The model would be evaluated by calculating the prediction errors”; We use prediction errors w.r.t. the ground truth to evaluate the performance. 
4. “However, they are inferior in modeling...”; We have deleted the discussion on previous methods and merged them into related works since this section is for methodology.
5. “To capture the hierarchy structure among meshes…”. Here is a typo, we have deleted that sentence. 
6. “As a foundational step, it is crucial to…”; We have revised it into "As a preliminary step"
7. “To learn interacting dynamics…”; We have revised into "To learn the evolution between mesh points". 
8. “Furthermore, we notice that data-driven…”; we have added the citations. 
9. “Firstly, our FAIR…, Secondly, our approach… (appeared repeated in 4.1 and 4.3)”; We thank the reviewers for pointing out the confusion in our description. We have revised the presentation. 
10. “Multi-task learning framework”; Our multi-task learning refers to the inference at the multiple time step forward into the future of the teacher model.

Reviewer comment (Writing):
Section 4 describing the methodology has structural issues, is generally difficult to follow, and it is not clear to me how the proposed method actually works in detail. Similar problems can be found across the other parts of section 4, but here an example issue from section 4.2: The term interpolation might be misleading for the refinement step. To my understanding, the refinement is mainly used to improve the outputs at discrete timesteps where a coarse prediction already exists. This refinement seems to be necessary to stabilize recurrent applications. Please clarify the refinement procedure in the paper.

Author response:
11. P2: Section 4 will be largely rewritten as mentioned in 9.
12. P3~P4: We thank you for pointing out and will illustrate them better

Reviewer comment (Experiments):
Please provide some physical characteristic numbers, i.e. Reynolds and Mach numbers, for the Navier-Stokes cases. This would greatly improve the comparability of this work to future research. In particular, it would be interesting to know whether the Aerofoil case includes the transonic regime. The conclusion section briefly mentions that the method currently fails when predicting non-steady dynamics. Three out of the four datasets were originally introduced in *Pfaff et al. (2021)*, and include a variety of steady-state as well as transient cases. To me, it is unclear whether this full dataset was used in training and testing, or whether only a small subset with steady-case simulations was used.

Author response:
13. E1, E5: we use the full dataset of Pfaff et al. (2021). We did not evaluate any steady-state problems and would like to clarify to the reviewer that we only use a longer, coarser prediction in the future. We did not predict the steady-state in too-long future.

Reviewer comment (Reproducibility):
Only after closer inspection of the appendix and the reproducibility statement (*“The experimental results of the baselines are consistent with Cao et al. (2023).”*) it becomes apparent that the entirety of the baseline results in Tab. 1 are directly copied from *Cao et al. (2023)*. Nevertheless, it is stated throughout the paper and appendix that the authors use their implementation of these methods. In my opinion, this is highly problematic and I want to urge the authors to clearly state the source of the values in the table heading. A fair evaluation against FAIR would require the usage of one consistent setup for all methods, as it is possible that e.g. details in the error aggregation change results.

Author response:
First, we **claimed** our results are consistent with Cao et al. in **REPRODUCIBILITY STATEMENT** **in our first version**. Therefore, we didn't have **Ethics Concerns**.

Reviewer comment (Evaluation):
I have serious doubts regarding the soundness of these quantitative evaluations. In three out of four cases, the paper reuses the datasets from *Pfaff et al. (2021)* (i.e. CylinderFlow, Airfiol, DeformingPlate), and proceeds to use the MeshGraphNet trained within that publication as the main comparative baseline for visualizations. However, all results in Tab. 1 significantly differ (sometimes by more than an order of magnitude) from the ones reported by *Pfaff et al. (2021)* for the very same datasets (e.g. CylinderFlow RMSE on 50 steps: 43.9e-3 here vs. 6.3e-3 from *Pfaff et al.*). At the same time, the limited information in the appendix suggests that exactly the same architecture (e.g. number of message passing layers) was used as in *Pfaff et al. (2021)*. No explanation is given for these large differences in the error metrics.

Author response:
14. E2: We thank the reviewers for mentioning the soundness of our data source. The major problem we encountered is to strictly replicate the MeshGraphNet based on their open-source code. There exist multiple inconsistencies between their article and the code, one biggest issue we noticed is the noise added to the trajectory. While the article mentioned they use random-walk noise, in the code (https://github.com/google-deepmind/deepmind-research/blob/f5de0ede8430809180254ee957abf36ed62579ef/meshgraphnets/dataset.py#L75) they use normal distribution (without integrate on the time dimension to replicate random-walk effect). Another difference is that in the paper, the authors mentioned they drew a small box (without detailed dimensions) around the airfoil to only calculate the RMSE inside; however, this is not reflected in the code. We re-implemented based on the open-source TensorFlow code but found much worse results compared to the original one. There are multiple replications online while Cao et al. (2023) have the most comprehensive coverage on experiments that are conducted under the same, consistent setting, and they also provide a new dataset. That’s the reason why we adopted their work.

Reviewer comment (Reproducibility):
In the original MeshGraphNet visualizations (https://sites.google.com/view/meshgraphnets#h.p95069gdfedm) all the prediction results are a lot closer to the ground truth than what is shown here. This raises concerns about the reimplementation and training of the model (as both architecture and data set are claimed to be identical with the work from *Pfaff et al. (2021)*).

Author response:
15. E3: 
- We visualized the composited velocities in Fig 3 and Fig 4., respectively.
- We will re-print based on different timestamps to visualize our model’s capability on long-term predictions
- Considering the soundness of the original MeshGraphNet, we would like to re-emphasize the point already made in 14.
- Fig 14.a. Plot was mistakenly plotted with another component, hence we would like to fix that in the refined version.

Reviewer comment (Reproducibility):
In Fig. 5a: For which fixed L was the sweep across r computed?

Author response:
16. E4:
- We appreciate that you mentioned adding more clarity and would like to provide the following in the revised figure: the fixed L, and r when the other parameter is swiping.

Reviewer comment (Presentation):
In Fig. 5c: I would suggest plotting this in log-scale.

Author response:
16. E4:
- Sure, we would love to revise the plot’s y-axis to log the scale.

Reviewer comment (Evaluation):
The scope of this paper is limited to steady-state predictions, where no temporal changes should occur once the solution converges. In this context, it is questionable whether fine temporal resolutions are at all desirable. Thus, the temporal interpolation capabilities have little value in the current state of the paper. Some existing approaches can even directly predict the steady state solution without intermediate steps at all. Due to the steady-state test cases, the evaluations in this paper mainly focus on:
1) whether a particular model architecture can calculate the steady state in the first place, and
2) whether the model can stop the solution from drifting over long horizons when the steady state is already reached.

Author response:
Thanks for your comment. Actually, extensive literature [**Xu Han, Han Gao, Tobias Pfaff, Jian-Xun Wang, Li-Ping Liu**, Predicting Physics in Mesh-reduced Space with Temporal Attention, ICLR 22] studies long-term predictions for mesh-based simulations. **Are you saying these previous contributions in the field are meaningless?** Moreover, we have clearly stated that we did not predict the steady-state in the too-long future. In other word, no steady state is reached in our problem.

Reviewer comment (Writing):
From the paper and figures I assume that the predictor outputs a series of temporally coarse frames, which are then predominantly used to refine the first prediction. The procedure is then repeated with this predicted and refined first prediction. To me, this means that $L>1$ predictions are made to advance the simulation by one frame. Is this the case? This should also be clarified in the paper.

Author response:
17. Q1: Yes; your suggestion is accurate, we will make it clear in the revision

Reviewer comment (Theory):
*Pfaff et al. (2021)* show that their method was stable across many steps. In Section 3.2 it is mentioned that *“[...] they are inferior in modeling long-term interacting systems that are continuous in nature and could suffer from severe error accumulation when making long-term predictions”*. Can you elaborate on this, especially regarding the inherent inferiority of GNNs?

Author response:
18. Q2: Considering the soundness of the original MeshGraphNet, we would like to re-emphasize the point already made in 14.

Reviewer comment (Theory):
In Section 4.1: *“Instead of using inefficient iterative rollouts [...]”* - It is not entirely clear to me how iterative approaches are inherently inefficient at solving PDEs. To my understanding, even the method presented in the paper fundamentally relies on iterative temporal advancement of the solution.

Author response:
19. Q3: Claiming the iterative rolling out is inefficient was indeed a mistake, we will refine it better

Reviewer comment (Writing):
In Section 4.1: *“While previous approaches often integrate neighborhood information into ODEs to model interacting dynamics, they typically fall short in explicitly capturing the evolving dynamics of edges”* - I do not fully understand this statement. Edges are not an inherent characteristic of the vector fields described by ODEs. Can you specify what you mean?

Author response:
21. Q5: Here we intended to transit from NODE to graphODE. We will refine in our revision that “GraphODE is more suitable to explicitly capture the interaction on graph/between nodes via edges”

Reviewer comment (Reproducibility):
In Section 4.2: The loss formulation $\mathcal{L}_{re}$ in equation (13) optimizes the refiner to match a ground truth $x^{t_1}$. The graph-ode predictor has, however, produced a series of future frames ($L$ to be precise). Is the refiner never trained to refine these steps?

Author response:
22. Q6: The L coarser future states are produced to be the input for the refiner, hence the refiner is still trained.

Reviewer comment (Experiments):
The current experiments provided in the paper are mainly in 2D. It is unclear whether the proposed method can exhibit similar performance when applied to larger scale 3D cases.

Author response:
A1: Thanks for your comment. Actually, our experiment includes deformation plates and inflating fonts, which are 3D cases. Particularly, the inflating font contains 10s of thousands of nodes which are the largest among all cases. We have included a 3D visualization to show our performance for 3D mesh-based predictions.

Reviewer comment (Writing):
A "multi-task learning framework" is mentioned multiple times in the paper for the first stage training. Does it indicate the encoder and decoder as multiple tasks?

Author response:
A3. Thanks for your comment. Here, we generate predictions with different prediction lengths, each of which corresponds to a task. These tasks share the same encoder and decoder.

Reviewer comment (Theory):
As the equations to solve in the experiments are generally PDEs, however the coarse long-term predictor i.e., Graph ODE is designed based on ODE. I am wondering are there any intuitions for how a ODE based neural solver can be applied on PDEs?

Author response:
A4. Thanks for your comment. Compared with ODE, PDE additionally includes both spatial derivates such as $u_x$ and $u_u$, which correspond to the relation between nodes and their neighbors. Therefore, combining neural ODEs with graph neural networks can simulate PDE solvers.