Paper: HelmSim: Learning Helmholtz Dynamics for Interpretable Fluid Simulation (ICLR-2024)

Reviewer comment (Writing):
It is unclear what the observed state is at all, "observed fluid" could be velocity, pressure, or density fields. This should be mentioned.

Author response:
> **Q1:** It is unclear what the observed state is at all.

Following the reviewer's suggestion, we have added the observed state information in $\underline{\text{Appendix A.1 of revised paper}}$.

Reviewer comment (Evaluation):
The experimental results are reported as single relative L2 errors. Are these compared over networks trained on various random seeds, or how was this single value chosen? Ideally we can see a statistically significant improvement with the proposed model.

Author response:
> **Q2:** Are these compared over networks trained on various random seeds, or how was this single value chosen?

Sorry for this missing information. We repeated all the experiments three times with seeds randomly selected from 0 to 1000. The presented results are the mean value of three repeated experiments.

Experimentally, the standard deviations of relative L2 are smaller than 0.001 for Navier-Stokes, Bounded N-S and Sea temperature and smaller than 0.003 for Spreading Ink. For scientific rigor, we keep four decimal places for all results. We have included this in $\underline{\text{Appendix A.2 and A.3 of revised paper}}$.

Reviewer comment (Writing):
The last sentence on page 4 should have hats on the states $x_T$ and $x_{T-1}$, is that correct?

Author response:
> **Q4:** Typo of the last sentence on page 4.

Thanks for the suggestion. We have corrected this typo in the $\underline{\text{revised paper}}$.

Reviewer comment (Evaluation):
Figure 9 shows an efficiency comparison, is it correct to assume the y-axis performance is equivalent to accuracy of the model? Higher is better? Why would HelmSim be at the lower end of the spectrum then? Especially since it's the second slowest model in runtime.

Author response:
> **Q5:** Explanation of Figure 9: Efficiency comparison.

Sorry for the misleading y-axis label. The y-axis is Relative L2, which means that a smaller value refers to better performance. We have rephrased this figure in the $\underline{\text{revised paper}}$.

Reviewer comment (Experiments):
The number of parameters is different between each model, it would make sense to make a fair comparison where the same number of parameters was provided for each model. In this particular case, FNO was provided roughly 9x less parameters to train on. The number of parameters should ideally also be mentioned in the main text.

Author response:
> **Q6:** FNO with roughly 9x less parameters to train on compared to HelmSim. Compare HelmSim to FNO with similar model size.

Firstly, we want to emphasize that all the baselines are compared with their official configurations in their paper, which means FNO is proposed as a lightweight model.

Following the reviewer's suggestion, we enlarge the FNO parameter and compare it with HelmSim as follows. We can find that, even with sufficient model parameters, FNO still performs worse than HelmSim.

| Model                       | Relative L2 in 64 × 64 Navier-Stokes | Model parameter |
| --------------------------- | ------------------------------------ | --------------- |
| FNO (offical configuration) | 0.1556                               | 1,188,641       |
| FNO (enlarge model)         | 0.1524                               | 10,633,265      |
| HelmSim                     | **0.1261**                           | 9,929,101       |

Reviewer comment (Evaluation):
Figure 10 should be presented in a more quantitative manner. The boundary condition errors do not look significantly different, it would help to know what sort of percentage difference on average it has on the overall prediction error.

Author response:
> **Q7:** Figure 10 should be presented in a more quantitative manner.

Actually, as we stated in the ablation paragraph, we have already included the quantative results in $\underline{\text{Table 8 of original submission}}$. For clarity, we also add the quantitive results to the $\underline{\text{Figure 10 of revised paper}}$.

Reviewer comment (Writing):
It was unclear until the multiscale integration network section that we were operating with a Lagrangian and not a Eulerian fluid learning setup, since we are learning a velocity field and using this to advect field quantities. Is this interpretation correct?

Author response:
> **Q9:** Whether Lagrangian or Eulerian fluid learning setup do you use?

We adopt the Eulerian learning setting. The dynamic tracking visualization is just to measure the model performance from a new persepective.

Reviewer comment (Evaluation):
In the experimental results, the authors mention that the blurry predictions from U-Net "impede its practicability". In what application areas will these models be applied?

Author response:
> **Q10:** Blurry predictions from U-Net "impede its practicability". In what application areas will the predictions from HelmSim and other baselines be applied?  

The experimental benchmarks clearly present the applications. For example, in the Sea Temperature benchmark, temperature prediction can provide guidance to the fishery industry. A blurry prediction will fail to provide instructive information to this application due to the inaccurate results in predicting extreme values.

Reviewer comment (Experiments):
The argument for why directly estimating the velocity will overwhelm the model in the ablation study is not very convincing. Is that not what the other baseline models are already capable of?

Author response:
> **Q11:** Why directly estimating the velocity will overwhelm the model in the ablation study is not very convincing.

As we presented in $\underline{\text{Figure 1,6,8,10 of original submission}}$, directly learning velocity will make the model fail in learning correct dynamics, especially the divergence-free part. However, the divergence-free part is well covered by the learning stream function in HelmSim.

The quantitive results are also provided in $\underline{\text{Table 9 of original submission}}$. We also place the comparison here, where learning Helmholtz Dynamics can boost the performance with a little extra computation costs.

| 64 × 64 Navier-Stokes Dataset      | Relative L2 | GPU memory (MiB) | Running Time (s/epoch) |
| ---------------------------------- | ----------- | ---------------- | ---------------------- |
| Directly learning velocity         | 0.1412      | 14.86            | 72.18                  |
| Learning Helmholtz Dynamics (Ours) | **0.1261**  | 16.30            | 80.20                  |

Reviewer comment (Experiments):
The paper claims that Vortex (Deng et al., 2023) cannot be tested on other datasets, which seems unusual, as they are the same type of task and data that are disconnected from the choice of dynamics modeling itself. It should be further clarified why Vortex cannot be applied to other datasets.

Author response:
> **Q3:** Why Vortex cannot be applied to other datasets besides Spreading ink dataset?

Vortex models multiple vortex trajectories as a function of time. Since different video sequences have inherently different vortex trajectories, we need to re-train Vortex to fit every video sequence. However, the other three benchmarks, for example, the Navier-Stokes dataset, have more than 1000 different video sequences. It means that we need to train 1000+ Vortex models for these benchmarks, which is unacceptable. But as per your request, we still implement this experiment in $\underline{\text{Table 15 of revised paper}}$, where we train one vortex model on one single video sequence and generalize it to others. Here are part of the results. We can find that Vortex degenerates a lot.

| 64 × 64 Navier-Stokes Dataset | Relative L2 |
| ----------------------------- | ----------- |
| Vortex                        | 8.1379      |
| HelmSim                       | 0.1261      |

Reviewer comment (Evaluation):
The paper's claim about interpretability is not well-explained. If the interpretability claim is based on the model's prediction of an explicit term of velocity, it needs further comparison and a more comprehensive explanation. Does the Helmholtz decomposition significantly improve interpretability compared to baseline models, such as Vortex (Deng et al., 2023)?

Author response:
> **Q4:** Does the Helmholtz decomposition significantly improve interpretability compared to baseline models?

As we keep emphasizing in defining "Helmholtz dynamics" in $\underline{\text{Section 3.1}}$, HelmSim not only learns the velocity field but also estimates the potential and stream functions of fluid. We believe that this design surpasses other baselines in learning "physically interpretable evidence", not just the velocity fields.

Reviewer comment (Evaluation):
In Figure 4, it appears that the model predicts incoherent velocity fields around the circle boundary, even with non-zero velocity outside the boundary, while baseline models do not exhibit such artifacts. This weakens the interpretability claim.

Author response:
> **Q5:** In Figure 4, the model predicts non-zero velocity outside the boundary.

Sorry for this misleading visualization. Since the Spreading Ink dataset is only within the circle boundary, we only compute the loss within the boundary. Thus, the part outside of the boundary is without restriction.

We have added the implementation details in the $\underline{\text{Appendix A of revised paper}}$.

Reviewer comment (Reproducibility):
The aggregation operation after "Integration" needs further clarification. Please provide more details in the main paper, and if you refer to other architectures, acknowledge their structure properly.

Author response:
> **Q6:** The aggregation operation after "Integration" needs further clarification.

We conduct this operation following U-Net. As per the reviewer's suggestion, we have included details in the $\underline{\text{Appendix A.2 of revised paper}}$.

Reviewer comment (Presentation):
Ensure proper citation format for baseline models (Authors, Year).

Author response:
> **Q9:** Ensure proper citation format for baseline models.

Thanks for the suggestion. We have corrected all the citations of baselines in the $\underline{\text{revised paper}}$.

Reviewer comment (Theory):
Are there additional insights or reasons for employing multi-head integration beyond the expected capacity and performance improvements? It would be helpful to understand the broader intuitions behind this approach.

Author response:
> **Q11:** Are there additional insights or reasons for employing multi-head integration beyond the expected capacity and performance improvements?

Firstly, as we stated in **Q1**, the Helmholtz Dynamics is one of the primary sources of performance improvement.

Secondly, when we try to implement physics insights into deep models, adopting well-acknowledged tools to ensure optimization is a natural way. Thus, we use the multi-head style design, which is widely used in Transformers to augment model capacity.

If the reviewer does expect more insights, we think the multi-head design can make the model learn multiple dynamics of the fluid, which can decompose the complex dynamics into multiple complementary components. But this is still under verification.

Reviewer comment (Experiments):
Have the authors attempted to compare the Helmholtz decomposition with a Clifford Layer (Brandstetter et al., 2022) as they both aim to achieve the correlation of different components of the velocity field?

Author response:
> **Q8:** Compare the Helmholtz decomposition with a Clifford Layer (Brandstetter et al., 2022) 

As per the reviewer's request, we have added Clifford Layer as a baseline in all datasets. We report Relative L2 for Navier-Stokes dataset and Bounded N-S dataset, MSE and relative L2 for Sea Temperature dataset, and Perceptual loss, Relative L2 and MSE for Spreading Ink.

| Relative L2    | 64 × 64 / 128 × 128 / 256 × 256 Navier-Stokes | Bounded N-S | Sea Temperature | Spreading Ink           |
| -------------- | --------------------------------------------- | ----------- | --------------- | ----------------------- |
| Clifford Layer | 0.1841 / 0.1088 / 0.1759                      | 0.1071      | 2.2390 / 0.0606 | 2.469 / 0.1939 / 0.0124 |
| HelmSim        | 0.1261 / 0.0807 / 0.1310                      | 0.0652      | 0.1704 / 0.0368 | 1.464 / 0.1399 / 0.0065 |

Reviewer comment (Experiments):
The ablation study is not comprehensive and convincing.

Author response:
**We have provided a comprehensive ablation study with quantitive results in $\underline{\text{Appendix B of original submission}}$. Most of the ablations mentioned by the reviewer have been included in that section.**

(1) "The directly learned velocity is not too bad. What is the L2 error with and without the Helmholtz dynamics? What is the computational overhead for training with Helmholtz dynamics?"

We have provided the relative L2 in $\underline{\text{Table 9 of original submission}}$. As per the reviewer's request, we newly added the computational overhead comparison, where learning Helmholtz Dynamics can significantly boost the model performance with marginal extra computation costs.

| 64 × 64 Navier-Stokes Dataset      | Relative L2 | GPU memory (MiB) | Running Time (s/epoch) |
| ---------------------------------- | ----------- | ---------------- | ---------------------- |
| Directly learning velocity         | 0.1412      | 14.86            | 72.18                  |
| Learning Helmholtz Dynamics (Ours) | **0.1261**  | 16.30            | 80.20                  |

(2) "The error map looks the same for including and not including the boundary condition terms, making it hard to understand the benefits of including the boundary terms."

We also have provided the relative L2 for this ablation in $\underline{\text{Table 10 of original submission}}$. To highlight the benefits of incorporating boundary conditions, we also zoom in the essential area in $\underline{\text{Figure 10 of revised paper}}$.

| Bounded N-S Dataset                 | Relative L2 | GPU memory (MiB) | Training Time (s/epoch) |
| ----------------------------------- | ----------- | ---------------- | ----------------------- |
| Not including boundary condition    | 0.0846      | 26.98            | 226.20                  |
| Including boundary condition (Ours) | **0.0652**  | 29.48            | 267.63                  |

(3) "Include the ablation study for the multihead, multiscale structure of your proposed model."

Please see $\underline{\text{Table 8 of original submission}}$ for these ablations. To trade-off efficiency and perfromance, we choose 4 heads and 3 scales as the offical configuration to 64$\times$64 Navier-Stokes Dataset.

| 64 × 64 Navier-Stokes Dataset | 1 head | 4 head     | 8 head | 16 head |
| ----------------------------- | ------ | ---------- | ------ | ------- |
| Relative L2                   | 0.1344 | **0.1261** | 0.1279 | 0.1249  |
| Training time (s/epoch)       | 59.69  | 81.20      | 120.86 | 171.97  |

| 64 × 64 Navier-Stokes Dataset | 2 scale | 3 scale    | 4 scale | 5 scale |
| ----------------------------- | ------- | ---------- | ------- | ------- |
| Relative L2                   | 0.1514  | **0.1261** | 0.1361  | 0.1330  |
| Training time (s/epoch)       | 64.43   | 81.20      | 99.83   | 120.06  |

Reviewer comment (Theory):
There is no general formulation of stream function in 3D. How will you generalize the proposed method for 3D cases?

Author response:
> **Q1:** How will you generalize the proposed method for 3D cases?

According to the official formalization of Helmholtz decomposition ${\mathbf{F}}(\mathbf{r}) = \nabla\Phi(\mathbf{r}) + \nabla \times {\mathbf{A}}(\mathbf{r}), \mathbf{r} \in \mathbb{V}$, the stream function for a 3D fluid field is a 3D vector ${\mathbf{A}}(\mathbf{r}) = (\mathbf{A}_x(\mathbf{r}), \mathbf{A}_y(\mathbf{r}), \mathbf{A}_z(\mathbf{r}))$. In 2D cases, we set the the velocity component on the $z$-axis to zero, that is $\mathbf{A}_x(\mathbf{r})=\mathbf{A}_y(\mathbf{r})=0$. To extend HelmSim to 3D fluid, the key point is to change the HelmDynamic block into learning 3D potential and stream functions $\widehat{\Phi}\in\mathbb{R}^{1\times D \times H\times W}$ and $\widehat{\mathbf{A}}\in\mathbb{R}^{3\times D \times H\times W}$, where $D$ is the additional depth dimension of 3D fluid. Then, following the Helmholtz decomposition presented in $\underline{\text{Eq. (1) of revised paper}}$, we can easily obtain the inferred 3D vector velocity field, thereby enabling HelmSim to achieve the velocity-aware 3D fluid prediction.

Reviewer comment (Experiments):
What are the Reynold numbers for the cases? They look like laminar flows. How is your method's performance on turbulence?

Author response:
> **Q2:** What are the Reynold numbers for the cases? How is your method's performance on turbulence?

Sorry for this missing information. The Reynold number for the Navier-Stokes dataset is about $10^4$, and for the Bounded N-S dataset is about 300.

To better evaluate the performance of the turbulence dataset, we experimented with a turbulence dataset provided by TF-Net [1]. We strictly followed its setting and reported the RMSE for predicting future 20 timesteps. In this dataset, Helmsim still perfroms best. These results have also been included in $\underline{\text{Appendix D of revised paper}}$.

| Turbulence Dataset | HelmSim     | FNO     | UNet    | MWT     | U-NO    | LSM     | TF-Net  |
| ------------------ | ----------- | ------- | ------- | ------- | ------- | ------- | ------- |
| RMSE               | **1042.38** | 1187.44 | 1062.13 | 1276.64 | 3276.09 | 1069.26 | 1061.78 |

Reference: 

[1] Wang, Rui, et al. "Towards physics-informed deep learning for turbulent flow prediction." KDD. 2020.

Reviewer comment (Evaluation):
Missing movies for the predictions to check for temporal coherence.

Author response:
> **Q3:** Missing movies for the predictions to check for temporal coherence.

Following the reviewer's suggestion, we have provided the prediction movies in the $\underline{\text{revised supplementary material}}$, where HelmSim achieves favorable temporal coherence.

Reviewer comment (Experiments):
In the ablation study part, the directly learned velocity is not too bad. The error map looks similar except for a narrow region. What is the L2 error with and without the Helmholtz dynamics? What is the computational overhead for training with Helmholtz dynamics? Moreover, the error map looks the same for including and not including the boundary condition terms, making it hard to understand the benefits of including the boundary terms. Can you also include the ablation study for the multihead, multiscale structure of your proposed model to demonstrate their effects?

Author response:
**We have provided a comprehensive ablation study with quantitive results in $\underline{\text{Appendix B of original submission}}$. Most of the ablations mentioned by the reviewer have been included in that section.**

(1) "The directly learned velocity is not too bad. What is the L2 error with and without the Helmholtz dynamics? What is the computational overhead for training with Helmholtz dynamics?"

We have provided the relative L2 in $\underline{\text{Table 9 of original submission}}$. As per the reviewer's request, we newly added the computational overhead comparison, where learning Helmholtz Dynamics can significantly boost the model performance with marginal extra computation costs.

| 64 × 64 Navier-Stokes Dataset      | Relative L2 | GPU memory (MiB) | Running Time (s/epoch) |
| ---------------------------------- | ----------- | ---------------- | ---------------------- |
| Directly learning velocity         | 0.1412      | 14.86            | 72.18                  |
| Learning Helmholtz Dynamics (Ours) | **0.1261**  | 16.30            | 80.20                  |

(2) "The error map looks the same for including and not including the boundary condition terms, making it hard to understand the benefits of including the boundary terms."

We also have provided the relative L2 for this ablation in $\underline{\text{Table 10 of original submission}}$. To highlight the benefits of incorporating boundary conditions, we also zoom in the essential area in $\underline{\text{Figure 10 of revised paper}}$.

| Bounded N-S Dataset                 | Relative L2 | GPU memory (MiB) | Training Time (s/epoch) |
| ----------------------------------- | ----------- | ---------------- | ----------------------- |
| Not including boundary condition    | 0.0846      | 26.98            | 226.20                  |
| Including boundary condition (Ours) | **0.0652**  | 29.48            | 267.63                  |

(3) "Include the ablation study for the multihead, multiscale structure of your proposed model."

Please see $\underline{\text{Table 8 of original submission}}$ for these ablations. To trade-off efficiency and perfromance, we choose 4 heads and 3 scales as the offical configuration to 64$\times$64 Navier-Stokes Dataset.

| 64 × 64 Navier-Stokes Dataset | 1 head | 4 head     | 8 head | 16 head |
| ----------------------------- | ------ | ---------- | ------ | ------- |
| Relative L2                   | 0.1344 | **0.1261** | 0.1279 | 0.1249  |
| Training time (s/epoch)       | 59.69  | 81.20      | 120.86 | 171.97  |

| 64 × 64 Navier-Stokes Dataset | 2 scale | 3 scale    | 4 scale | 5 scale |
| ----------------------------- | ------- | ---------- | ------- | ------- |
| Relative L2                   | 0.1514  | **0.1261** | 0.1361  | 0.1330  |
| Training time (s/epoch)       | 64.43   | 81.20      | 99.83   | 120.06  |