Paper: From Zero to Turbulence: Generative Modeling for 3D Flow Simulation (ICLR-2024)

Reviewer comment (Writing):
I think it is a little unclear that you write "Thus, we propose to circumvent the roll-out dilemma of autoregressive models by modeling the marginal distribution E_t[p((u, p)t | u0, p0)] directly, capturing the distribution of turbulent flows with a generative model.", but then in the next paragraph remove the dependence on the initial state and remove the expectation. As far as I can see, the stochastic process perspective has been used previously (e.g. ref Yang and Sommer), and what is done here is removing the condition on the initial state. I'm note sure where the expectation comes in.

Author response:
**Q: Why do you introduce the initial state $\boldsymbol{u}\_{0}, p\_{0}$?**

We have changed the term $\mathbb{E}\_{t}[\mathrm{p}((\boldsymbol{u}, p)\_{t} \mid \boldsymbol{u}\_{0}, p\_{0})]$ in Section 4 to $\mathbb{E}\_{t}[\mathrm{p}((\boldsymbol{u}, p)\_{t} \mid \boldsymbol{u}\_{0}, p\_{0}, t)] = \mathrm{p}((\boldsymbol{u}, p) \mid \boldsymbol{u}\_{0}, p\_{0})$ to clarify that our model learns to sample from the flow distribution marginalized across time, i.e., approximates the underlying distribution with a stationary distribution.

In Section 4.1, we propose the generative turbulence simulation task by noting that the above marginal distribution $\mathrm{p}((\boldsymbol{u}, p) \mid \boldsymbol{u}\_{0}, p\_{0})$ is independent of the choice of initial state $\boldsymbol{u}\_{0}, p\_{0}$ under the assumption of ergodicity. It may seem unnecessary to introduce the initial state given the ergodicity assumption, however, this is important to convey our method clearly to the majority of readers who are likely most familiar with neural surrogate models in terms of time series modeling.

Reviewer comment (Experiments):
Would it help to pre-processing the data, e.g. rotations, flips to augment the dataset?

Author response:
Data augmentation could certainly be helpful since the underlying physics also obeys various symmetries. Ultimately, we believe that the most promising direction for future research is equivariant generative models that obey the same symmetries as the Navier-Stokes equation, e.g. translation and rotation equivariance but also uniform motion and scaling equivariance.

Reviewer comment (Experiments):
Is it relatively easy or hard to apply this method to more complex turbulent generation, e.g., with fluid–solid interaction?

Author response:
In principle, it is possible to encode, for example, phase information to model multiphase flows or use a graph neural network as the denoising model to work with an arbitrary mesh discretization. In this work, we chose single-phase flows with a grid discretization to focus on the question whether generative models are viable neural surrogate solvers isolated from other, orthogonal modeling aspects, such as multi-scale graph neural networks.

Reviewer comment (Experiments):
The grid size used for generating dataset is 192 × 48 × 48. In fluid simulations, larger grid size are often used to achieve higher accuracy. Is this grid size chosen due to performance limitation of the CFD solver?

Author response:
Our choice of resolution balances dataset size, solver performance, and model memory requirements against the small-scale turbulence phenomena important to our work. We picked a small time step of $\Delta t = 0.1\mathrm{ms}$ between snapshots to ensure that it is possible for the autoregressive baselines to learn the short-lived dynamics of the smallest scales in the turbulent flows. However, these choices already result in a dataset of 9 TB. Even a moderate increase in resolution to 256×64×64 would more than double the size and strain our available computing resources. Furthermore, computational requirements grow quickly with resolution for operations like convolutions and attention. The resolution we chose already limited the batch size in training, even though we trained on A100s and chose a U-Net backbone for our model to minimize memory requirements. Even more efficient architectures are an important research area for future applications in 3D simulations.

Reviewer comment (Reproducibility):
There is no mention of releasing the data-set and the source-code. This is a major issue for reproducibility.

Author response:
Please find the link to the anonymized source code and a subset of the data in our non-public global reply.

We will release the source code publicly upon acceptance of the paper. Regarding the dataset, we are still looking for a viable hosting solution that can deal with the size of our dataset (9 TB). Alternatively, the accompanying scripts make it easy to generate the data yourself.

Reviewer comment (Experiments):
The size of the data-set is surprisingly small. While it's quite remarkable the model seems to capture the essence of the problem in such a small data-regime, this is also puzzling.

Author response:
Our dataset consists of $27 + 9 + 9 = 45$ simulations for training, validation, and testing, respectively, with each simulation comprising 5000 snapshots, one taken every 0.1 milliseconds for 0.5 seconds. This gives us $27 \times 5000 \approx 10^5$ samples for training minus the ones corresponding to the transition phase from the initial condition to turbulence. The raw data takes up roughly 150GB for each simulation with a further 50GB after preprocessing the data into an HDF5 file. Overall, this puts our dataset at 9TB for all 45 simulations or 2.25TB for the preprocessed data only. We have amended the paper to clarify this in the main text.

Reviewer comment (Experiments):
It would be great to scale to deep-learning type of sample size (10^5 samples – of course what's a sample in the CFD case is a bit up to interpretation).

Author response:
Given the significant size of the dataset, further scaling up our experiments is not viable given our hardware constraints.

Reviewer comment (Theory):
The limitations of A#F are palpable, especially the lack of equivariance and the geometry limitations, which are quite important.

Author response:
While we are aware of the limitations concerning equivariance and mesh geometries (described in Appendix F), we respectfully disagree with the reviewer that these are fundamental weaknesses of our approach. As our main contribution is to establish generative models as viable neural surrogate solvers for turbulent flows, we opted for the U-Net architecture widely used for 2D and 3D data. However, our approach is not limited to it; any architecture, including equivariant or graph-based ones, can be plugged in as a backbone. We hope that our framework will benefit from future model developments in these areas.

Reviewer comment (Novelty):
The architecture has much potential to grow once such limitations are addressed, but this also suggests that this is likely an initial work and possibly not yet ready for primetime publishing in a top venue like ICLR.

Author response:
While we are aware of the limitations concerning equivariance and mesh geometries (described in Appendix F), we respectfully disagree with the reviewer that these are fundamental weaknesses of our approach. As our main contribution is to establish generative models as viable neural surrogate solvers for turbulent flows, we opted for the U-Net architecture widely used for 2D and 3D data. However, our approach is not limited to it; any architecture, including equivariant or graph-based ones, can be plugged in as a backbone. We hope that our framework will benefit from future model developments in these areas.

Reviewer comment (Experiments):
The speedup is somewhat disappointing: the nominal 30x achieved is not so important when considering that A100 vs Xeon-2630 is probably already 50x faster (depending on workloads).

Author response:
Though GPUs can outperform CPUs in many tasks, it does not mean that CFD solvers benefit from GPUs in the same way. In fact, [OpenFOAM has been reported to achieve a performance boost of just 2x on GPUs](https://www.esi-group.com/sites/default/files/resource/other/1806/8th_OpenFOAM_Conference_Cineca_Spisso_2_0.pdf). Accordingly, a speed-up of 30x is highly relevant in practice regardless of the hardware differences. Further, please note that the performance of our model is constant with respect to the simulation parameters. In contrast, the runtime of the numerical solver depends on the velocity field $\boldsymbol{u}$ because it needs to scale its internal time step $\Delta t$ to keep the simulation stable.

Reviewer comment (Reproducibility):
Q1: plans to release code and database.

Author response:
Please find the link to the anonymized source code and a subset of the data in our non-public global reply.

We will release the source code publicly upon acceptance of the paper. Regarding the dataset, we are still looking for a viable hosting solution that can deal with the size of our dataset (9 TB). Alternatively, the accompanying scripts make it easy to generate the data yourself.

Reviewer comment (Experiments):
Q2: plans to scale up experiments.

Author response:
Given the significant size of the dataset, further scaling up our experiments is not viable given our hardware constraints.

Reviewer comment (Evaluation):
Q3: why does it generalize in such a small data regime? Any thoughts?

Author response:
Our training set comprises over $10^5$ samples based on 27 objects representing a large collection of turbulence patterns. The objects are sufficiently varied that the interactions between different components, e.g. corner configurations or shape widths, and their effects on downstream turbulence can be learned and transferred to unseen objects.

Reviewer comment (Experiments):
Since the proposed model is a combination of sophisticated blocks—DDPM, U-NET, and transformers—it is important to understand how these blocks cohabitate to generate good quality turbulent flow.

Author response:
As our main aim is to investigate whether generative models are viable neural surrogate solvers for 3D turbulent flows, we have purposefully chosen each component of our model. DDPM is an established, powerful framework for generative modeling in the image domain, which we chose as the basis of our generative model. U-Nets are well known as a capable neural architecture for grid-based data that produce outputs of the same dimension as the inputs while also creating a compressed internal representation through first down- and then up-scaling combined with skip connections. The spatial downscaling is important for our model so that we can apply a transformer at the coarsest scale and facilitate global communication between all parts of the domain without being held back by the quadratic scaling of attention. Consequently, the three components work together as one unit and neither can be meaningfully ablated.

To emphasize the interplay between the components, we have expanded the description of the model in Appendix D. Furthermore, please see our global reply for the project's complete source code.

Reviewer comment (Experiments):
Have you tested the proposed model on unstructured data like graph-meshes and cloud of points? To what extent is it applicable? Turbulent flows are irregular, and regular grids can hide some patterns.

Author response:
To focus on our primary research objective of investigating generative modeling for 3D turbulent flows, we need to disentangle it from other complex issues, such as the representational capacity of graph neural networks. Therefore, from the beginning, we decided to work with a regular grid discretization, which has been thoroughly researched in the ML community, particulary in the image domain, with established components like CNNs and U-Nets. As such, we did not test our model on irregular discretizations such as meshes or point clouds.

Reviewer comment (Evaluation):
What are the capabilities of the proposed model at the boundary layers? Do you have any metrics on the surface of the geometry, which is the most challenging part?

Author response:
In this work, we focus on modeling the downstream turbulence patterns away from the boundary and the object in the flow influences them. Accurately resolving the boundary layer on a regular grid would require a much higher resolution than is feasible given our storage requirements, as our dataset at its current resolution already needs over 9 TB of disk space.