Paper: Symmetric Basis Convolutions for Learning Lagrangian Fluid Mechanics (ICLR-2024)

Reviewer comment (Experiments):
Most real-world applications of fluid simulation are in 3D cases. It would be good if the paper has some 3D examples or discussions regarding how to extend the method to 3D cases.

Author response:
_Regarding three-dimensional systems_: We have added an additional three-dimensional test case in Appendix Sec. C.5, analogous to test case I in Sec. C.1.1 and Appendix Sec. C.1.2, see Section 4.1 of the main paper for a discussion of the results. This additional test case demonstrates how insights from one and two dimensions, e.g., regarding using symmetric and antisymmetric bases, also transfer to three dimensions and how our proposed SFBC approach is also more resilient to non-ideal training setups.

Reviewer comment (Experiments):
When applying this method to 3D scenarios, is there a concern that the weight matrix (W) might contain too many parameters, and intermediate matrix B (in Eq. 20) might become too large, causing challenges for training?

Author response:
_Regarding the intermediate Matrix B_: The size of the intermediate matrix is certainly a concern and is a potential limitation even in two spatial dimensions for larger particle counts as the size scales with n x u x v x w (for n edges and three spatial dimensions using u,v and w weights per dimension). However, in our backend, which will be published alongside the paper, we perform this computation in a batched format by splitting the set of edges into separate batches of size b. This limits the size of the intermediate matrix to b x u x v x w. However, this would still require storing potentially large matrices for backpropagation, which we avoid by recomputing the components of the outer products using a custom backprop operation. Accordingly, we only need to store the relative positions and input features for backpropagation, which allows us to train our network, even in three dimensions, using very low memory requirements, e.g., less than 4GByte. Note that some implementations of this Einsum operation can perform the operation without storing explicit intermediates, e.g., Nvidia’s Cutlass library. We have expanded the discussion Appendix Sec. A.1 to better reflect this.

Reviewer comment (Presentation):
There are no movies for the learned dynamics to check for temporal coherence.

Author response:
_Regarding temporal unrolling behavior_: We apologize for the omission and agree that it is a good idea to provide sequences that show the temporal behavior of the different methods and will include examples alongside the source code and data repositories upon publication. You can find an example of the temporal unrolling behavior for LinCConv with and without a window function and an example of our SFBC approach with and without a window function in the supplementary material. These gifs clearly highlight the difference in temporal behavior, i.e., how our Fourier-based approach yields a significantly lower and more temporally stable behavior than LinCConv

Reviewer comment (Novelty):
The unique contribution compared with previous research needs to be elaborated.

Author response:
_Regarding differentiation from prior work_: Fey's method and similarly LinCConv by Ummenhofer both used a separable basis, with the latter not being aware that their Linear Interpolation is separable. We expanded this concept to a generic formulation of separable bases and showed how having a generic framework to construct continuous convolution layers. This allows for readily choosing the best basis function for a specific task by investigating bases with different inherent properties, e.g., symmetries and smoothness.

Note that with our framework a much broader set of convolutional filters can be explored that are, for example, rotationally symmetric instead of just axisymmetric. Such a filter can be constructed, e.g., by using a spherical coordinate mapping with a single constant filter for the angular component and a different basis, e.g., a Chebyshev base, for the radial components. We have found promising initial results in this direction, but this opens up a much broader hyperparameter space that is beyond our current scope and computational resources. We have added this as an outlook in the Conclusions in Sec. 5. As an example of using such a combination consider the following example of using a combination of a linear basis for the radial component and a cubic spline basis for the angular component: https://i.imgur.com/iaZCnxy.png (uploaded as an anonymous user so double blind reviewing is preserved properly) .

Reviewer comment (Experiments):
The result of the proposed separable basis is demonstrated on the 1D and 2D tests. Since the method uses the outer product, will it cause issues with the scalability to higher dimensions?

Author response:
_Regarding the outer product_: Using an outer product can certainly be a limitation in performance as the number of weights of a filter scales with n^d (n basis terms, d dimensions); however, our approach shares this scaling behavior with regular convolutions and with prior work, e.g., Fey et al. [2018] and Ummenhofer et al. [2019]. Based on prior work and our own results in 3D, see Appendix Sec. C.5, this is not yet a limitation in 3D simulations, which are commonly used in fluid mechanics and other simulation problems. It could, however, become a limiting factor for convolutions in higher dimensions.

Reviewer comment (Evaluation):
What is the computational overhead compared with other baseline models?

Author response:
_Regarding computational overhead_: We have included a runtime evaluation in one, two, and three dimensions for different architectures and hyperparameters in Appendix Sec. C.6 (Figures 36 and 38) . This evaluation shows that for most basis term counts, especially in three dimensions, the computational requirements are very similar between the different basis functions, i.e., they are within 7% of each other. For one dimensional networks, we were able to observe a worse performance of our proposed approach and the Chebyshev basis, as the other basis functions are limited in computational performance by other parts of the network, but only for 16 and 32 base terms. For larger networks, we also observed the expected scaling of O(n^d), with n being the number of basis terms and for d dimensions. We added a reference to these results in Section 4.

Reviewer comment (Novelty):
What is the limitation of the current methods?

Author response:
_Regarding Limitations_: We have revised the conclusion section of the paper to highlight the limitations of our proposed method. These limitations include (a) investigating other physical systems and simulations outside of SPH, (b) exploring more complex architectures based on continuous convolutions, and (c) a lack of exploring a large space of hyperparameter choices that seem potentially relevant for future work, e.g., using dissimilar bases within a single network. Our aim was to provide a framework and a thorough investigation of how continuous convolution kernels can be built, and correspondingly, these topics are very interesting directions for follow-up work. We plan to explore some of these avenues and hope that our submission and source code will likewise inspire follow-up work by others.

Reviewer comment (Experiments):
I find it puzzling that the study did not incorporate "WaterRamps" and "Liquid3d" from prior research, considering their complexity. Utilizing these challenging scenarios would have been more suitable for demonstrating the model's effectiveness.

Author response:
_Regarding the datasets_: We agree that the Liquid3D and Waterramps datasets are useful datasets for machine learning in general, but we chose to generate our own datasets built on a more numerically quantifiable approach. For example, consider the Liquid3D dataset with random shapes colliding in random directions and the error being evaluated using particle distances. While this evaluation is useful from a data science perspective, it is not obvious how the error being evaluated relates to the actual fluid mechanics or the viewer's perception. However, when constructing the datasets, primarily for test cases I, II, and IV, we intentionally chose settings with narrow scopes that have clearly defined and expected behavior. Instead of evaluating particle distances, we can directly evaluate the resulting flow field regarding density, velocity and divergence. We have added this explanation to Appendix B.

Reviewer comment (Theory):
Regarding the proposed technique for obtaining symmetric/anti-symmetric basis functions (Eqn 9 and 10), it is essential to clarify whether these equations pertain to symmetry along the x-axis, y-axis, or a combined symmetry along both axes [x,y]. Additionally, what is the assumed domain of the basis functions $b_x$ and $b_y$? Furthermore, it is crucial to determine whether a given set of orthonormal basis functions retains their orthonormality after applying this technique.

Author response:
_Regarding the basis questions_: When talking about anti-symmetry beyond one-dimensional cases, we refer to antisymmetry with respect to the origin, i.e., the classical physical antisymmetry of f(\bf{x}) = -f(-\bf{x}). The basis functions we use are defined over the domain [-1,1]^d, i.e., the unit cube in 3D, where the relative distance of particles, i.e., the edge distances, are first scaled inversely by the particle support radius and then mapped using an optional coordinate mapping, analogous to the work of Ummenhofer et al. [2019] regarding the orthogonality of the combined basis, we have included proof of this property in Appendix A.1.

Reviewer comment (Reproducibility):
Did you directly use the even and odd Fourier Basis for symmetric and anti-symmetric basis functions (after proper modification of the 0 frequency)? If not, what is the explicit equation of the SFB? And if so, do you think, with proper regularization, filters with Fourier basis will be able to learn proper symmetric and anti-symmetric kernel?

Author response:
_Regarding the Fourier basis_: We did use the bases directly. For an explicit equation of SFB, see Eq. (53) in Appendix Sec. A.4. We believe that the results for the purely antisymmetric and symmetric terms in 1D (and now 3D) show that the combined basis can already learn purely antisymmetric and symmetric problems with the current learning setup, albeit at a higher error than a purely even or odd basis shown in Figure 2 (which is the expected behavior). Furthermore, the odd Fourier basis only utilizes antisymmetric terms in 1D and is significantly smoother than the DMCF approach, resulting in the lower observed error in Figure 2.

Reviewer comment (Evaluation):
In Figure 2, we can see that with a high number of basis functions, Fourier even (or odd) is outperforming DMCF. What might be the reason for it?

Author response:
_Regarding the Fourier basis_: We did use the bases directly. For an explicit equation of SFB, see Eq. (53) in Appendix Sec. A.4. We believe that the results for the purely antisymmetric and symmetric terms in 1D (and now 3D) show that the combined basis can already learn purely antisymmetric and symmetric problems with the current learning setup, albeit at a higher error than a purely even or odd basis shown in Figure 2 (which is the expected behavior). Furthermore, the odd Fourier basis only utilizes antisymmetric terms in 1D and is significantly smoother than the DMCF approach, resulting in the lower observed error in Figure 2.

Reviewer comment (Novelty):
The theoretical novelty is somewhat limited as the core concepts like symmetric bases are adapted from prior work in other domains. Additional theoretical analysis could further strengthen the approach.

Author response:
_Regarding theoretical foundations_: We have added a short proof that shows that the separable basis construction of our work preserves orthogonality, i.e., if two orthogonal bases b_i and b_j are chosen, then the resulting two-dimensional basis is orthogonal in 2D, and analogously in 3D, see Appendix sec. A.1.

Reviewer comment (Experiments):
The evaluations are extensive but restricted to a specific type of fluid simulation problems. It is hard to extract insights that can generalize to a broader scope. Testing generalization on more diverse physical systems could better establish applicability.

Author response:
_Regarding the evaluations_: We have added an additional test case that is constructed analogously to test case I but performs the evaluations in 3D instead of 1D, which highlights how some insights, e.g., how symmetric bases help with learning even symmetric functions, remain consistent across dimensions, see Section 4.1 in the main paper and Appendix C.5. 

_Regarding more diverse physical systems_: Our paper currently uses a compressible, an explicit weakly compressible, and an implicit incompressible simulation approach with very different requirements for numerical solvers and their respective stability. This provides a broad basis of evaluation for our network, albeit admittedly limited to particle-based hydrodynamics. We believe that exploring other physical systems and areas of application will be an exciting and promising direction for future work.