Paper: Graph Neural Networks with Directional Encodings for Anisotropic Elasticity (ICLR-2024)

Reviewer comment (Experiments):
The equation 3 has a similar format of attention with 3 heads. What's the performance of Graph Attention Network (GAT)? Will attention scores be able to replace the weights and become a more general format of equation 3?

Author response:
Although attention mechanisms might offer some assistance, our approach introduces a simpler yet highly effective strategy. While attention masks must be learned during training, our method only requires no extra mechanism. Our modifications incur minimal additional cost for integrating into an existing Graph Neural Network (GNN) architecture.

Reviewer comment (Experiments):
Does the "5 rollouts" in Fig.3 mean the model autoregressively predicts 5 steps?

Author response:
Yes “5 rollouts” means using the model to autoregressively predict 5 steps. This plot is meant to show that the network is stable for longer rollouts. We will include animations in the supplementary video for the final version.

Reviewer comment (Experiments):
The performance of the baseline seems much worse than what the original MeshGraphNet paper reported. For example, in Fig. 6, if MeshGraphNet caused 60% volume change, their “DeformablePlate” example would have exhibited very obvious artifacts. There is uncertainty about whether the variational loss + MeshGraphNet combination negatively influenced its performance. Furthermore, there is a lack of crucial comparison to the reference simulator in terms of speed and accuracy.

Author response:
To test whether the sub-par performance of MeshGraphNets might be due to our unsupervised learning approach (the original work used supervised training), we trained MeshGraphNets with only isotropic material using unsupervised learning. We observe that MeshGraphNets performs significantly better for isotropic materials - see results in this anonymous [link](https://i.postimg.cc/xTFRWRyf/image-259.png). These results suggest that the sub-par performance of MeshGraphNets for anisotropic materials is indeed due to its inability to capture anisotropic behavior, not to the way in which it is trained.

Reviewer comment (Presentation):
In Figure 2, “f^{v->e}” in “vertex processing” is suspected to be a mistake and should perhaps be replaced with “f^{e->v}”.

Author response:
We will fix the typo in Figure 2.

Reviewer comment (Theory):
The choice to use fiber orientation as an edge feature is unclear. It might be more reasonable to consider fiber orientation as a finite element feature if such exists in the network module, since F, the deformation gradient, is typically an element quantity in linear tetrahedron finite elements, and Eqn. (6) suggests it is convenient to consider d as an element quantity as well.

Author response:
We generally think of fiber directions as continuous vector fields. These fields can be sampled at vertices, edge midpoints, or element centroids. One representation could be converted into any other using simple interpolation. We agree that fiber orientation could be an element feature, but we chose the edge representation for simplicity (edges and vertices are modeled in the MeshGraphNets graphs whereas elements are not).

Reviewer comment (Writing):
The notation “E_j” in Eqn. (4) is unclear and needs clarification.

Author response:
E_j denotes an edge in the undeformed configuration, i.e. E_j = x_k - x_l, where x_k and x_l are mesh vertices.

Reviewer comment (Writing):
The definition of E_x, E_y, and E_z in Eqn. (4) is not explained. It is guessed that one is d and the other two are orthogonal to d, but this is uncertain.

Author response:
E_x, E_y, E_z are Cartesian basis vectors, [1, 0, 0], [0, 1, 0] and [0, 0, 1]. These basis vectors are decoupled from fiber orientations, which are randomly oriented during training and are in general not aligned with E_x, E_y, and E_z. We opt for this projection operation to decouple information in different directions. As can be seen in all our experiments, we observe significant performance gains with this approach.

Reviewer comment (Theory):
The intuition behind explicitly formulating three weighted sums of e_j in Eqn. (3) is not fully clear. It is questioned whether this is truly necessary since the directional information d is already provided in the edge feature. While this may not hurt performance, more insight into this design choice would be valuable as it seems crucial to the paper’s technical contribution.

Author response:
We opt for this projection operation to decouple information in different directions. As can be seen in all our experiments, we observe significant performance gains with this approach.

Reviewer comment (Writing):
In Eqn. (5), it is unclear whether x^{t+1} is computed from a^{t+1} or is an independent variable. It is guessed that the network produces a^{t+1}, which is then used to compute x^{t+1}, and both are fed into the loss function.

Author response:
Yes, x^{t+1} should be a function of a^{t+1}.

Reviewer comment (Theory):
In Eqn. (6), although the text mentions that the loss function sums over per-element potentials, a more rigorous writing of the strain energy is desired, with a proper sum over all finite elements and a definition of how F is computed from x^{t+1} (e.g., citing 16 or similar literature). Additionally, clarification on how dFFd is computed in a single element is requested.

Author response:
Computing deformation gradient follows standard finite element theory for linear tetrahedral elements. We will include the computation in the supplementary material.

Reviewer comment (Theory):
There is a theoretical concern with using the incremental potential as the loss function L after incorporating the network. Let x be the new position and theta be the network parameters. While a stationary point of min_x L(x) solves implicit Euler integration, a stationary point of min_theta L(x(theta)) only satisfies dL/dtheta = dL/dx * dx/dtheta = 0. It is not obvious if this guarantees dL/dx = 0 (the true solution to the implicit Euler integration).

Author response:
The first-order optimality condition with the network in the loop is indeed that the gradient w.r.t. network weights should be zero. We will clarify this in our final version.

Reviewer comment (Reproducibility):
Clarification is requested on how fiber orientations are uniformly sampled in each element. Is each element assigned an independently sampled direction? The results appear to only contain horizontal and vertical directions shared by all anisotropic elements.

Author response:
Different fiber orientations are evaluated in Table 2 when examining force imbalance. In the remaining examples, we specifically opt for fiber directions aligned with and orthogonal to the applied forces to highlight their effects. Aligning the deformations with fibers in these directions is more intuitive for readers than scenarios involving randomly oriented fibers. We consider homogeneous fibers, i.e. same fiber directions for all elements, with directions and magnitude randomly sampled during training.

Reviewer comment (Evaluation):
In the “Convergence” experiment, it is unclear whether Fig. 3 displays the training loss or its difference from the ground truth incremental potential solved by the reference simulation.

Author response:
Fig.3 displays the difference between the network predictions on the test set and the ground truth simulation, i.e. test loss normalized by ground truth value.

Reviewer comment (Evaluation):
In Fig. 4, the reason why the “total” energy difference is lower than the “fiber” term difference is unclear.

Author response:
The energy corresponding to the external force load can be negative. Therefore, the total energy difference can be smaller than the fiber energy difference.

Reviewer comment (Experiments):
For both Figs. 3 and 4, it is uncertain whether the worse performance of MeshGraphNet should be attributed to the lack of direction-aware message-passing mechanism or to the decision to use the new loss function in MeshGraphNet.

Author response:
To test whether the sub-par performance of MeshGraphNets might be due to our unsupervised learning approach (the original work used supervised training), we trained MeshGraphNets with only isotropic material using unsupervised learning. We observe that MeshGraphNets performs significantly better for isotropic materials - see results in this anonymous [link](https://i.postimg.cc/xTFRWRyf/image-259.png). These results suggest that the sub-par performance of MeshGraphNets for anisotropic materials is indeed due to its inability to capture anisotropic behavior, not to the way in which it is trained.

Reviewer comment (Experiments):
In the “Volume Preservation” experiment, Poisson’s ratio = 0.48 does not mean zero volume change. Adding a third curve in Fig. 6 showing the volume change from the reference simulator would be informative.

Author response:
We agree that a Poisson ratio of 0.48 does not imply volume preservation, which is why we are plotting the relative percentage error with respect to the ground truth simulation. This description is indeed in the caption but we are happy to make it clearer.

Reviewer comment (Evaluation):
In the “Tip Displacement” experiment, the significance of the “Tip Displacement Error” is unclear without knowing the average size of the finite elements in these scenes.

Author response:
The cantilever beam mesh has dimensions  0.1m * 0.1m * 0.5m, resulting in a total volume of 0.005m^3. If we distribute 100 elements within the bar, each element, on average, would have a volume of 50 cm^3 and an average edge length of approximately 7.5cm.

Reviewer comment (Novelty):
The technical contribution seems potentially incremental from prior work (MeshGraphNets) -- however, this is not necessarily an issue, as the experiments are well-designed, the results are solid and findings are conclusive.

Author response:
Thank you for your feedback and for acknowledging that the solution is simple but effective. 

__Incremental__

We would like to emphasize that although the solution we presented appears simple, identifying the key limiting factor and proposing a simple yet effective solution was no trivial task. We argue that, rather than a disadvantage,  algorithms that combine simplicity and effectiveness are highly desirable.

Reviewer comment (Theory):
I am curious about the "material space bases" mentioned on page 4. Could you elaborate on how the "material space bases" are defined exactly? How are these defined relative to global coordinates, and are they defined in a canonical way?

Author response:
__Material Space Basis__

 We apologize for the confusion about the material space basis, we opt for this term to denote a coordinate system in the undeformed (i.e., material) configuration of the object. In our implementation, we simply use the canonical bases, i.e. unit vectors in the x, y, and z directions. The edge vectors in the undeformed configuration are projected onto these basis vectors by simply selecting the x, y, and z components.

Reviewer comment (Theory):
Would defining the local bases in a different way change the three axis-aligned weights/coefficients that are computed during preprocessing? In turn, would this make it difficult to learn? (Presumably the x-axis is aligned with the edge, what about the others.).

Author response:
__Material Space Basis__

 We apologize for the confusion about the material space basis, we opt for this term to denote a coordinate system in the undeformed (i.e., material) configuration of the object. In our implementation, we simply use the canonical bases, i.e. unit vectors in the x, y, and z directions. The edge vectors in the undeformed configuration are projected onto these basis vectors by simply selecting the x, y, and z components.

Reviewer comment (Experiments):
The major one is the lack of sufficient comparisons and evaluations; there is only one alternative method as the baseline to compare and demonstrate the superiority of the proposed methods. More baselines are strongly recommended, adding to the experiments and evaluations to support the proposed methods by thorough evaluations.

Author response:
While we only compare to one GNN-based baseline, this baseline is nonetheless state-of-the-art. Follow-up works such as HOOD [Grigoriev2023] introduce various improvements, but these extensions do not address the problem we target with our work – the inherent limitations of MeshGraphNets when modeling anisotropic materials.

Reviewer comment (Experiments):
For the network architecture, the message-passing operation connects the position after encoding and decoding. If we add more blocks (including message passing, edge processing, and vertex processing), what about the performance of the proposed methods?

Author response:
Improving the learned behavior by increasing the number of message-passing steps is possible, but it comes with a trade-off—more network parameters may result in longer training times and a higher likelihood of encountering local minima. Following MeshGraphNets, we opt for 15 message-passing steps to strike a balance between accuracy and computational efficiency.

Reviewer comment (Experiments):
For the loss function, how to determine the weight for each term and the ablation study on the different weight combinations should be evaluated.

Author response:
These energy terms do not conflict with each other. Rather, they describe the total mechanical energy of the physical system. Minimizing the energy in its current form amounts to finding the equilibrium configuration of the system. Although each term involves different physical parameters (Young’s modulus, Poisson’s ratio, mass density, etc), re-scaling individual terms by arbitrary weights would mean breaking the physics.

Reviewer comment (Evaluation):
What about the running times, such as the deformation efficiency? failure cases.

Author response:
We reported in the training paragraph of section 3.3 that the inference of our network on a mesh of 100 elements takes only 9ms. The same examples are around 10 to 20 times slower in our reference FEM simulation. MeshGraphNets reports two to three orders of magnitude performance gain over the reference FEM solver solver. Since our network architecture and hyperparameters follow MeshGraphNet, we expect the same performance when scaling to higher resolutions.

Reviewer comment (Novelty):
Some related works should be considered as  
[1] SO (3)-invariance of informed-graph-based deep neural network for anisotropic elastoplastic materials  
[2] Polyconvex anisotropic hyperelasticity with neural networks  
[3] RIMD: Efficient and Flexible Deformation Representation for Data-Driven Surface Modeling

Author response:
Thanks for pointing out these related works. After close examination, we notice that the first two works do not use Graph Neural Networks, despite the occurrence of  “graph-based” in the title of the first one. The third paper does not involve constitutive modeling or machine learning. It appears to us that these works do not have an immediate relationship to our method. Please let us know if we misunderstood.