Paper: Clifford Group Equivariant Simplicial Message Passing Networks (ICLR-2024)

Reviewer comment (Theory):
The authors overstate the naturality of their method. While the approach feels intuitive at a quick glance, it mixes two different concepts: Simplices are a topological concept, while the geometric product is related to the geometric space that the graph is embedded in. This leads to a number of issues. Even in 3D space, one can construct 100-simplices, but the Clifford algebra only goes up to grade 4. While intuitively higher-order simplices encode more information, higher-order grades eventually become lower-dimensional. More generally, the geometric product is lossy: different simplices will be described by the same multivector. These all seem like undesirable properties for a simplex representation. I wish the authors would acknowledge and discuss these issues, which to me seem central to this paper.

Author response:
We thank the reviewer for the highly valuable feedback. We address their concerns and questions in the following.   

1. **Using the Clifford algebra in combination with simplicial message passing**
    * *Is it reasonable to use Clifford algebra to represent 100-simplices in 3D space if Clifford algebra only has 4 subspaces?*  We agree that the naturality of the method might have been overstated, and have adjusted the language accordingly. We appreciate the reviewer's provision of this specific example, and we would like to use it as an opportunity to elaborate on the overarching concept of our paper. Our objective is to develop a model capable of comprehending both the topological and geometric aspects of geometric graphs. In 3D Euclidean space, Clifford algebra will indeed map 100-simplices to representations within four subspaces. However, note that the geometric features describable in 3D space are confined to positions (defined by 1 point), lengths (defined by 2 points), areas (defined by 3 points), and volumes (defined by 4 points). One might be interested in geometric features like angles between planes, but those are derivable from the positions, lengths, and areas. When the Clifford algebra is employed to represent a 100-simplex, it effectively condenses the simplex's geometric features into a combination of geometrically meaningful subspaces within the algebra. While a 100-simplex is defined topologically, the dimensional extent of its geometric features is limited by the space it occupies, which in this scenario is 3D space.  
    - *Can the Clifford algebra encode arbitrary higher-order simplices?*. 
     The reviewer correctly notes that Clifford algebra can constrain higher-order simplices to representations with limited grades based on the geometric space where the graphs are embedded. As we stated, we think that Clifford algebra is able to represent meaningful geometric features corresponding to the space in which the data lives. An alternative approach to capturing multiplicative geometric interactions within arbitrarily high-order simplices $\sigma^D$ could involve tensor products of the representations of vertices $\{v_i\}_{i=0}^D$. However, this method is computationally intensive due to the exponentially growing dimensions of tensor products and perhaps the associated calculations of irreps (e.g. Clebsch–Gordan coefficients). Although Clifford algebra might limit the expressiveness in representing complex higher-order interactions, one possible workaround is to employ additional hidden channels. These channels can represent higher-order simplices in larger functional spaces, potentially leading to more effective representations. In general, finding optimal machine-learning representations of higher-order objects is currently an active research area.
     - *Will different simplices be described by the same multivector?*  
    We apologize for any lack of clarity in the main text regarding using Clifford algebra to learn simplicial features. For a detailed explanation of this process, particularly for higher-order simplices, we would like to refer to section 3.2 in the main text. Furthermore, we recommend going through a more detailed clarification, which we responded to reviewer q1Xe. We can infer from the Clifford simplicial feature embedding that the model is expected to learn unique simplices comprising distinct vertices features. This is because varying vertices typically have different geometric features (e.g. positions) within the geometric space, leading to distinct input features for the Clifford learnable layers. Consequently, this ensures that the output, namely the learned higher-order simplicial features, will be distinct for different simplices. We regret any ambiguity in the main text and hope this explanation adequately addresses your concerns.

    * Finally, we would like to highlight that the Clifford algebra is not limited to three-dimensional space. Indeed, we carried out a supplementary experiment showing that including higher-dimensional simplices in this case improves the performance on the 5D convex hulls task. The simplex dimension is captured by the simplicial message passing framework, and our paper shows that fusing this with good geometric representations through the Clifford algebra is fruitful.

    | Method                    | MSE (↓) |
    |---------------------------|---------|
    | CSMPNs (max dim = 2)      | 0.002   |
    | CSMPNs (max dim = 3)      | 0.002   |
    | CSMPNs (max dim = 4)      | 0.001   |

    *Table 2: Convex Hull experiments with an increased maximal simplex dimension using CSMPNs*

Reviewer comment (Experiments):
While I like the experiments, it would be useful to see the performance on more standard benchmarks of geometric deep learning. The authors acknowledge that they do not achieve SOTA results on some of these tasks. That's fine, but it would still be useful to see how the method performs compared to other generic equivariant methods.

Author response:
4. **Other benchmarks in geometric deep learning and contributions.** Thank you for the suggestions! Preliminary experiments we conducted on QM9 (L. Ruddigkeit et al, 2012.) and n-Body system datasets (for n-Body dataset, we matched SEGNN (Brandstetter, J.et al (2021)) performance on initial tries) looked promising, but soon we figured out that the SOTA architectures were too optimized using domain knowledge and hence we decided to take a different route. It is not within our resource constraints to run these experiments during the rebuttal period.

Reviewer comment (Novelty):
The paper's contributions are somewhat thin: the authors essentially plug together two existing ideas (simplical message passing and Clifford group equivariant networks). Of course, the same can be said about many other papers.

Author response:
4. **Other benchmarks in geometric deep learning and contributions.** Thank you for the suggestions! Preliminary experiments we conducted on QM9 (L. Ruddigkeit et al, 2012.) and n-Body system datasets (for n-Body dataset, we matched SEGNN (Brandstetter, J.et al (2021)) performance on initial tries) looked promising, but soon we figured out that the SOTA architectures were too optimized using domain knowledge and hence we decided to take a different route. It is not within our resource constraints to run these experiments during the rebuttal period. 

      Regarding the significance of contributions, integrating simplicial message passing with steerable GDL methods is rather unexplored territory. Our initial study has subtly suggested potential further avenues for combining geometry and topology within graph learning tasks, which we hope may offer some meaningful addition to this growing area of study. Furthermore, we would like to point out that we were presented with computational challenges, which we alleviated by suggesting strategies such as sharing simplicial message passing layers across different simplex orders.

Reviewer comment (Experiments):
As the authors mention experiments on MD17 and QM9 are beyond the scope of their research; I hope the authors give a discussion or possible directions on how the manual lift could be implemented for small and medium-sized molecules to strengthen the paper's applicability, not just using H2O as an example.

Author response:
2. **Discussion and direction of manual lift.**  Thanks for the feedback! We now include promising directions for testing manual lifts in the appendix (section D), also detailed here for completeness. There are various methods to transform molecular graphs into simplicial complexes. To use a manual lift, graphs can be initiated with the Vietoris-Rips lift to form simplicial complexes. Subsequently, one can impose manually set thresholds for different dimensional simplices (like edge lengths for 1-simplices, areas of triangles for 2-simplices, and volumes of tetrahedra for 3-simplices). A group of (geometrically) close nodes form higher-order chemical structures, such as functional groups (e.g., aldehydes, ketones, nitrates, etc). For example, if we want to predict some properties of (amino) acids, including a simplex corresponding with a hydrophilic group such as carboxyl could be useful. Additionally, since molecular graphs often contain carbon rings, it's logical to break these rings into 2-dimensional or higher-order simplicial complexes to better represent the rings in a topological sense. Moreover, higher-order simplices can be manually constructed based on expert knowledge to capture the specific chemical and physical properties of the molecules. For instance, in biochemical structures like proteins or DNA, certain arrangements of atoms or residues have specific functional implications. By manually constructing higher-order simplices, one can incorporate these domain-specific insights into the model. This approach allows for the explicit representation of essential molecular structures, such as hydrogen bonding patterns and aromatic systems, which are critical for understanding molecular behavior and interactions.