Paper: Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
Lack of novelty compared to what was done in [16]. See my points 2d., 4., 7a., 8.

Author response:
Thank you for your thoughtful review and valuable feedback. We fear the reviewer has some misunderstandings of our work which we have worked to clarify in this rebuttal and in our revision.

Regarding your concern about the novelty, we would like to clarify there are some misunderstandings. In particular, our results cover $k$-hop geometric GNN, such as ComENet, which is mentioned in the limitation section in Joshi et al.’s work. Moreover, we investigate whether, when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ increases as $L$ grows.

In what follows, we provide point-by-point responses to your comments.

Reviewer comment (Theory):
In the definition of k-hop geometric GNNs, k is not defined (its role is not defined). The definition of k-hop GNN is a bit overlooked (although it's rather intuitive what the meaning of k-hop is), and more importantly, it is not clear how these are not essentially equivalent to regular GNNs with more layers (since a single layer of a 2-hop GNN aggregates information with a similar receptive field as a 1-hop GNN with 2 layers). Authors should elaborate on that or cite appropriate references.

Author response:
**Q1. In the definition of $k$-hop geometric GNNs, $k$ is not defined (its role is not defined). Actually, the definition of $k$-hop GNN is a bit overlooked (although it's rather intuitive what the meaning of $k$-hop is), and more importantly, it is not clear to me how these are not essentially equivalent to regular GNNs with more layers (since a single layer of a 2-hop GNN aggregates information with a similar receptive field as a 1-hop GNN with 2 layers). Probably authors should elaborate on that or cite appropriate references.**

**Reply:** We present the architecture of $k$-hop geometric GNNs in Eq (1), where $k$ denotes the size of neighborhoods from which a node aggregates information -- or equivalently, the maximum distance over which it aggregates information in a single message-passing step. Additionally, we reference ComENet as an instance of 2-hop invariant GNNs in Section 1.1.

Moreover, it is crucial to note that $k$-hop geometric GNNs are not necessarily equivalent to regular 1-hop geometric GNNs with $k$ layers. As proved in Appendix D of the revised version, under some assumptions on UPD and AGG, k-hop invariant GNNs can distinguish any two $k$-hop distinct geometric graphs but fail to distinguish any two $k$-hop identical geometric graphs. ***This indicates that 1-hop invariant GNNs might be less powerful than their 2-hop counterparts, and this trend continues.***

The reason lies in the fact that the value of $k$ influences the size of local neighborhoods, consequently affecting the input of the aggregation process. Utilizing a smaller $k$ may cause $k$-hop invariant GNNs to overlook changes in geometry between small local neighborhoods, whereas using a larger $k$ can capture these variations.

However, it's noteworthy that, for any finite $k$, we can always construct two $k$-hop identical but $(k+1)$-hop distinct geometric graphs. In this scenario, $k$-hop invariant GNNs cannot differentiate them even with thousands of layers, while 1-hop equivariant GNNs, due to the faithfulness of features, can achieve this with $k+1$ layers. This finding sheds light on the advantages of learning steerable features up to type-$L\geq1$. We have substantiated this point through empirical results for ComENet. In the revision, we have bolstered these observations by considering the LEFTNet model in Appendix F.1 Table 5 of the revision.

To clarify this implication, we have refined the concluding remarks. In particular, we have added the following remark: relying solely on propagating invariant features confined to specific k-hop neighborhoods is insufficient for enabling geometric GNNs to capture the global information of geometric graphs precisely.

Reviewer comment (Theory):
About Lemma 1, page 4: When defining the function c, why can't it simply be always the identity? One could always choose the neutral element from the group G. While the set {g ∈ G | g · c(G · X) = X} is not empty, it is unclear what is non-trivial about c(G · X).

Author response:
**Q2. About Lemma 1, page 4. When you define the function $c$, why can't it simply be always the identity? One could always choose the neutral element from the group $G$. I do understand why the set {$g\in G | g\cdot c(G\cdot {\bf X})={\bf X}$} is not empty. I do not understand what is non trivial about $c(G\cdot X)$**

**Reply:** The function $c$ is defined on the quotient space $\mathbb{R}^{3\times m}/G$ and maps to $\mathbb{R}^{3\times m}$. Therefore, it cannot be a trivial identity, which is a function mapping from a space to itself. The composition that sends ${\bf X}$ to $c(G \cdot {\bf X})$ is also not an identity because, for any two ${\bf X}$ and ${\bf X}'$ lying in the same orbit, we have $c(G\cdot{\bf X}) = c(G\cdot{\bf X}')$.

Reviewer comment (Writing):
V_0^{⊕d} is not explicitly defined early enough. What is d? Does it relate to the maximal rotation order L?

Author response:
**Q3. Also, $V_0^{\oplus d}$ is not explicitly defined early enough. What is $d$? Does it relate to the maximal rotation order $L$?**

**Reply:** We appreciate your feedback. However, we have introduced $d$ in the first sentence of the statement of Lemma 1, where we mention, "Let $V$ be a $d$-dimensional $G$-steerable vector space…". The parameter $d$ represents the dimension of the $G$-steerable vector space $V$ and is not directly related to the maximal rotation order $L$.

Regarding $V_0$, we acknowledge that we overlooked mentioning in Lemma 1 that we use $V_0$ to denote the trivial 1-dimensional representation of an arbitrary group $G$. We have made a revision to address this.

Reviewer comment (Theory):
In the proof that the decomposition of Lemma 1 is unique, what is lambda concretely? Isn't it almost always simply the norm of the irrep (for a vector, its norm; for a higher-order tensor, also its norm)? Then \rho(g_X) would just be the usual (matrix) representation of g_X. Can it be otherwise, or is it more complicated? If in practice things are simple in most usual settings, this should be stated for the reader. Alternatively, maybe g_X simply encodes the orientation of X (if it's a vector)?

Author response:
**Q4. You prove that the decomposition of Lemma 1 is unique. But, concretely, what is lambda? Isn't it almost always simply the norm of the irrep? Like, for a vector, its norm, or for a higher-order tensor, also its norm? And then $\rho(g_X)$ is just the usual (matrix) representation of $g_X$. Can it be otherwise? Or is it more complicated? Am I missing the point? If in practice things are simple, in most of the usual settings, saying so is helpful to the reader. OR, maybe $g_X$ simply encodes the orientation of $X$ (say if it's a vector)?**

**Reply:** There seem to be a few points of misunderstanding. 

Firstly, the output of the G-invariant function $\lambda$ is a vector of invariant scalars, while the norm is just a scalar. They cannot be the same, especially when the dimension $d$ is greater than $1$. Concretely, $\lambda({\bf X}) = f(c(G \cdot{\bf X}))$ is equal to evaluating the equivariant function $f$ at the representative $c(G \cdot{\bf X})$. Further details can be found in the proof of Lemma 1 (refer to Appendix C in the revised version).

Regarding $\rho$, it represents the group representation on the steerable vector space $V$. On the other hand, $g_{{\bf X}}$ is a group element that satisfies $g_{{\bf X}}\cdot c(G \cdot {\bf X})={\bf X}$. It is true that $g_{{\bf X}}$ encodes the orientation of ${\bf X}$ in the sense that applying rotations or reflections to ${\bf X}$ may lead to a change in $g_{{\bf X}}$ while $c(G\cdot{\bf X})$ remains unchanged. We have provided additional clarification on this point in Remark 1.

It is worth mentioning that $g_{\bf X}$ and $\lambda({\bf X})$ share similarities with local frames and the scalarization proposed in [1], respectively. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between them. Please refer to Appendix A in the revised paper for details.

[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

Reviewer comment (Novelty):
In ref [16], page 3, the authors already separate (factorize) the scalar (feature-only) contributions from the geometric ones. In this respect it is not obvious how contribution (1) (your Lemma 1) is new. Furthermore, there is a lack of an intuitive view of what may concretely go into your scalar function lambda(X).

Author response:
**Q5. In ref [16], page 3, it says: At initialisation, we assign to each node $i\in V$ a scalar node color $c_i\in C’$ and an auxiliary object $g_i$ containing the geometric information associated with it i.e. they already separate (factorize) the scalar (feature only) contributions from the geometric ones (vectorial one v + previous color). In this respect, it is not obvious how contribution (1) (your lemma 1) is new. Furthermore, I lack an intuitive view of what may concretely go into your scalar function lambda(X).**

**Reply:** We believe this is a misunderstanding. In the initialization of the GWL test in [16], the scalar node color only considers the node features. As iterations progress, the color $c_i^{(t)}$ and the auxiliary object $g_i^{(t)}$ incorporate not only the node features but also the geometric information and the information about the graph structure. In contrast, Lemma 1 exclusively focuses on the coordinates (geometric information) and remains independent of node features and the graph structure. In other words, both  $\rho(g_{{\bf X}})$ and $c(G \cdot {\bf X})$ are unaffected by node features and the graph structure. 

The separating technique used in Lemma 1 separates ${\bf X}$ into the representative $c(G \cdot{\bf X})$ and the associated group element $g_{{\bf X}}$. Using this technique, we can effectively separate any steerable feature $f({\bf X})$ into its equivariant part $\rho(g_{{\bf X}})$ and its invariant part $\lambda({\bf X})$. It provides us with a framework to investigate steerable features by examining their corresponding invariant features. While we acknowledge that this separating technique might share similarities with the one you mentioned, it is important to emphasize that our framework is more flexible -- it is not limited to GNNs but can be applied to other neural networks or algorithms.

Concretely, $\lambda({\bf X}) = f(c(G \cdot{\bf X}))$ is equal to evaluating the equivariant function $f$ at the representative $c(G \cdot{\bf X})$. Further details can be found in the proof of Lemma 1 (refer to Appendix C in the revised version).

Reviewer comment (Theory):
In Eq. (4) or the last line of the previous equations (called Eq. 3bis), the gothic-style notation is inconsistent (missing in Eq. (4), which is likely a typo). It is unclear how to identify the right term in Eq. 3bis as a lambda (invariant) when g^-1 appears in it; g^-1\cdot x_ij seems like an equivariant feature, not an invariant one.

Author response:
**Q6. gothic{g} appears in the r.h.s. of Eq (4) or in the last line of the previous est of equations (let me call this last line Eq. 3b is) (actually the gothic-style has been forgotten in Eq. (4), this is a typo I believe). How can we identify the right term in the r.h.s. of Eq. 3b is as a lambda (invariant), when $g^-1$ appears in it? I don't see why $g^{-1}\cdot x_{ij}$ should be invariant, on the contrary, it seems to me like an equivariant feature.**

**Reply:** Thank you for pointing out the typo. Notice that $g$ is a group element satisfying $g\cdot c(G \cdot{\bf X})={\bf X}$. In other words, we have $ g^{-1}\cdot{\bf X}=c(G\cdot {\bf X})$. It is crucial to note that $c(G \cdot{\bf X})$ is invariant as it only depends on the orbit. The term $g^{-1}\cdot x_{ij}$ is a linear combination of the columns in $g^{-1}\cdot{\bf X}= c(G\cdot{\bf X})$. From this, we deduce its invariance.

Reviewer comment (Theory):
In Eq. 7, first line, why isn't it 2l+2 instead of 2l+1 on the right-hand side? Since representing O(3) requires one additional component, an intuitive explanation would be helpful.

Author response:
**Q7. In Eq. 7, first line, why isn't it $2l+2$ instead of $2l+1$ (in the r.h.s.)? I understood that representing O(3) required one additional component? Could you provide an intuitive explanation?**

**Reply:** The construction of O(3) representations has been thoroughly documented in existing literature, and we have provided a concise review in Section 2. Notably, the irreducible representations of O(3) arise from the product of the irreducible representations of $SO(3)$ and those of ${\pm {\bf I}}$ (rather than a direct sum). Hence, the dimension remains $2l+1$.

Reviewer comment (Theory):
On page 6, the question “Does any (2l + 1)-dimensional invariant feature λ(X) correspond to a type-l steerable feature f(X)?” is posed. The reviewer notes that this sounds like a known fact, as type-l steerable features need 2l+1 numbers using Spherical Harmonics.

Author response:
**Q8. You write, in page 6: Does any $(2l+1)$-dimensional invariant feature $\lambda({\bf X})$ correspond to a type-l steerable feature $f({\bf X})$? Which sounds like an interesting question (although ideally I'd like to know the mapping, not just know about its existence). Note that this question is essentially asking whether the space of all type-l steerable features $f({\bf X})$ has a dimension of $2l+1$ since $D^l (gX)$ is invertible. But that sounds like a known fact: using Spherical Harmonics, it is known that type-l steerable features need components to be represented. That is, they need $2l+1$ numbers (dimensions)**

**Reply:** We believe the reviewer has a misunderstanding regarding the definition of steerable features in our context. It is true that SO(3)-steerable vector spaces are spanned by Spherical Harmonics, and thus, a type-$l$ steerable vector space $V_l$ has a known dimension of $2l+1$. However, the space of steerable features is a subspace of $V_l$, consisting of type-$l$ steerable vectors that are generated by some equivariant function from the input ${\bf X}$. More formally, it can be expressed as {$ f({\bf X}) \mid \text{ G-equivariant }f:\mathbb{R}^{3\times m}\to V_l $}. In Fig 4 of the revised version, we provide an illustrative example to emphasize that the space of type-$1$ features can be 1-dimensional rather than 3-dimensional, showing that they might not be equal.

The map sending $f({\bf X})$ to $\lambda({\bf X})$ is defined by the composition $\lambda({\bf X}):=f(c(G \cdot{\bf X}))$. The function $c$ is highly related to the notion of alignment in reference [1] listed below. We have included relevant discussions in Appendix A of the revised version.

[1] Winter, R. et al. Unsupervised learning of group invariant and equivariant representations. NeurIPS 2022.

Reviewer comment (Theory):
The remark below corollary 3 is a very strong, original conclusion, but it is unclear how it follows from corollary 3. If lambda is the norm of the representation, the result feels trivial. The claim that "the expressiveness of learning steerable features is primarily characterized by the feature dimension – independent of the highest type utilized" is not clearly supported.

Author response:
**Q9. The remark below corollary 3 is a very strong, original conclusion. I think I understand Corollary 3 (although I feel like in practice lambda is the norm of the representation, and in that case it's kind of trivial..). In any case I do not see how the remark "the expressiveness of learning steerable features is primarily characterized by the feature dimension -- independent of the highest type utilized" follows from corollary 3.**

**Reply:** We have clarified the non-triviality of $\lambda$ in the above responses. Here we would like to address your concern about the implication of Corollary 3. First, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality.

Our discussion on the utilization of different types of steerable features focuses on the former, given that the latter is subject to the model’s architecture. Therefore, asserting that “the expressiveness of learning steerable features is primarily characterized by the feature dimension -- independent of the highest type utilized” might lack precision without considering the model's universality. We have refined the assertion to state that: “the invariant features carried by steerable features are primarily characterized by the feature dimension -- independent of the highest type utilized.”

Additionally, we have rephrased the relevant sections and included a new section in Appendix B of the revised version to delve into universality and potential methods. We believe these revisions address your concerns and provide more precise implications of our results.

Reviewer comment (Presentation):
In Table I, the math for lines 1 and 2 is correct, but not for line 3. For example, (1+3+5+7+9+11+13)*256 = 12544, not 7424.

Author response:
**Q10. In table I, I can do the maths for line 1 and 2, but not line 3. (5+3+1)\*256 = 2304 (5+3+1)\*824 = 7416 but then, (1+3+5+7+9+11+13)\*256 =12544, not 7424**

**Reply:** Two hyperparameters are used to control the degree $l$ and order $m$ of the spherical harmonics when generating steerable features. The degree $l$ is directly determined by the steerable feature type $L$. However, the maximum absolute order $m$ is set to $|m|\leq 2$ by default [1] in both eSCN and EquiformerV2. The order $m$ of the spherical harmonics can range from $-l$ to $l$, and when $|m|\leq l$ it serves to restrict the feature space. When $|m|\leq 2$ the equation becomes (1+3+5+5+5+5+5)*256 =7424.

[1] https://github.com/Open-Catalyst-Project/ocp/tree/main

Reviewer comment (Novelty):
In the Conclusion, the two key findings are problematic: (1) is vague and already known, for example from [16], which distinguishes invariant (L=0) from equivariant (L≥1) networks; (2) is essentially the known oversquashing phenomenon. The term “oversquashing” is absent from the manuscript but mentioned in [16]; it should appear at least once, with explanation if the authors find it unsuitable.

Author response:
**Q14. Conclusion. Two key findings are claimed: (1) Propagating steerable features of type $\geq 1$ equips geometric GNNs with the inherent ability to automatically capture the geometry between local structures. (2) When the feature dimension remains constant, increasing the use of steerable features up to type-L cannot essentially improve the performance of equivariant GNNs But I think that (1) is vague and already known. For instance from (16), which clearly distinguishes invariant ($L=0$) from equivariant $L\geq 1$ networks. And that (2) is known as oversquashing and was known before. I note that the term oversquashing is absent from the manuscript, although it is mentioned several times in [16]. I think the term should appear.**

**Reply:** We respectfully disagree with these comments and fear that there are substantial misunderstandings of our contribution. Please allow us to clarify our contribution below.

Concern about (1), we would like to clarify our results not only ***cover cases like ComENet -- as outlined in the limitation section of [16]*** but also emphasize ***the significance of faithfulness in effectively capturing geometry between local neighborhoods during message passing***. It's noteworthy that [16] suggests that incorporating pre-computing non-local features, as in ComENet, may offer a straightforward approach to overcoming the limitations of invariant GNNs. However, our results, both theoretical and empirical, indicate that this approach may not be as effective as suggested. Please refer to Theorem 1 and Table 2. Moreover, [16] asserts that their setup can be extended to higher-order tensors. However, we found that similar results for this extension might not hold if the geometric objects, which correspond to the steerable features, do not lie on faithful representations. In particular, our framework highlights that significant geometric information between local neighborhoods could be lost during message passing if the representations lack faithfulness. Please refer to Theorem 1 and Appendix B in the revised paper for details.

Concern about (2), we did not include a discussion of over-squashing since it is not the problem we are studying and unrelated to our contributions. [16] does mention the over-squashing phenomenon for equivariant features in geometric GNNs, using the example of EGNN, which utilizes a single vector feature to aggregate and propagate geometric information. Specifically, these models are constrained to learn growing information within a fixed-length vector, potentially failing to propagate long-range information and learning only short-range signals from the training data [1].

Let us clarify the empirical evidence supporting the two contributing components of our work. Table 2 presents empirical evidence for **first** part of our theory involving $k$-hop models like ComENet – this was previously unsupported in [16]. In Table 2, we observe the over-squashing in some models as in [16] and **we have included a discussion on over-squashing in Appendix F.2.**

The ablation studies in Figure 2, Table 3, and Table 4 support the **second** part of our contribution which is aimed at answering: **When the feature dimension remains constant, does the performance of equivariant GNNs employing steerable features up to type-$L$ increase as $L$ grows?**

We study the information-carrying ability of different types of steerable features by investigating their corresponding invariant features. In Table 3 and Table 4 we fix the steerable feature dimension while varying the steerable feature type $L$. We select the steerable feature dimension based on the best-performing model (see Table 1). We find that the performance may not necessarily increase when increasing $L$. We do not investigate the phenomenon of over-squashing in this ablation study and we do not observe any effects of over-squashing in our results. Moreover, our theory on the information-carrying ability of steerable features is not limited to GNNs.

We believe that Section 5.2 in [16] is more relevant to the problem we study. As discussed in Section 4, [16] has conducted a comparison of the utilization of different types of steerable features. However, that comparison is not explicitly designed for invariant classification.

In summary, our paper addresses the limitation mentioned in [16] and is the first research highlighting the significance of faithful representations in capturing geometry between local neighborhoods during message passing. Moreover, we provide theoretical reasons supporting the idea that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

[1] Alon, U., and Yahav, E. On the bottleneck of graph neural networks and its practical implications. ICLR 2021.

[16] Joshi, C. K., Bodnar, C., Mathis, S. V., Cohen, T., and Lio, P. On the expressive power of geometric graph neural networks. ICML 2023.

Reviewer comment (Novelty):
I think a highly related work is Villar et al. (2022) ("Scalars are universal...") with several related and perhaps even identical results: please include this in your related work section.

Author response:
**Q1. Can you contrast your results with Villar et al. (2022)? How do your results differ/improve upon theirs? Let's take a basis vector $e_1 \in \mathbb{R}^{3 \times 1} $ and $f: \mathbb{R}^{3 \times 1} \rightarrow \mathbb{R}^3$ with $e_1 \mapsto \alpha e_1, \alpha \in \mathbb{R}$. This is clearly an equivariant function. What would the corresponding invariant function be such that Lemma 1 holds?**

**Reply:** We have integrated the work by Villar et al. (2022) into our related work, including a discussion in Appendix A of the revision on the distinctions between their research and ours. Here, we provide a brief discussion.

Our results exhibit clear distinctions from theirs, mainly due to our consideration of steerable features of higher types $l>1$ and the different approaches we have taken. Instead of investigating the spanning space, we leverage the concept of "representatives" of orbits and apply the equivariance of the function to derive Lemma 1. 

Our form also differs from Villar et al. (2022) in terms of the number of basis vectors and the uniqueness of invariants. In our form, the number of basis vectors aligns with the dimension of (steerable) features, rather than the number of input coordinates. Taking your example into account, where the function $f$ sends $e_1$ to $\alpha_1 e_1$. In the form provided in Villar et al. (2022), $f$ would be expressed as a linear combination of $e_1$, resulting in one basis vector. However, in our form, the number of basis vectors corresponds to the number of column vectors in $\rho(g_{{\bf X}})$ (which is three in this case). Moreover, $\lambda({\bf X})$ is directly determined by $\rho(g_{{\bf X}})^{-1}f({\bf X})$, using the invertibility of $\rho(g_{{\bf X}})$. This not only ensures the uniqueness of $\lambda$ but also highlights a key distinction. In Villar et al. (2022)’s framework, the invariants may not be unique, especially if the number of input coordinates exceeds the rank of their spanning space.

Let’s consider the example you provided, where the function $f$ sends $e_1$ to $\alpha_1 e_1$. In this case, the input embedding contains only one vector, namely $e_1$. There exists a rotation that transforms $e_1$ to the vector $(||e_1||, 0, 0)$, which lies on the $x$-axis. Here, we choose the representative $c(G\cdot{\bf X})$ to be the vector $(||e_1||, 0, 0)$ and the associated group element $g_{{\bf X}}$ to be the transformation that rotates $(||e_1||, 0, 0)$ to $e_1$. Then, $\lambda({\bf X}) = f(c(G\cdot{\bf X})) = f((||e_1||, 0, 0)) =(\alpha_1 ||e_1||, 0, 0)$ is equal to evaluating $f$ at the representative.


Additionally, we have established different correspondences between equivariant functions and invariant functions. Our results also involve a detailed investigation into the message-passing mechanisms of equivariant GNNs and invariant GNNs. We believe our results contribute to a more comprehensive understanding of the benefits offered by learning steerable features in geometric GNNs.

Reviewer comment (Writing):
The notations sometimes could be clarified a bit further. For example, the logic above Lemma 1 could be laid out a bit clearer.

Author response:
**Q6. Concern about notations mentioned in weakness?**

**Reply:** Thank you for your feedback. We have improved the presentation of the argument above Lemma 1. 

Regarding your concern about the group action in Lemma 1, we have provided further clarification in Remark 1, particularly emphasizing that the group action is absorbed by $\rho(g_{\bf X})$. An alternative perspective is to consider the product $\rho(g_{\bf X})\lambda({\bf X})$ as a linear combination of the column vectors of $\rho(g_{\bf X})$ with the coefficients given by the scalars in $\lambda({\bf X})$. Induced by the definition of group homomorphism $\rho(h\cdot g) = \rho(h)\cdot \rho(g)$, the group action then acts on the column vectors of $\rho(g_{\bf X})$.

Furthermore, the well-defined mapping in Lemma 1, which associates equivariant functions with invariant functions, allows us to use the corresponding invariant function $\lambda({\bf X})$ to understand an equivariant functions $f({\bf X})$. In the proof of Lemma 1, we define $\lambda({\bf X})$ as $f(c(G\cdot{\bf X}))$, evaluating $f$ at the representative $c(G\cdot{\bf X})$. The key reason $\lambda({\bf X})$ sufficiently represents $f({\bf X})$ is that we can use the associated group element $g_{\bf X}$ to recover $f$ from $\lambda({\bf X})$ by computing $\rho(g_{\bf X})\lambda({\bf X})$. It is essential to note that $g_{\bf X}$ is pre-selected and is independent of $f$. In fact, $g_{\bf X}$ shares similarities with local frames in reference [1] listed below. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between $g_{\bf X}$ and local frames. Please refer to Appendix A in the revised paper for details.


[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

---

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Reviewer comment (Theory):
It's not clear to me how $\rho(g_X)$ acts on $\lambda(X) \in V_0^{\oplus d}$. First off, what is the action of $\rho(g_X)$? Further, $\lambda(X)$ is a tuple of scalars, so are they all left-invariant? Or are you considering them as a vector in $V$? This is also how you define $f$ in Lemma 1.

Author response:
**Q6. Concern about notations mentioned in weakness?**

**Reply:** Thank you for your feedback. We have improved the presentation of the argument above Lemma 1. 

Regarding your concern about the group action in Lemma 1, we have provided further clarification in Remark 1, particularly emphasizing that the group action is absorbed by $\rho(g_{\bf X})$. An alternative perspective is to consider the product $\rho(g_{\bf X})\lambda({\bf X})$ as a linear combination of the column vectors of $\rho(g_{\bf X})$ with the coefficients given by the scalars in $\lambda({\bf X})$. Induced by the definition of group homomorphism $\rho(h\cdot g) = \rho(h)\cdot \rho(g)$, the group action then acts on the column vectors of $\rho(g_{\bf X})$.

Furthermore, the well-defined mapping in Lemma 1, which associates equivariant functions with invariant functions, allows us to use the corresponding invariant function $\lambda({\bf X})$ to understand an equivariant functions $f({\bf X})$. In the proof of Lemma 1, we define $\lambda({\bf X})$ as $f(c(G\cdot{\bf X}))$, evaluating $f$ at the representative $c(G\cdot{\bf X})$. The key reason $\lambda({\bf X})$ sufficiently represents $f({\bf X})$ is that we can use the associated group element $g_{\bf X}$ to recover $f$ from $\lambda({\bf X})$ by computing $\rho(g_{\bf X})\lambda({\bf X})$. It is essential to note that $g_{\bf X}$ is pre-selected and is independent of $f$. In fact, $g_{\bf X}$ shares similarities with local frames in reference [1] listed below. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between $g_{\bf X}$ and local frames. Please refer to Appendix A in the revised paper for details.


[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

---

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Reviewer comment (Theory):
In general, how can an *equivariant* function be described by *invariants*? I guess I'm not following some notations here.

Author response:
**Q6. Concern about notations mentioned in weakness?**

**Reply:** Thank you for your feedback. We have improved the presentation of the argument above Lemma 1. 

Regarding your concern about the group action in Lemma 1, we have provided further clarification in Remark 1, particularly emphasizing that the group action is absorbed by $\rho(g_{\bf X})$. An alternative perspective is to consider the product $\rho(g_{\bf X})\lambda({\bf X})$ as a linear combination of the column vectors of $\rho(g_{\bf X})$ with the coefficients given by the scalars in $\lambda({\bf X})$. Induced by the definition of group homomorphism $\rho(h\cdot g) = \rho(h)\cdot \rho(g)$, the group action then acts on the column vectors of $\rho(g_{\bf X})$.

Furthermore, the well-defined mapping in Lemma 1, which associates equivariant functions with invariant functions, allows us to use the corresponding invariant function $\lambda({\bf X})$ to understand an equivariant functions $f({\bf X})$. In the proof of Lemma 1, we define $\lambda({\bf X})$ as $f(c(G\cdot{\bf X}))$, evaluating $f$ at the representative $c(G\cdot{\bf X})$. The key reason $\lambda({\bf X})$ sufficiently represents $f({\bf X})$ is that we can use the associated group element $g_{\bf X}$ to recover $f$ from $\lambda({\bf X})$ by computing $\rho(g_{\bf X})\lambda({\bf X})$. It is essential to note that $g_{\bf X}$ is pre-selected and is independent of $f$. In fact, $g_{\bf X}$ shares similarities with local frames in reference [1] listed below. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between $g_{\bf X}$ and local frames. Please refer to Appendix A in the revised paper for details.


[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

---

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Reviewer comment (Writing):
I find it slightly confusing what the exact message is that the paper is trying to convey, especially towards the end of the paper.

Author response:
**Q4. Concern about exact message in the paper. In particular, elaborate on the last comments in the weaknesses section?**

**Reply:** We would like to address each of your concerns individually.

For your first concern, we want to clarify that there is no inconsistency. While Lemma 1 allows us to study equivariant features by examining their corresponding invariant features, it is essential to consider the message-passing mechanisms, especially the local aggregation, when investigating geometric GNNs. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods. However, hindered by trivial representations, invariant GNNs struggle in this regard significantly affecting their ability to precisely capture global geometry and obtain global invariant features from local invariant features. Basically, there are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot. 
In Appendix B of the revision, we have added a detailed explanation of why faithfulness is important.

The response to your second concern is contained in our response to Q3.

For the last concern, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality. Our discussion on the utilization of different types of steerable features focuses on the former, as the latter is subject to the given model architecture. It is not sufficient to conclude that $L=1$ is enough without considering universality. We have added a section in Appendix B of the revised version to discuss universality and potential methods. Nevertheless, our results still show that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

We have revised the relevant sections in our paper to enhance clarity and eliminate potential misunderstandings. We believe these revisions address your concerns and provide more precise implications of our results.

Reviewer comment (Theory):
It is claimed that one needs more than invariants to capture the global symmetries, yet Lemma 1 states that one can use invariants to represent equivariant functions (see also my previous comments).

Author response:
**Q4. Concern about exact message in the paper. In particular, elaborate on the last comments in the weaknesses section?**

**Reply:** We would like to address each of your concerns individually.

For your first concern, we want to clarify that there is no inconsistency. While Lemma 1 allows us to study equivariant features by examining their corresponding invariant features, it is essential to consider the message-passing mechanisms, especially the local aggregation, when investigating geometric GNNs. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods. However, hindered by trivial representations, invariant GNNs struggle in this regard significantly affecting their ability to precisely capture global geometry and obtain global invariant features from local invariant features. Basically, there are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot. 
In Appendix B of the revision, we have added a detailed explanation of why faithfulness is important.

The response to your second concern is contained in our response to Q3.

For the last concern, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality. Our discussion on the utilization of different types of steerable features focuses on the former, as the latter is subject to the given model architecture. It is not sufficient to conclude that $L=1$ is enough without considering universality. We have added a section in Appendix B of the revised version to discuss universality and potential methods. Nevertheless, our results still show that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

We have revised the relevant sections in our paper to enhance clarity and eliminate potential misunderstandings. We believe these revisions address your concerns and provide more precise implications of our results.

Reviewer comment (Experiments):
I find it slightly confusing that according to the paper, one doesn't need more than $ L=1 $, but many experiments (Table 1, 2, 3) use $ L=2$ and higher.

Author response:
**Q3. Why did you use $L=2$ and higher in your experiments in Tables 1, 2, and 3?**

**Reply:** Table 1 rows 1&3 report the baseline hyperparameters for the eSCN model. Table 1 row 2 illustrates what we propose as a suitable comparison. In particular, we fix $L=2$ to maintain the steerable feature type in the baseline, while adjusting the hidden channels $c$ in order to maintain the dimension of the steerable feature. Table 2 reports results for different models which use $L=0,1,2$. We report results for $L=2$ to demonstrate the effects of $k$-Chains when using higher-order steerable features. Table 3 reports results for $L=2,4,6$ for eSCN and $L=2,4$ for equiformerV2. This serves as an ablation study over the steerable feature dimension $L$. We have enhanced the robustness of this study in the revision which now includes results for $L=1,3,5$.

Reviewer comment (Theory):
In the conclusion, it is claimed that $L \geq 1$ captures local structures, but one doesn't need more than type $L$. Why don't the authors claim that $L=1$ is enough?

Author response:
**Q4. Concern about exact message in the paper. In particular, elaborate on the last comments in the weaknesses section?**

**Reply:** We would like to address each of your concerns individually.

For your first concern, we want to clarify that there is no inconsistency. While Lemma 1 allows us to study equivariant features by examining their corresponding invariant features, it is essential to consider the message-passing mechanisms, especially the local aggregation, when investigating geometric GNNs. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods. However, hindered by trivial representations, invariant GNNs struggle in this regard significantly affecting their ability to precisely capture global geometry and obtain global invariant features from local invariant features. Basically, there are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot. 
In Appendix B of the revision, we have added a detailed explanation of why faithfulness is important.

The response to your second concern is contained in our response to Q3.

For the last concern, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality. Our discussion on the utilization of different types of steerable features focuses on the former, as the latter is subject to the given model architecture. It is not sufficient to conclude that $L=1$ is enough without considering universality. We have added a section in Appendix B of the revised version to discuss universality and potential methods. Nevertheless, our results still show that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

We have revised the relevant sections in our paper to enhance clarity and eliminate potential misunderstandings. We believe these revisions address your concerns and provide more precise implications of our results.

Reviewer comment (Theory):
Regarding (a): This correspondence has an important limitation which I believe the authors should mention: Update and Aggregation functions which are actually useful for learning are reasonably `nice': for example, differentiable almost everywhere, continuous, etc. The correspondence suggested by lemma 1 uses a mapping of every orbit to a canonical element which indeed exists, but is not in general continuous or very well behaved. As a result an equivariant `nice' aggregation function will correspond to a `wild' invariant aggregation function.

Author response:
**Reply:** The point you raised is accurate. In the revised version, we have provided additional explanations and potential methods to address this concern. For detailed information, please refer to Appendix B.

Reviewer comment (Theory):
The paper's stand on invariant vs equivariant features seems inconsistent. On the one hand the paper maintains that there is a one-to-one correspondence between invariant and equivariant features, and that "propagation of steerable features can be effectively understood as propagation of invariant features, therefore analyzing the message passing using the corresponding invariant features is a reasonable approach (page 4)" on the other hand once this analysis is carried out the paper maintains that it points to an advantage of equivariant over invariant.

Author response:
**Reply:** We appreciate the reviewer's careful examination of our paper. We want to clarify that there is no inconsistency in our stance. Corollary 1 establishes one-to-one correspondences between invariant and equivariant features. However, when investigating geometric GNNs, it is essential to consider the message-passing mechanisms. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods, while invariant GNNs -- hindered by trivial representations -- struggle in this regard. This distinction significantly affects their ability to precisely capture global geometry and derive global invariant features from local invariant features. **There are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot.** 

To enhance clarity and mitigate potential misunderstandings, we have revised all relevant contexts in our paper. We believe these revisions address your concerns and provide more precise implications of our results.

Reviewer comment (Theory):
My main qualm is with Corollary 3 and similar statements. In this Corollary, the authors note that given an equivariant function on a k-dimensional input representation space, it can be converted to an equivariant function on any other k-dimensional input representation space. As a result, they claim that the choice of representation to which the internal activations of a neural network transform is irrelevant (modulo its dimension). However, the conversion mapping from one equivariant vector space to a different (equal-dimensional) equivariant vector space may be arbitrarily complex or difficult to compute, and indeed, it may or may not be computable with a given architecture. In the case of rotations, one needs to recover g from a faithful representation (e.g. l=1), and then evaluate another representation at g — but computing a Wigner D matrix of high L may take many layers/parameters in the GNN. This is in the same spirit as universality results, which e.g. require increasing numbers of layers to approximate polynomials of increasing degree (see e.g. the appendix of Bogatskiy et al 2022, Lorentz Group Equivariant Neural Network for Particle Physics). In other words, there is a critical distinction between whether the **information** to compute a particular mapping is available, and whether a given architecture can actually **efficiently compute** that mapping. The authors of this work seem to focus more on the former perspective; namely, whether or not there is information loss from a certain architecture, which precludes one from ever computing a particular function (by any means). This has been a fruitful perspective for existing impossibility results on invariant GNNs — since these results roughly establish that, by only working with invariants, some neighborhoods are indistinguishable, and so **no** architecture can distinguish between them. This is a strong notion, but the converse does not hold: when the information is **not** lost, this does not imply that any architecture can actually compute the mapping. All of this is to say that, in my opinion, Corollary 3 does not sufficiently imply that the choice of equivariant representation is irrelevant up to dimension. I suspect even that the choice of representation may affect the universality of a fixed architecture family.

Author response:
**Reply:** We acknowledge two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information. We intended to focus on the first aspect for the exact reason you mentioned in the passage on weakness (1) -- the latter is subject to the architecture of the model given, while the former is not. We want to clarify that our intention is not to directly conclude that the choice of equivariant representation is irrelevant up to dimension. We appreciate your effort in pointing out the lack of precision of some claims in the paper, especially the misleading conclusion derived from Corollary 3. We have carefully revised our statements to convey the implications of our results more precisely. Additionally, we have added a section in Appendix B to discuss the universality and potential methods. Here, we present a part of that discussion.

The relationship between internal representation and universality is delicate. At first glance, the topological properties of the Lie group $G=SO(3)$ (or $G=O(3)$) prevent us from constructing a continuous function ${\bf X}\mapsto g_{\bf X}$. That is, the poor regularity of $\lambda({\bf X})$ is inevitable, making any general analytical tool that relies on regularity assumption unsuitable for studying universality. However, a potential solution to this regularity issue is to extend the function ${\bf X}\mapsto g_{\bf X}$ to a function from a suitable covering space of the spatial embedding to $G$. To put it simply, lifting the spatial embedding to the covering space makes it possible to select a continuous extension of ${\bf X}\mapsto g_{\bf X}$. Once we have that, we naturally obtain an extended version of Corollary 3 with well-behaved correspondence between two representations on the covering space.
As a result, we suspect that under any architecture with universality, changing the internal representation while fixing the dimension does not affect the class of function the intrinsic invariant feature $\lambda({\bf X})$ can approximate. However, we cannot deny the possibility that certain choices of representation could benefit the model, allowing it to approximate the target function with fewer layers. To demonstrate this idea will require extensive work, and given the page limitation, we leave this to future works.

Reviewer comment (Theory):
On a related note, many of the paper’s claims are informal — e.g. “there is no fundamental distinction in message-passing mechanisms arising from different values of L” in Remark 3, or “analyzing the message passing using the corresponding invariant features is a reasonable approach” on page 4. It would be very helpful and important to make these precise.

Author response:
**Reply:** We appreciate the reviewer's feedback and concerns. In response, we have made several revisions to enhance the precision and clarity of our statements. Additionally, we have incorporated a new section (refer to Appendix A in the revised paper) to discuss additional related works, including Puny et al.'s frame averaging paper and the works by Du et al. on ClofNet and LEFTNet, and the distinctions between these existing notions and Lemma 1.

Indeed, within our framework, the function ${\bf X}\mapsto g_{\bf X}$ is exactly a singleton-valued equivariant moving frame defined in [1], if we restrict the function’s domain to where the action of $G$ is free. While Puny et al.'s frame averaging paper studies set-valued equivariant moving frames, the domain of these equivariant moving frames is still restricted to guarantee the equivariance. In contrast, our defining $g_{\bf X}$ remains well-defined even in cases where the action of $G$ is not free, making it not necessarily equivariant. 

The scalarization proposed in Du et al.'s papers, while similar to how we define $\lambda({\bf X})\coloneqq \rho(g_{\bf X})^{-1} \cdot f({\bf X})$, differs in that the local frames used for scalarization are generated from two coordinates rather than all the coordinates in ${\bf X}$. In contrast, our $g_{\bf X}$ involves a consideration of all input coordinates in ${\bf X}$ since it is related to the orbit $G\cdot{\bf X}$.

To augment the paper's comprehensiveness, we have included the empirical results of ClofNet and LEFTNet on $k$-chains. In particular, we believe the invariant design for LEFTNet belongs to the class of 2-hop invariant GNNs. So it's important to highlight that LEFTNet may still face the issues discussed in the last part of Section 3.1. Please refer to the table below for further details.

Lastly, we have cited Smidt et al. (2021) to acknowledge and reference the well-established fact about the output symmetry of an equivariant function being at least the input symmetry, as noted at the beginning of Section 3.2. We believe these revisions address your concerns and provide a more precise and well-referenced discussion of related works and concepts.

---

**K=2 & K=3**

|        	| 1 Layer       	| 2 Layers     	| 3 Layers     	| 1 Layer       	| 2 Layers     	| 3 Layers     	| 4 Layers     	|
|------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
|        	| k-hop chain   	| k = 2         	| k = 2         	| k = 3         	| k = 3         	| k = 3         	| k = 3         	|
| ClofNet	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 100.0 ± 0.0   	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 100.0 ± 0.0    	| 100.0 ± 0.0   	| 

---

**K=4**
|                	| 2 Layers      	| 3 Layers      	| 4 Layers      	| 5 Layers      	| 6 Layers      	|
|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------
| ClofNet        	| 50.0 ± 0.0    	| 100.0 ± 0.0   	| 95.0 ± 15.0   	| 95.0 ± 15.0   	| 95.0 ± 15.0   	|
| LEFTNet        	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0 |100.0 ± 0.0   	|               	|


[1] Olver, P. J. (2009). Lectures on moving frames.

Reviewer comment (Novelty):
There are a few very related lines of work which aren’t cited, but probably should be. For example, the paper’s reasoning relies heavily on the concept of “equivariant moving frames,” which are a classical notion (dating back to Elie Cartan); yet this term does not appear in the paper, nor does a citation to e.g. Puny et al’s frame averaging paper, which is a more recent machine learning paper that harnesses the concept of frames. A small section (the start of Section 3.2 on page 6) in this paper also notes that the output symmetry of an equivariant function must be at least the input symmetry; this is a well-established fact about equivariant functions, e.g. see Smidt et al 2021. Finally, and perhaps most significantly, related ideas were discussed in ClofNet (Du et al 2022) and its follow-up LeftNet (Du et al 2023), both of which use the precise idea of moving frames and invariantization to obtain equivariants from invariants. The latter work in particular includes an expressivity result for two-hop geometric GNNs.

Author response:
**Reply:** We appreciate the reviewer's feedback and concerns. In response, we have made several revisions to enhance the precision and clarity of our statements. Additionally, we have incorporated a new section (refer to Appendix A in the revised paper) to discuss additional related works, including Puny et al.'s frame averaging paper and the works by Du et al. on ClofNet and LEFTNet, and the distinctions between these existing notions and Lemma 1.

Indeed, within our framework, the function ${\bf X}\mapsto g_{\bf X}$ is exactly a singleton-valued equivariant moving frame defined in [1], if we restrict the function’s domain to where the action of $G$ is free. While Puny et al.'s frame averaging paper studies set-valued equivariant moving frames, the domain of these equivariant moving frames is still restricted to guarantee the equivariance. In contrast, our defining $g_{\bf X}$ remains well-defined even in cases where the action of $G$ is not free, making it not necessarily equivariant. 

The scalarization proposed in Du et al.'s papers, while similar to how we define $\lambda({\bf X})\coloneqq \rho(g_{\bf X})^{-1} \cdot f({\bf X})$, differs in that the local frames used for scalarization are generated from two coordinates rather than all the coordinates in ${\bf X}$. In contrast, our $g_{\bf X}$ involves a consideration of all input coordinates in ${\bf X}$ since it is related to the orbit $G\cdot{\bf X}$.

To augment the paper's comprehensiveness, we have included the empirical results of ClofNet and LEFTNet on $k$-chains. In particular, we believe the invariant design for LEFTNet belongs to the class of 2-hop invariant GNNs. So it's important to highlight that LEFTNet may still face the issues discussed in the last part of Section 3.1. Please refer to the table below for further details.

Lastly, we have cited Smidt et al. (2021) to acknowledge and reference the well-established fact about the output symmetry of an equivariant function being at least the input symmetry, as noted at the beginning of Section 3.2. We believe these revisions address your concerns and provide a more precise and well-referenced discussion of related works and concepts.

---

**K=2 & K=3**

|        	| 1 Layer       	| 2 Layers     	| 3 Layers     	| 1 Layer       	| 2 Layers     	| 3 Layers     	| 4 Layers     	|
|------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
|        	| k-hop chain   	| k = 2         	| k = 2         	| k = 3         	| k = 3         	| k = 3         	| k = 3         	|
| ClofNet	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 100.0 ± 0.0   	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 100.0 ± 0.0    	| 100.0 ± 0.0   	| 

---

**K=4**
|                	| 2 Layers      	| 3 Layers      	| 4 Layers      	| 5 Layers      	| 6 Layers      	|
|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------
| ClofNet        	| 50.0 ± 0.0    	| 100.0 ± 0.0   	| 95.0 ± 15.0   	| 95.0 ± 15.0   	| 95.0 ± 15.0   	|
| LEFTNet        	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0    	| 50.0 ± 0.0 |100.0 ± 0.0   	|               	|


[1] Olver, P. J. (2009). Lectures on moving frames.

Reviewer comment (Theory):
On page 3, in the definition of geometric graphs, in the last two sentences: g inconsistently has a subscript $g_i$. Also, the last equality should presumably be an inequality (k-hop distinct if for all isomorphism, there is a node I, such that for any g, that equation does NOT hold).

Author response:
**Reply:** We appreciate your feedback, and we have made revisions to address the noted typos and have provided the requested clarifications. Specifically,  we have added the definition of $V_0$, representing the 1-dimensional trivial representation of a given group $G$. Regarding the use of the calligraphic g, we acknowledge the convention that it is commonly associated with the Lie algebra. In this context, we have chosen it to distinguish from the regular g, which is used as a function in the proof of Lemma 2.

Reviewer comment (Writing):
On page 3, in the section on k-hop geometry GNNs: “Given a geometric graph G=(V,E,F,X).” is not a full sentence; perhaps “Given” should have been “Consider”.

Author response:
**Reply:** We appreciate your feedback, and we have made revisions to address the noted typos and have provided the requested clarifications. Specifically,  we have added the definition of $V_0$, representing the 1-dimensional trivial representation of a given group $G$. Regarding the use of the calligraphic g, we acknowledge the convention that it is commonly associated with the Lie algebra. In this context, we have chosen it to distinguish from the regular g, which is used as a function in the proof of Lemma 2.

Reviewer comment (Writing):
As a very minor nitpick, the calligraphic g the authors have chosen for the group element is used almost universally to refer to an element of a Lie algebra, not of a Lie group. I would recommend sticking to regular $g$.

Author response:
**Reply:** We appreciate your feedback, and we have made revisions to address the noted typos and have provided the requested clarifications. Specifically,  we have added the definition of $V_0$, representing the 1-dimensional trivial representation of a given group $G$. Regarding the use of the calligraphic g, we acknowledge the convention that it is commonly associated with the Lie algebra. In this context, we have chosen it to distinguish from the regular g, which is used as a function in the proof of Lemma 2.