Paper: A Characterization Theorem for Equivariant Networks with Point-wise Activations (ICLR-2024)

Reviewer comment (Writing):
The paper is rather dense in representation theoretic notions. While I believe the paper is still self-contained, it may be beneficial for readers if the authors can expand on some intuition on the main theorems, maybe even in the appendix.

Author response:
Regarding the first point, we are currently enhancing the paper's readability and comprehension for a broader audience. In particular, we will incorporate examples and strengthen the appendices.

Reviewer comment (Theory):
It doesn’t look like the technique in this paper can generalize to infinite dimensional representation, which the author acknowledged in Page 4.

Author response:
Concerning the issue of infinite-dimensional representations: these play a significant role in the current theoretical discourse on equivariant models and our framework does not cover this approach. We acknowledge that this is one of the main limitations of our work, and we will address the matter in the manuscript with more precision. It is however remarkable to note that, while the descriptive models employed generally involve infinite-dimensional representations and concepts of harmonic analysis, the final deployed model is essentially a discretization of their theoretical formulation. In certain instances [1, 2], these models maintain exact equivariance solely through finite-dimensional representations and finite groups, conditions addressed in our framework as mentioned on page 4, though perhaps too briefly.

Reviewer comment (Novelty):
The main result looks like a relatively simple add-on to the study of equivariant networks by Wood & Shawe-Taylor (1996). Most results found in the appendix are either simple facts from the group and representation theory or are borrowed from Wood & Shawe-Taylor (1996) (e.g., the key equivariance condition (3)). Theorem 1 seems to be a relatively simple consequence of these results.

Author response:
We agree with the reviewer, and explicitly acknowledge in the paper, that our proposed results are largely built on top of the pioneering work by Wood and Shawe-Taylor [1]. 
However, we introduce the following extensions and improvements:

* We generalize the theorem to non-finite groups and continuous activations, as new classes of networks emerge from this extension. 
* We note the importance of identifying certain network classes by considering representations up to isomorphism. Note that two isomorphic representations are equivalent in terms of learning capabilities and this aspect was not covered by the original paper. Indeed, we solve an issue raised in Section 4.2 of Wood and Shawe-Taylor where they provide an example of a positive monomial representation that is not a permutation representation. That representation is actually isomorphic to a permutation representation. With our approach we fix all issues like this in a *general* and *complete* setting. 
* We integrate the theory developed by Wood and Shawe-Taylor with several adjustments that make the classification effective. For example, our Lemma 1 ensures the maximality of the families of matrices and activations presented. This aspect is not discussed by Wood and Shawe-Taylor, and it is necessary for the rigor of the characterization. 
* We want to highlight that despite acknowledging the authorship of some technical lemmas presented at the end of the appendices to Wood and Shawe-Taylor, our proposed proofs place a greater emphasis on algebraic aspects, especially grounding our characterization on the classification of multiplicative subgroups of $\mathbb{R}$, diverging by Wood and Shawe-Taylor. This approach not only improves the readability of the proof but also highlights the potential to generalize these results to complex numbers as a plausible solution to certain limitations of real equivariant networks as suggested in the Conclusions.

In the manuscript that we will upload, we will further update and refine these proofs. When compared to the original ones, they will show their kinship, but also their novelty.
We believe that the impact and implications of these innovations are significant for the machine learning community, especially considering the evolution and growth of the use and study of equivariant representation learning during the twenty years passed since the publication of Wood and Shawe-Taylor's paper.

Reviewer comment (Presentation):
Though the paper includes a discussion of applications of Theorem 1 in Sections 5 and 6, these sections are not easy to follow. They essentially consist of a long and not so well structured list of comments surrounding a few corollaries. The context of these comments is explained relatively vaguely, partly through references to other papers. It is not really clear why any of these corollaries and comments are actually important. The paper does not include specific examples, illustrations, or experiments showing how Theorem 1 can help improve designs of equivariant models.

Author response:
We thank the reviewer for pointing out the difficulty in understanding the focus and significance of Sections 5 and 6. We intend to refine these sections in the updated version of the manuscript to enhance comprehension. This will involve introducing fundamental concepts essential for understanding, along with the inclusion of simple examples. In instances where space constraints may limit these improvements, we will ensure to adequately strengthen the appendices and incorporate appropriate references in the main text. The reviewer rightfully questions how our proposed results may influence the design of equivariant models. We indirectly address this concern by highlighting how our findings show gaps in understanding specific phenomena. For instance, in Section 5.2, we present a list of permutation-equivariant neural networks which, to the best of our knowledge, are not found in the existing literature. The existence of these networks show an inadequate understanding of the permutation equivariance, a gap stemming from the scarcity of theoretical results in the field. In the long term, addressing this gap appropriately may lead to the development of new, more efficient, and better-performing models. With this work, we aim to contribute to this endeavor.

Reviewer comment (Writing):
The writing is sloppy in several places, with typos and unclear grammar. For example, only in Lemma 5: the index $i$ seems to be missing in the definition of $\mathcal{T}(\mathcal M)$; the indices and coefficients are wrong in the formula for $f(\sum_{j\in S}M_{rj}x)$; a word seems to be missing in "It is a simple to check".

Author response:
We thank the reviewer for their comments, particularly appreciating the precision of their remarks. We acknowledge the errors identified in Lemma 5 and will promptly rectify them in the manuscript. We will also revise the proof of Lemma 1 to enhance its clarity. Furthermore, we acknowledge and will rectify the errors and ambiguities noted in Lemma 7.

Reviewer comment (Writing):
The sentence "Hence, we can restrict our classification.." after the proof of Lemma 7 seems to consist of two different statements, but it is unclear where the first one ends and the second begins.

Author response:
We thank the reviewer for their comments, particularly appreciating the precision of their remarks. We acknowledge the errors identified in Lemma 5 and will promptly rectify them in the manuscript. We will also revise the proof of Lemma 1 to enhance its clarity. Furthermore, we acknowledge and will rectify the errors and ambiguities noted in Lemma 7.

Reviewer comment (Writing):
In Lemma 7: "dived into three different type" is another example of unclear writing.

Author response:
We thank the reviewer for their comments, particularly appreciating the precision of their remarks. We acknowledge the errors identified in Lemma 5 and will promptly rectify them in the manuscript. We will also revise the proof of Lemma 1 to enhance its clarity. Furthermore, we acknowledge and will rectify the errors and ambiguities noted in Lemma 7.

Reviewer comment (Writing):
I find this paper quite difficult to read at times, and I think that it would benefit from more explicitly defining certain concepts and writing out certain arguments. In the proof of Corollary 3, I do not quite understand the application of Theorem 1; why does the representation have to be trivial? Also, Section 6.2 about invariant graph networks is very dense. See also questions.

Author response:
As suggested, we will provide more explicit definitions of concepts and arguments. In particular, we will modify the proof of Corollary 3 to emphasize how it derives from Theorem 1 and how those representations are trivial. Subsequently, we will proceed to clarify and expand Section 6.2.

Reviewer comment (Experiments):
The applicability of these results to empirical equivariant machine learning is unclear. Already practitioners do not use pointwise nonlinearities for $SO(3)$ equivariant networks. Also, are there any expectations of potential benefits from using other representations of $S_n$ for invariant graph networks?

Author response:
This theorem may be of interest even to the ordinary machine learning practitioner as it provides the complete list of cases where the application of point-wise activations is viable and those can generally lead to increased computational efficiency. 

We agree that many different approaches to non-pointwise activations for equivariant $SO(3)$-equivariant networks are employed in practice but they in general require more computational resources, present a slow convergence rate, or both. For this reason, it may be favorable to employ pointwise activation to mitigate those shortcomings, but we prove in the paper that this is not a viable option.
Indeed, we believe that this result is of greater significance for machine learning researchers aiming to construct new equivariant models. In fact, in some relevant cases (e.g. exact rotation equivariance), this theorem suggests that natural research directions (e.g., pointwise activations coupled with any $SO(3)$-representation) might require more careful thinking.

Regarding the list of other presented $S_n$-equivariant models, we have no strong results about their general properties. We deemed it significant to highlight their existence not only for sake of completeness but also because this manifests a lack of understanding of the broader concept of permutation equivariance, a deficiency arising from a scarcity of theoretical work in the field, a lacuna that, if strongly addressed, could lead to the construction of more efficient and expressive models. With this work, we hope to contribute positively to the long-term effort of filling this gap.

Reviewer comment (Theory):
In Section 6.1 / Section 6.3, why is the following case ruled out? Say I let $f$ be linear, so that $\tilde f$ is just scaling and is hence equivariant to $SO(3)$. Then the output representation is the same as the input, and is in particular not invariant. This can be done in the $SO(3) \times S_n$ case as well.

Author response:
We thank the reviewer for catching our oversight formulating Corollary 3; indeed, an hypothesis was missing from the statement and the statement holds true only for nonlinear activations which rules out the provided example.

Reviewer comment (Writing):
Typo: Page 4, sentence with "Although employing techniques analogue to..." has several typos that make it confusing, incluidng use of "hypothesis" and "thesis".

Author response:
We appreciate the reviewer for pointing out these typos. The corrections will be promptly incorporated into the manuscript.

Reviewer comment (Writing):
Typo: Page 14, "Aff" is incorrectly typeset before Theorem 5.

Author response:
We appreciate the reviewer for pointing out these typos. The corrections will be promptly incorporated into the manuscript.

Reviewer comment (Writing):
The paper is a theoretical contribution and emphasizes that the results are of practical value. However, I believe that in order for it to be of practical value the paper should be more accessible to a broader audience. Although formal and precise, there is little effort put in describing the main results in plain English.

Author response:
We acknowledge the importance of improving the paper's accessibility and readability. In our revisions, we ensure to present the main results in a more straightforward manner to enhance clarity for a broader audience.

Reviewer comment (Theory):
By the previous item, remarks such as (see abstract) "we prove that rotation-equivariant networks can only be invariant" is in practice more nuanced, and I consider the statement actually incorrect. Another example of a rotation equivariant method based on equivariantly obtained spherical grids (they call it equivariant mesh) is GemNet:

Gasteiger, J., Becker, F., & Günnemann, S. (2021). Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34, 6790-6802.

Effectively, the paper works with scalar fields over the homogeneous space $\mathbb{R}^3 \times S^2$, as also discussed in recent work https://arxiv.org/abs/2310.02970 , which also fully operates in the domain of regular representations. I remark that this reference seem to have appeared only after the ICLR deadline so of course I do not expect this to have been included.

Author response:
In agreement with the reviewer, we acknowledge the necessity for a broader contextualization of Theorem 1 within the current literature covering all nuanced takes on equivariance. For brevity in this comment, we categorize these nuances as approximate equivariance compared to exact equivariance, and finite-dimensional frameworks compared to harmonic analysis approaches. Adopting this simplified view, our restriction to  exact equivariance and finite-dimensional representations should be pointed out earlier and more clearly in the text, highlighting how the analysis conducted under these constraints also affects practices derived from harmonic analysis approaches. We proceed to elaborate on the matter in greater detail:
We agree, as pointed out by the reviewer, that a substantial portion of the current literature on equivariant models relies on harmonic analysis approaches [1, 2, 4]. Processed inputs are continuous finite-energy signals defined on homogeneous spaces (e.g., $L^2(G)$ functions) which are infinite-dimensional spaces and thus fall outside the scope of our study.
However, as pointed out at page 4 of the paper, in practice continuous models are often discretized, relaxing the equivariance condition to an approximate notion. This process is often obtained by truncating finite Fourier transforms of the signal or discretizing the symmetry group and domain. This discretization frequently boils down to considering equivariant models with finite-dimensional representations of finite groups [5, 8], which is dealt with by our framework and, in line with our findings, allow pointwise activations and non-trivial representations.
Contrary to our approach, other methods suggested by the reviewer (such as GemNet [3]) employ tools akin to Tensor Field Networks [7], featuring non-pointwise activations that, as implied by the paper's title, go beyond the scope of our work (utilizing tensor representations, a nonlinear function, formula 4 in Dym and Maron [7]).
These are however very interesting directions that we will address as future research.

Reviewer comment (Writing):
On page 3 the symbol $\mathbb{R}^*$ is used. What does the asterix indicate?

Author response:
These are the multiplicatively invertible elements in $\mathbb{R}$. The definition is initially presented in Lemma 6 in the Appendix. We appreciate the reviewer for bringing this to our attention; we will introduce the definition in the main body of the text before referencing this set.

Reviewer comment (Writing):
I did not understand the introduction of the main result "Although employing techniques analogue to already known ones, we highlight how hypothesis can generalize ... of the basis" Apart from this sentence being too long, I really do not understand what is being said here, please rewrite.

Author response:
This segment highlights how our work contributes to advancing the state of the art, specifically building upon the work of Wood and Shawe-Taylor [6]. Notably, we want to emphasize how we achieve more refined results by considering representations up to isomorphism, despite employing more general assumptions. We will ensure to enhance the presentation of this section in the manuscript.

Reviewer comment (Writing):
Theorem 1 could benefit from some subsequent, explicit examples. Take ReLU and e.g. a cyclic permutation group. Or any example you see fit. I think the paper remains abstract throughout and I think it could be helpful to see some common examples.

Author response:
We appreciate feedback on Theorem 1 and agree that providing explicit examples, such as employing ReLU with a cyclic permutation group, would enhance the clarity and applicability of the paper. We will certainly integrate these examples into the paper to offer a more concrete understanding of the theoretical concepts presented.

Reviewer comment (Theory):
Section 5.2 makes sense, but as discussed it is limited in scope regarding the use of irreducible representations exclusively.

Author response:
We concur. This section introduces a standalone corollary that does not impact the subsequent results we present. Nevertheless, given the role of disentangled networks in the literature, we deem it relevant to include this presentation.

Reviewer comment (Theory):
Could you clarify Corollary 3. It is hard to understand what it is about. In particular what is the significance of the connected component (are we talking about subgroups here?).

Author response:
In this scenario topological groups, such as $SO(3)$, come into play. These groups exhibit topological properties such as connectedness and compactness, which we employ in Corollary 3. We will ensure to provide additional details and examples in both the main text and the appendices.

Reviewer comment (Experiments):
"... but not more general equivariant tasks such as segmentation or detection" again this is an artifact of sticking to irreps.

Author response:
This point stems from the finiteness of the matrix group induced by the representations. This statement is general and not specific to disentangled networks alone. We believe that our misalignment on this point can be traced back to the discussion in Answer 3 Discretization.