From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-25%
metareview:_summary,_strengths_and_weaknesses: This paper introduces two new architectures (signNet and basisNet) that are invariant to sign flip of eigenvector and also to other basis vector symmetries arising from degenerate eigenvalues in higher dimensions. This is especially useful for models that use Laplacian eigenvectors. The paper demonstrates that these architecture innovations improve performance for molecular benchmarks.

All reviewers were impressed by the paper and recommend strong acceptance.
  
justification_for_why_not_higher_score: Still not fully clear what the what the practical implication is of the second model basisNet.
justification_for_why_not_lower_score: All reviewers feel that this is a significant contribution.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1915/Authors
title: Author Response to cG9j
comment: We thank the reviewer for appreciating the motivation, the theoretical arguments, the empirical performance of our models, and the ease of which we can use SignNet with existing GNNs.

> “Is BasisNet included in any experiment other than Appendix J.3? I am wondering whether BasisNet suffers high complexity because there might be many distinct multiplicities … In that sense, is there any way to implement BasisNet efficiently?”

Yes, you are correct — BasisNet is only included in the spectral graph convolution regression experiments. Indeed, a naive implementation of BasisNet would require a whole IGN for each encountered multiplicity. This is reasonable for the experiments in Appendix J.3, as there are only 3 possible multiplicities for the grid graph (see Table 6 in Appendix C.2).

There is, however, a more efficient way to implement BasisNet using one IGN. This is done by taking $\phi_{d_i}(V_iV_i^\top) = \mathrm{IGN}(V_iV_i^\top, d_i)$. In other words, we may input the dimension of the eigenspace to the single IGN. This has the same expressive power as having separate IGNs, but we have not done experiments to verify the learning capabilities of this variant; this is a good avenue for future work. We have added a note on this in Appendix B.3 in red text.

> “ Also, how is the runtime of BasisNet in Table 9?”

We have added runtimes in Appendix J.3 in red text:

In the spectral graph convolution experiments, for 100 training epochs on the same machine, GCN takes .435 seconds, ChebNet takes .675, DeepSets with SignNet takes 3.741 seconds, and DeepSets with BasisNet takes 6.196 seconds. SignNet and BasisNet are significantly more expensive than the GNN methods in this experiment, in large part because we use all 1024 eigenvectors of a 1024 node graph. In contrast, recall in Section 4.1 that SignNet only has a small overhead when used with GNNs for smaller molecular graphs.

From: ICLR.cc/2023/Conference/Paper1915/Authors
title: Author Response to a8ck
comment: We thank the reviewer for appreciating the problems that we work towards and the theoretical backing for our architectures.

> “Weaknesses: Some of the notions are not easy to understand from the text. Theorem 1 which is one of the main results of the paper should be discussed and explained more … While Theorem 1 is interesting and insightful, could you please provide an explanation of what it implies in practice? Does it mean that when the group is a product, we could do separate sums over the elements of the group to achieve invariance?”

Thank you for the feedback on this. We have added further explanation in Section 2.2 when stating the theorem (in red text). We may try to add even more explanation in a future version, if we can find the space to do so.

In practice, the product group $G_1 \times \ldots \times G_k$ is large and hard to deal with (of size $2^k$ for sign invariance). Our Theorem 1 shows that if we can make models for the much smaller and simpler $G_i$ groups (of size $2$ for sign invariance), then we can combine these models in a simple way to get invariance for the larger group, without losing any expressive power.

> “How does the computational complexity of IGN affect the overall complexity of the algorithm?”

We have now added a discussion of complexity in Appendix B.3, in red text.

For eigenvectors $V \in \mathbb{R}^{n \times k}$, SignNet forms $O(nkd)$ sized tensors with $\phi$, where $d$ is the hidden dimension or output dimension of $\phi$.

On the other hand, for bases of eigenspaces $V_1, \ldots, V_l$, with dimensions $d_i$ (where $\sum_i d_i = k$), BasisNet forms $O(n^2 l d)$ tensors, where $d$ is the hidden dimension of the $\phi_{d_i}$. Thus, BasisNet has an extra multiplicative $n$ factor when compared with SignNet. If we instead use $p$-order IGNs, then there is an extra multiplicative $n^{p-1}$ factor.

> “Is there an inductive bias towards locality in SignNet (as in other graph convolutional methods)?”

Yes, for the choices of $\phi$ that we take in most experiments, there is a local inductive bias. In particular, for our main graph learning experiments we take $\phi$ to be a graph neural network (GIN), as noted in Sections 4.1 and 4.2.

> “Figure 2 needs more explanation. Since the proposed positional encoding schemes can take as input node features, why do we need another GNN layer i.e., GNN(A,X,sigNet or basisNet(V,X)) on the top of that? Is that related to localisation aspects?”

There are a few reasons for this choice.
1. Efficiency: in Equation (6), note that adding $X$ to the SignNet would require processing it with $\phi$ up to $2k$ times, where $k$ is the number of eigenvectors. This is inefficient when compared with using a base GNN to only process $X$ once.
2. Initial Experiments: we ran some rough experiments with adding $X$ to the SignNet input, but this did not work well, though a more comprehensive study is required to fully rule out this design choice.
3. Comparison with baselines: this design choice of using SignNet to first learn positional encodings, and then processing $X$ afterwards allows controlled comparison to (a) GNNs without positional encodings, (b) baseline positional encoding methods (sign flipping, absolute values of eigenvectors). Our experiments in Table 1 thus clearly show benefits of SignNet.

It would be interesting for future work to try variations on this strategy, though.

Thanks again for the review, let us know if you have any further questions!

From: ICLR.cc/2023/Conference/Paper1915/Authors
title: Author Response to aMn3
comment: We thank the reviewer for appreciating the problem that we tackle, the writing of the paper, the universal approximation theory, and the experimental evaluation. We address your quoted concern:

> “Proposition 2, Theorem 2 and Proposition 3 are stated somewhat verbally/informally in the main paper. It would be ideal to include a mathematically rigorous statement in the main paper. In particular, given how commonly the term "universally approximates" is used in the paper, it was not rigorously defined in the main paper.”

Thank you for the feedback on this. We have added a precise definition of universal approximation in Section 2 (see red text), before our theoretical results. The precise definition of universal approximation that we use is:

**Definition (Universal Approximation):** A set of functions $\mathcal F_{\mathrm{model}}$ with domain $\mathcal X$ and output space $\mathcal Y$ is said to *universally approximate* a set of functions $\mathcal F_{\mathrm{target}}$ if for any $f \in \mathcal F_{\mathrm{target}}$, any $\epsilon > 0$, and any compact domain $\Omega \subseteq \mathcal X$, there exists a function $\tilde f \in \mathcal F_{\mathrm{model}}$ such that $||\tilde f(x) - f(x) || < \epsilon$ for all $x \in \Omega$.

After adding this definition, we think that the statements of Proposition 2, and Theorem 2 should be clearer. We have also added a footnote defining the meaning of “distinguishing graphs” (see red footnote under Prop 3), which should clarify Proposition 3. In particular, we say that a class of functions $\mathcal F_{\mathrm{model}}$ can *distinguish* two graphs $G_1$ and $G_2$ if there exists a function $f \in \mathcal F_{\mathrm{model}}$ such that $f(G_1) \neq f(G_2)$.

From: ICLR.cc/2023/Conference/Paper1915/Authors
title: Author Response to BB19
comment: We thank the reviewer for taking the time to review and for voting to accept our paper. As noted in our responses to other reviewers, we have made some changes to improve readability. Let us know if you have any further questions we can help you with!

From: ICLR.cc/2023/Conference/Paper1915/Reviewer_BB19
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, SignNet and BasisNet new neural structures are proposed, which are the same as the two key SY invariants displayed by eigenvectors. It is also proved that under certain conditions, they can approximate any continuous function of invariants with the expected eigenvectors. The experimental results show that the proposed method is superior to the forward baseline in molecular graph regression, learning expression graph representation and learning neural field on triangular grid.
strength_and_weaknesses: In this paper, SignNet and BasisNet new neural structures are proposed, which is interesting
clarity,_quality,_novelty_and_reproducibility: This paper is of fair quality and novelty.
summary_of_the_review: My research field is not related to this manuscript and I am not very familiar with the research content. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1915/Reviewer_aMn3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose SignNet and BasisNet which are invariant to the sign and rotation symmetries that exist in eigenspaces, particularly in the Laplacian context for graphs. They demonstrate a universal approximation theorem with the approach under some conditions. They evaluated SignNet and BasisNet with various experimental tasks/datasets. 
strength_and_weaknesses: Strengths:

- The task that the authors are tackling is of general interest and importance, especially given the prevalence of Laplacian eigendecompositions in the literature, and the method/architecture that they propose are to the best of my knowledge novel and original. 

- The paper is well written and clearly motivated. 

- Their theorem on universal approximation for spectral graph convolutions is significant. 

- Their experimental evaluation is comprehensive and demonstrates the efficacy of the proposed approach well. 

Weaknesses: 

- Proposition 2, Theorem 2 and Proposition 3 are stated somewhat verbally/informally in the main paper. It would be ideal to include a mathematically rigorous statement in the main paper. In particular, given how commonly the term "universally approximates" is used in the paper, it was not rigorously defined in the main paper.  Given that there can be different types of universal approximation, it would be best to rigorously define it in the main paper to avoid any ambiguity/confusion. 


clarity,_quality,_novelty_and_reproducibility: Clear paper, well-written, novel approach that has strong experimental evaluation. 
summary_of_the_review: Overall a strong paper. Would be stronger if they make their theoretical statements more clear/specifically stated. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1915/Reviewer_a8ck
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces two architectures SignNet and BasisNet that respect some basic symmetries inherited by spectral graph methods i) sign flips: if v is an eigenvector then so is −v; and (ii) more general basis symmetries: in higher dimensional eigenspaces there might be infinitely many choices of basis eigenvectors. The authors show that under certain conditions the proposed architectures are universal and can approximate any continuous function of eigenvectors with the proper invariances. Moreover, the networks are theoretically powerful for graph representation learning—they can provably approximate and go beyond both spectral graph convolutions and powerful spectral invariants. Experimental results are shown on molecular graph regression tasks, learning expressive graph representations, and texture reconstruction on triangle meshes. 
strength_and_weaknesses: Strength: Designing proper architectures that preserve symmetries and are more expressive than current ones is a very interesting and open topic in the graph machine learning literature. The proposed architectures in the paper are theoretically sound, and lead to nice integrations of symmetry-preserving positional encodings into a GNN framework. The results are adequately analysed and sufficiently supported.

Weaknesses: Some of the notions are not easy to understand from the text. Theorem 1 which is one of the main results of the paper should be discussed and explained more. 
clarity,_quality,_novelty_and_reproducibility: This is a relatively well-written paper, of high quality. The results are novel, and the authors provide enough material for reproducing them. 
summary_of_the_review: In summary, this is a strong paper with nice theoretical results. Some aspects that I think should be clarified are the following:

1) While Theorem 1 is interesting and insightful, could you please provide an explanation of what it implies in practice? Does it mean that when the group is a product, we could do separate sums over the elements of the group to achieve invariance? 

2) How does the computational complexity of IGN affect the overall complexity of the algorithm?

3) Is there an inductive bias towards locality in SignNet (as in other graph convolutional methods)? 

4) Figure 2 needs more explanation. Since the proposed positional encoding schemes can take as input node features, why do we need another GNN layer i.e., GNN(A,X,sigNet or basisNet(V,X)) on the top of that? Is that related to localisation aspects? 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No specific concerns. 
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1915/Reviewer_cG9j
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work investigates the function invariance w.r.t. eigenvectors of graph Laplacian to reduce the learning complexity, including sign invariance and basis invariance. Then it proposes corresponding SignNet and BasisNet to incorporate these two invariances where many popular models can be directly plugged in as basic models. Moreover, it provides theoretical guarantees that the proposed models are expressive enough to conduct tasks beyond eigenvectors. Finally, the proposed methods are evaluated by synthetic and real-world experiments.
strength_and_weaknesses: Pros:
1. The motivation of introducing invariances into the model naturally reduces the learning complexity, which is verified sufficiently by excellent performance on real tasks against strong baselines.
2. It is very easy to directly plug many popular GNNs into SignNet to achieve performance boost, as shown in Table 1.
3. The theoretical arguments are comprehensive, including constructing SignNet and expressiveness of SignNet/BasisNet.

Cons:
1. Is BasisNet included in any experiment other than Appendix J.3? I am wondering whether BasisNet suffers high complexity because there might be many distinct multiplicities $d_i$ so that too many IGNs are included. In that sense, is there any way to implement BasisNet efficiently? Also, how is the runtime of BasisNet in Table 9?
clarity,_quality,_novelty_and_reproducibility: The details are presented clearly. 

The quality of theoretical arguments is good, and it enjoys great numerical performance. 

The novelty and reproducibility are also good.
summary_of_the_review: I would like to suggest an acceptance.

=== Acknowledgement ===

Due to time limit, I did not check proof in Appendix. The theorems seem to be good intuitively.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper