From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Thank you for submitting your work to ICLR 2023. This paper stirred many discussions. The main standing issues with this paper are: a) the main observation presented in this paper is considered mostly known in the GNN/equivariant community: A square-root of the Laplacian / adjacency matrix (constructed with the eigen-decomposition) contains all edge information and therefore any set-universal function leads to universal models (maybe computable in exponential time). b) This model is not guaranteed to be *invariant/well-defined* since these representations are not unique, and we are not aware of universal and practical basis invariant nets. Although known fact, it might be still useful to publish such a work, however we feel it does not pass the bar for ICLR.
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: We are not misleading the discussion
comment: Thanks for your further detailed response. First, we want to note that we are not trying to mislead anyone in any sense in the last reply. For your question on the importance of ambiguity (also we do not fully agree), we believe that it is a proper and constructive review of this work. However, the strong reject evaluation of Reviewer 8PYU and Reviewer sg6T is **only based on that they believe our theorems are totally wrong**, and they insist on not responding to our requests to point out any errors in the theorem. For any authors submitting to ICLR, this is a very frustrating experience, as the reviewers do not admit their mistakes and re-evaluate the paper as they should. Therefore, the act that you recognize the correctness of our theorems (not necessarily meaning our papers can be accepted) has been an encouragement for us, and therefore, we keep urging the two reviewers to re-evaluate our paper in the last reply.

For your new comments, we admit that we indeed miss the paper [Rethinking Graph Transformers with Spectral Attention], which also combines eigenvalue and eigenvectors as inputs and achieve universality. The main difference is that we achieve this with an even simpler architecture (no interactions between nodes), and propose a simpler way to combine the two (no additional input dimensions compared to vanilla Laplacian embedding). For ambiguity, we arguably disagree with you that "if you directly use Laplacian embedding in graph transformer or augment GNN, you see nothing improvement and sometimes even hurt the performance.". For example, in Graphomer (table 5), using Laplacian embedding decreases the valid MAE significantly from 0.2276 to 0.1483 on PCQM4M-LSC. As for SignNet, in our experiment, we indeed do not observe any significant gain from it compared to RandSign. We believe that it still suggests that ambiguity is indeed not very important in certain scenarios. Though, inspired by your comments, we do come up with a new way to resolve the sign ambiguity without causing computation overhead as in SignNet. But it seems beyond the scope of the current paper, and we shall leave it to future work.

At last, thanks for your detailed and constructive comments and we will revise our paper accordingly. Have a good day!

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_gwSj
title: Please don't mislead the discussion
comment: Thank you for your response, however I don't wish my response to be misguided by your response. I have acknowledged that representing graph as eigenvectors and using a DeepSet without dealing with the ambiguity of eigenvector can be universal, which is similar to say that using MLP to directly process the adjacency matrix is universal. These "universality" is really misleading in this community, and I think it's harmful to publish this kind of argument (especially in title to be eye catching) for this community. In fact, as I pointed, [Rethinking Graph Transformers with Spectral Attention] already uses the same argument to argue that graph transformer is a universal model with laplacian embedding as positional encoding. Notice that deepset is weak than transformer in processing set, although both are universal for set functions. However, most of us still question the universality or ability of graph transformer, as we all know that the graph transformer doesn't handle the symmetry we want to deal with, and the universality is not what we truly want. 

When we talking about universality, we have to talk what kind of symmetry it holds. There is always a tradeoff between universality and equivariance, as equivariance adds strong hard constraints to the network. It is **FAULTY** to claim universality without considering symmetry. If you meet researchers from the equivariant network area, your claim really doesn't make any sense. 

Regarding your new results: I won't believe any result over TUDatasets, and for results over large ogb dataset, the performance is really bad. I think you better look at SignNet paper again. As I said, the graph transformer is already "proved" to be universal in the paper I mentioned, without dealing with symmetry we want. SignNet deals with these ambiguity directly and gain huge performance improvement. In fact, if you directly use Laplacian embedding in graph transformer or augment GNN, you see nothing improvement and sometimes even hurt the performance. 

To summarize, the contribution is really trivial, and the contribution is already existing in the literature. The author either doesn't have enough knowledge in the area of GNN/Transformer and Equivariant Network, or intends to mislead the reviewers to believe their contributions.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Thanks for acknowledging the correctness of our theorems (1/2)
comment: Thanks for your response! We address your further concerns in details.

---

**Q1. Correctness.** 

**A1.** We appreciate a lot for Reviewer gwSj for *careful reading of our theorems and proofs* and conclude that ***“there is no error in the theorems”*, which is opposed to the current evaluation of Reviewer 8PYU and Reviewer sg6T.** Instead, Reviewer 8PYU and Reviewer sg6T **give strong reject merely because they think our results are *wrong*:**

> Reviewer 8PYU: *“Unfortunately, I think that the proofs of Theorem 3 and 4 are not correct”. “There is a major flaw in the main theoretical theorems of this paper.”*
> 

> Reviewer sg6T: *“This is a fundamental flaw in a key argument made in the paper, and thus there is no option but to reject this paper at the moment”.*
> 

Yet, with discussions till now, none of them provides concrete evidence for any errors in our theorem. Therefore, **if Reviewer gwSj’s current evaluation is correct, i.e., there is no error in our theorems, then the current evaluation of Reviewer 8PYU and Reviewer sg6T would be unacceptably wrong**. Therefore, we strongly suggest that Reviewer 8PYU and Reviewer sg6T could 1) re-evaluate our paper; 2) give correct and supported evaluation instead of the current unsupported ones; and 3) spot concrete errors in our proofs if you insisted that our theorems are wrong.

---

**Q2.** **Concerns on novelty compared to Speilmann, and on the ambiguity.** 

> Using eigenvectors and eigenvalues can make the model as a universal approximator is a well-known result (that doesn't need a proof actually) in the community.
> 

**A2.** Thanks for bringing up Spielmann (2007)’s discussion on the use of eigenvectors for graph isomorphism, which is indeed related to ours. Though, his discussion is not rigorous, and he did **NOT** explicitly stated or proved the universality of using eigenvalue and eigenvectors for approximating permutation-invariant graphs with MLPs. To prove this, we actually leverage the universality of DeepSet Function, which is proposed and proven by Zaheer et al. (2017), and we do not know any prior works that have established this universality result. Please let us know if we miss some previous works here.

In fact, the fact that there are two expertise ICLR reviewers that totally cannot believe our results can hold, actually provides strong evidences that our universality result using only MLPs is a **very new, surprising, and even unbelievable** message to the ICLR community and GNN researchers. Given the lack of knowledge of this property in this community, I believe that our UBoN can refresh people’s mind and inspire more researches on message-passing-free GNNs.

**Comparison to Spielman’s dicussion and why ambiguity could be less important in UBoN.**  Indeed, for solving graph isomorphism with eigenvectors, as discussed by Spielmann, the ambiguity with exponential flippings could also make this algorithm very inefficient on deciding graph isomorphism. In comparison, the key difference in UBoN is that **we do not directly compare eigenvectors**, but adopt an MLP network to decide graph isomorphism based on **predicted outputs**. With the universal approximation of MLP networks, it is capable of fitting the ambiguities when we randomize the input signs and rotations (e.g, RandSign). And when they do, only a single forward pass is needed for graph isomorphism testing, which is quite different from Speilmann’s setting. Therefore, we believe that the ambiguity could be a less important issue for UBoN, as we further verified by experiments. 

**Evaluation of the effect of ambiguity.** To further examine whether ambiguity matters in real-world applications, we design the two variants, UBoN-sign and UBoN-basis. In particular, UBoN-basis does not have any ambiguity problem. As shown in Table 5, **theoretically fixing the ambiguity (UBoN-basis) provides little gains over the vanilla UBoN in real-world applications**. Meanwhile, RandomSign is already an effective strategy to resolve sign ambiguity.

| Method     |          MUTAG         |           PTC          |          COX2          |        PROTEINS        |         ENZYMES        |
|------------|:----------------------:|:----------------------:|:----------------------:|:----------------------------:|:----------------------:|
| UBoN       | $\mathbf{91.0\pm 5.4}$ |      $65.4\pm 5.6$     |      $84.1\pm 3.9$     | $\mathbf{75.7\pm 5.2}$ | ${57.2\pm 5.1}$ |
| UBoN + RandSign (RS)      | ${90.9\pm 4.8}$ |      $\mathbf{66.5\pm 6.5}$     |      $\mathbf{84.4\pm 5.2}$     | $\mathbf{75.7\pm 3.6}$ | $\mathbf{65.7\pm 6.3}$ |
| UBoN-sign  |      $89.9\pm 5.6$     |      $64.5\pm 7.8$     | ${84.2\pm 2.6}$ | $\mathbf{75.7\pm 2.8}$ |      $55.7\pm 6.2$     |
| UBoN-basis (no ambiguity) | $\mathbf{91.0\pm 5.3}$ | ${66.3\pm 6.8}$ | ${84.2\pm 2.6}$ |      $70.9\pm 4.1$     |      $44.5\pm 5.4$     |

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Thanks for acknowledging the correctness of our theorems (2/2)
comment: **Q3. Concerns on Performance**. 

> If the author really believe that universality itself can be the good stuff, please demonstrate your designed work with great performance trained on large dataset (you need a large dataset to overcome these ambiguity at least, and also with data augmentation).
> 

**A3.** As shown in Table 2, we compare our method with many other expressive GNNs like 1-2-3-GNN with sophisticated designs, and we find that our method could still outperform them on many tasks. In fact, many expressive higher-order GNNs do not bring better performance in practice while being computationally costly. The fact that our simple UBoN can achieve better expressive power (universal),  comparable or better performance (Table 2), using much less computation (Table 3) and parameters (Table 4) than these expressive GNNs suggests that we might need to reconsider the meaning of extensive studies on GNN expressiveness today. This is how we see the potential value of this work. 

As for the concerns on large datasets, we have added experiments on three large scale datasets, including two from OGB: DD (Yanardag & Vishwanathan, 2015), ogbg-molhiv and ogbg-moltoxcast (Hu et al., 2020) in **Appendix D**. As quoted below, we can see that UBoN can indeed attain good performance (comparable to GIN) on these large-scale datasets with much less (100x) parameters. 

| Model | ogbg-molhiv | ogbg-moltoxcast | DD | #params |
| --- | --- | --- | --- | --- |
| GIN | 75.58 ± 1.40 | 63.41 ± 0.74 | 75.3 ± 2.9 | 1885 k |
| BoN | 69.02 ± 2.30 | 58.02 ± 0.25 | 68.8 ± 3.5 | 16 k |
| UBoN | 73.41 ± 0.49 | 62.50 ± 0.32 | 76.1 ± 2.3 | 19 k |

---

Thank you again for careful reading and acknowledge the correctness of our results! Hope our elaborations and added experiments above could address your concerns. Please let us know if there is more to clarify. We are looking forward to your reply.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_gwSj
title: Regarding tradeoff between universality and ambiguity 
comment: The author has put a lot effort on clarifying the contribution and helping understanding the paper much better. However I have to admit that the content is somewhat already know in the literature. The author argued that Theorem 1 is the main contribution, while theorem 3 and theorem 4 are from the SignNet paper. However even Theorem 1 has been mentioned in one graph transformer paper [Rethinking Graph Transformers with Spectral Attention]. I personally find that theorem 2's result is kind of interesting. So just to summarize, there is no error in the theorems, but there is not much novelty I can find. Using eigenvectors and eigenvalues can make the model as a universal approximator is a well-known result (that doesn't need a proof actually) in the community if doesn't deal with the ambiguity and symmetry inside the eigenvectors and eigenvalues. Putting this result as a paper, to be honest, is kind of trivial. 

To be more specific, in the paper of [Spectral Graph Theory and its Applications] of Daniel A. Spielman. He has written: 

>Given the sucess of spectral grpah drawing, one would think that even if two graphs have the same eigenvalues, an examination of a few eigenvectors would suffice to distinguish non-isomorphic graphs. After all, the eigenvectors provide coordinates for every vertex in a graph, and these coordinates do not depend upon the labeling of the vertices. Often this works. But, there are three problems that prevent this approach from working in general:

>1. Eigenvectors are only determined up to sign. That is, if v is an eigenvector then −v will be one also. So, to compare the spectral embeddings of two graphs made from k eigenvectors, one must try all 2k possible ways of flipping their signs.
>2. There are some graphs for which all non-trivial eigenvalues have high multiplicity, in which case one must consider not just coordinate flips in the spectral embedding, but also rotations.
>3. Some vertices may be given the same coordinates by many eigenvectors. In fact, one needs to consider all eigenspaces to guarantee that each vertex receives unique coordinates.

I don't really know why the author keeping insisting on universality without addressing these problems. If the author really believe that universality itself can be the good stuff, please demonstrate your designed work with great performance trained on large dataset (you need a large dataset to overcome these ambiguity at least, and also with data augmentation). The current result is kind of bad, which indicates these problems mentioned by Prof. Spielman need to be addressed.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title:  Fürer's spectral features are different from ours
comment: First, we hope that our previous relies have addressed your concerns on the NP property of UBoN, and we would be glad to hear your feedback on it.

Second, for your new concerns on Martin Fürer's work. We remark that his discussion only applied to **a specific type of spectral invariants that he defined (Sec 2.3)**, which is **different from our proposed EASE**. In particular, his result does **not** exclude the possibility for all possible spectral invariants to be universal. Since EASE is different from his spectral variants, our universality result is not contradictory to his. In fact, the original spectral embedding (SE) that we discuss is indeed not universal, which is exactly why we develop EASE to restore the eigenvalue information. Hope it could address your concerns.

Third, we still strongly insist the reviewer look into the details of our paper, instead of drawing endless conjectures on it. ***Our result can hold by itself if no error is spotted in the proof***. Therefore, if you would insist that our theory is wrong, please **point out any specific technical errors that we made in our paper and our proof**. We are looking forward to your further reply.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_8PYU
title: spectral features cannot provide universality
comment: The power of spectral invariants is studied in "On the power of combinatorial and spectral invariants" by Martin Fürer. Your result contradicts the results obtained in this paper.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: UBoN is NP because training NN is NP-hard
comment: Thanks for your response. Here we further clarify our methods on your concerns about whether UBoN could solve / approximate the maximum independent set problem in polynomial time, and thus violating NP$\neq$P.

To clarify, as in other universal GNNs, our universality result is about the **approximation** ability of our UBoN (whether a desired $\theta^*$ **exists** in the hypothesis space), NOT about the computation complexity to actually find $\theta^*$ in the hypothesis space. In fact, a classic result by Blum & Rivest [1] shows that **training NN itself is an NP-complete problem**, which makes all NN solutions to these combinatorial problems fundamentally NP solutions. Therefore, the fact that UBoN is capable of universal approximation **has no violation of the NP$\neq$P principle** (if it holds).

Besides, we note that this universal property studied in our paper (Definition 2) is not our own invention. In fact, it is commonly adopted by existing studies on universal GNNs [2-7], and we have no special modification on it. The main differences to prior universal GNNs [2-7] is that we are the first to achieve universal approximation for permutation-invariant functions without using any message passing layers.

Hope this explanation could address your concerns, and please let us know if there is more to clarify.

**References:**

[1] Blum, A., & Rivest, R. (1988). Training a 3-node neural network is NP-complete. Advances in neural information processing systems, 1. [https://proceedings.neurips.cc/paper/1988/file/3def184ad8f4755ff269862ea77393dd-Paper.pdf](https://proceedings.neurips.cc/paper/1988/file/3def184ad8f4755ff269862ea77393dd-Paper.pdf)

[2] Yarotsky, D. (2022). Universal approximations of invariant maps by neural networks. *Constructive Approximation*, *55*(1), 407-474.

[3] Keriven, N., & Peyré, G. (2019). Universal invariant and equivariant graph neural networks. *Advances in Neural Information Processing Systems*, *32*.

[4] Maron, H., Fetaya, E., Segol, N., & Lipman, Y. (2019, May). On the universality of invariant networks. In *International conference on machine learning* (pp. 4363-4371). PMLR.

[5] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. In *International Joint Conferences on Artificial Intelligence Organization* (pp. 2112-2118).

[6] Kanatsoulis, C. I., & Ribeiro, A. (2022). Graph Neural Networks Are More Powerful Than we Think. *arXiv preprint arXiv:2205.09801*.

[7] Brüel Gabrielsson, R. (2020). Universal function approximation on graphs. *Advances in Neural Information Processing Systems*, *33*, 19762-19772.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_8PYU
title: still a problem
comment: If you can approximate any permutation-invariant function with your proposed UBoN, then you can approximate the maximum independent set number of a graph. In general, the maximum independent set problem cannot be approximated to a constant factor in polynomial time (unless P = NP).
Your discussion about MLPs approximating continuous functions is irrelevant.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: 3rd Response to Reviewer sg6T (1/2) 
comment: Thanks for your quick and detailed reply. We further elaborate our work to you for clarifying some misunderstandings.

---

**Q1.** The authors simply must realize that they are claiming a universality result while using *deterministic* features which are known to not be sufficient to solve graph isomorphism.

**A1**. We are **NOT** claiming that we solve the graph isomorphism using a deterministic feature. One thing you must be aware is that all MLP can obtain is a (very close) ***approximation*** to the exact solution, **not the exact solution**. This is exactly why even if MLP is well known to be a universal approximator (which means it can approximate the solution to any problem, like graph isomorphism), it does not actually solve an NP problem in poly time.  

> I asked the authors for a concrete proof, which the replies simply have still not provided.
> 

**We have already provided a very clear and concrete proof of *our claimed universality* in Appendix B.1, yet you keep refusing to read and comment on it**. In this proof, our definition of universality is clearly stated in Definition 2, and **this definition of universality is adopted by all existing universal GNNs [1-6], not only ours**. Second, every step of our proof of universality is strict, and it is not long (within one page). You can point out any error or unclear stuff **in our proof**, and we would like to explain it to you or revise it. Instead, you only accuse our proof to be incomplete **based on your “*believe*”**. We must remind you that **it is the reviewer’s duty to give concrete reasons for your judgment**, particularly when you say that our theoretical result is wrong. We are innocent until you spot some errors (or unclear proof steps) we actually made. Otherwise, your judgment is not grounded and is not acceptable. We strongly suggest the reviewer(s) to **go read our proof and find some concrete errors as you should**. 

---

**Q2**. The authors need to 1) prove the first claim: EASE features are overly discriminative, and thus that non-isomorphic graphs always mapped to distinct outputs; and 2) you need to be very explicit about how the MLP is necessary for the result.

**A2**. **First problem.** We first remark that because non-isomorphic graphs cannot have the same EASE representations (proven in the first point in our previous reply), **the EASE representations are always different for non-isomorphic graphs**. Second, we remark that our universality result does not guarantee that “non-isomorphic graphs always mapped to distinct outputs”. We are not saying this, and nobody can achieve this. **All existing universality results on GNNs [1-6]** only say that these universal GNNs are capable of approximating such a discriminative function (graph isomorphic mapping) up to any precision, as in the definition of universality in Def 2. As our UBoN can approximate such a function, UBoN is **capable of telling two arbitrary non-isomorphic graphs apart** (what we claimed in our paper) by mapping the two (with different EASEs) to different values, while **$k$-WL GNNs (like GIN) cannot achieve this** for some graphs.

**Second problem.** The MLP is necessary because our goal is not only to solve graph isomorphism, but to approximate any graph invariant function $f$, to achieve the universality expressiveness (Def 2), which is the actual property that we require for practical graph-level tasks.

---

**Q3**. I am afraid the authors are building on redundancies, which fundamentally discard the desirable symmetries of eigen-decomposition (I believe this is what Reviewer gwSj was alluding to), to make universality work through an MLP. This proof, I am afraid, is rather trivial and claiming it to offer universality feels misleading.

**A3**. We note that existing universal GNNs often sacrifice some sort of symmetry for better expressiveness.  ****As explained in our reply to Reviewer gwSj, the well-known ***random feature technique for GCNs [4]* even discards permutation invariance to attain universality (and it works well in practice)**, while **our UBoN still preserves permutation invariance without any message passing layers**. 

**Why it is non-trivial?** This property is rather nontrivial for a graph MLP, because all previous graph MLPs, **either a vanilla MLP over $\mathbf{X}$ or an MLP with joint input $\mathbf{[X,A]}$, do not satisfy permutation invariance**. In fact, it relies on the key property of **permutation equivariance and structural perseverance of our proposed EASE representation**, NOT MLP.

**Recovering Symmetry via Expectation**. To restore permutation invariance, random feature technique relies on taking an expectation over all random features. If this is allowed, in EASE, we can also take expectation over random sign and basis variations. As a result, the resulting model still becomes symmetry w.r.t. decomposition, and consequently, obeys all symmetry as desired.

(more below)

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: 3rd Response to Reviewer sg6T (2/2)
comment: 
**Practical Performance**. Nevertheless, as can be seen from Table 5, ensuring sign and basis invariance with our UBoN-sign and UBoN-basis variants **does not bring real benefits in real-world datasets**, and our UBoN (with universal expressive power) can **bring significant improvement over the vanilla BoN and performs comparably to existing GNNs**, without these symmetries. This suggests that these symmetries are not that important in practical use. Therefore, we do not fully understand why lacking some of these unimportant properties, as in other universal GNNs, would result in strong reject. Perhaps we should keep an open mind and focus on the practical aspects of model expressiveness.

---

Thanks for your detailed reply. And hope our elaborations above could address your concerns. Look forward to your response.

**References:**

[1] Yarotsky, D. (2022). Universal approximations of invariant maps by neural networks. *Constructive Approximation*, *55*(1), 407-474.

[2] Keriven, N., & Peyré, G. (2019). Universal invariant and equivariant graph neural networks. *Advances in Neural Information Processing Systems*, *32*.

[3] Maron, H., Fetaya, E., Segol, N., & Lipman, Y. (2019, May). On the universality of invariant networks. In *International conference on machine learning* (pp. 4363-4371). PMLR.

[4] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. In *International Joint Conferences on Artificial Intelligence Organization* (pp. 2112-2118).

[5] Kanatsoulis, C. I., & Ribeiro, A. (2022). Graph Neural Networks Are More Powerful Than we Think. *arXiv preprint arXiv:2205.09801*.[6] Brüel Gabrielsson, R. (2020).

[6] Brüel Gabrielsson, R. (2020). Universal function approximation on graphs. *Advances in Neural Information Processing Systems*, *33*, 19762-19772.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_sg6T
title: Reviewer Response
comment: Thank you for your quick response:

> We are afraid you misunderstood the definition of EASE. ... . As clearly stated in Section 4.2, EASE is NOT eigenvectors  (as you suggest). Instead, EASE is a combination of eigenvalues and eigenvectors

I am aware that this is a combination of eigenvectors and eigenvalues, yet the implications of your statement about the properties of EASE still remain implausible, in my opinion. For the sake of the argument, let's assume the statement below is correct: 

> Therefore, if any two graphs have identical EASE representations, it directly implies that they have the same adjacency matrices, and of course, two graphs having the same adjacency matrices are isomorphic graphs. Therefore, the case you mentioned, that two non-isomorphic graphs having the same EASE representations, is simply impossible.

The authors simply must realize that they are claiming a universality result while using *deterministic* features which are known to not be sufficient to solve graph isomorphism. Recall in my initial review that I asked the authors for a concrete proof, which the replies simply have still not provided. If your argument is that EASE features are overly discriminative, and thus that non-isomorphic graphs always mapped to distinct outputs, and that isomorphic graphs are mapped to potentially different outputs, then you need to mathematically prove the first claim, which is precisely the challenge I raised, and (ii) you need to be very explicit about how the MLP is necessary for the result. I still am extremely concerned about the construction, and believe the proof to be incomplete. 

As for your isomorphic example, I am aware that eigenvectors can be scaled differently in the decompositions, leading to different decompositions. However, these are trivial manipulations and don't align with how your algorithm operates (deterministically). Indeed, how/when would EASE use negative and it use positive should ideally not be a factor in the result. I am afraid the authors are building on redundancies, which fundamentally discard the desirable symmetries of eigen-decomposition (I believe this is what Reviewer gwSj was alluding to), to make universality work through an MLP. This proof, I am afraid, is rather trivial and claiming it to offer universality feels misleading. I therefore strongly recommend that the authors carefully re-evaluate their work and return with a more constructive argument.

> As in the ICLR Code of Conduct, we believe that it is important for the ICLR openreview process to be objective and avoid subjective wording, concerning you saying “quite disappointing”

I am happy to retract the phrase if you found this subjective: This was never the intent. In any case, this does not affect the problems I have raised in any way whatsoever. I will also insist that I am **on the authors' side** and only trying to help their work in the best way I can. I understand my comments are not pleasant given the work you have undoubtedly put into the paper, but they serve a purpose, which is to solve the fundamental issues in your universality claim, which have also been spotted by 2 other reviewers. 

All in all, I will keep my verdict.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Further Response to Reviewer sg6T
comment: Thanks for your response, though you still misunderstood our method, as elaborated below.

**First, we focus on your question on non-isomorphic graphs:**

> If two non-isomorphic graphs have the same **eigenvectors**, **which *is possible***
 as EASE cannot solve isomorphism checking, then, by definition, providing the same input will yield the same output, and thus UBoN will return the same output on different graphs, violating universality.
> 

We are afraid you misunderstood the definition of EASE. Denote the eigendecomposition of $A$ as $A=U\Lambda U^\top$. As clearly stated in **Section 4.2**, **EASE is NOT eigenvectors** $U$ (as you suggest). Instead, **EASE is a combination of eigenvalues and eigenvectors**, i.e.,$P={U\Lambda}^\frac{1}{2}$. A key property of EASE is that we can reconstruct the original adjacency matrix from EASE by $A=PP^\top=U\Lambda U^\top$. Therefore, **if any two graphs have identical EASE representations, it directly implies that they have the same adjacency matrices, and of course, two graphs having the same adjacency matrices are isomorphic graphs.** Therefore, the case you mentioned, that two non-isomorphic graphs having the same EASE representations, is simply ***impossible***.

We have explained it to you in our previous reply in **A2 (second point)**. Hope the elaboration above could further ease your concern.

**Second, we revisit your claim on isomorphic graphs:**

> what the authors mention in their reply, namely, that two isomorphic graphs will be given distinct EASE distinct representations, I must highlight, is just **not possible**, as the eigenvectors and eigenvalues should be the same
> 

This claim is also wrong, as **it is actually possible that two isomorphic graphs have different EASE representations**. We give a simple example here. Consider a graph $A$ with EASE representation $P={U\Lambda}^\frac{1}{2}$ as above. Multiplying $U$ with -1, we have $\tilde{U}=-U$, and obviously, $A=\tilde{U}\Lambda\tilde{U}$ is still a valid eigendecomposition of $A$. Then, based on this new decomposition, we can define a new EASE representation $\tilde{P}=\tilde{U}\Lambda^{1/2}$, which also satisfies $A=\tilde{P}\tilde{P}^\top=U\Lambda U^\top$. As a result, we prove our claim that **the same graph $A$ can have different EASE representations, e.g., $P$ and $\tilde{P}=-P$**.

---

Hope our elaboration above could address your concerns. As in the ICLR Code of Conduct, we believe that it is important for the ICLR openreview process to be objective and avoid subjective wording, concerning you saying “quite disappointing”.  We tried our best to address your concerns, and in fact, some of your concerns above have already been discussed in our previous reply. 

Therefore, it would be more objective if you could look into the details of this paper for a re-evaluation. If you would insist that our theory is wrong, please **point out any specific technical errors that we made in our paper and our proof**. We are looking forward to your reply.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_sg6T
title: Reviewer Response
comment: I am afraid that I find the authors' response quite disappointing. First, my problem with the proof is not what the authors mention in their reply, namely that two isomorphic graphs will be given distinct EASE distinct representations (which, I must highlight, is just **not possible**, as the eigenvectors and eigenvalues should be the same). In fact, it's the other way around, as I mentioned quite clearly in my original review. If two non-isomorphic graphs have the same eigenvectors, which *is possible* as EASE cannot solve isomorphism checking, then, by definition, providing the same input will yield the same output, and thus UBoN will return the same output on different graphs, violating universality. This has nothing to do with the later MLP or any model construction that follows. Of course, one can rely on MLP universality to map different representations to the same output, that much I do not argue with. However, my point is that you **do not even have different representations** guaranteed by UBoN, and this part of the proof **does not** and **should not** depend on an MLP. 

Based on the authors' reply, I am now increasingly convinced that the proof in question is erroneous and falsely over-relying on MLPs. I therefore persist with my verdict.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer MeG5 (1/2)
comment: We thank Reviewer MeG5 for appreciating the simplicity and theoretical value of our work. We address your main concerns in the following points.

---

**Q1**. The proposed model is only evaluated on relatively small and old datasets. Experiments should be done with more recent larger datasets such as OGB.

**A1**. Following your suggestions, we compare our UBoN with GIN and vanilla BoN on the following large scale graph-level datasets.

| Model | ogbg-molhiv | ogbg-moltoxcast | DD  | #params |
| --- | --- | --- | --- | --- |
| GIN | 75.58 ± 1.40 | 63.41 ± 0.74 | 75.3 ± 2.9 | 1885 k |
| BoN | 69.02 ± 2.30 | 58.02 ± 0.25 | 68.8 ± 3.5 | 16 k |
| UBoN | 73.41 ± 0.49 | 62.50 ± 0.32 | 76.1 ± 2.3 | 19 k |

The results show UBoN can indeed greatly improve the performance of vanilla BoN and reach comparable performance to GIN. On the other hand, the number of parameters in our model (19k) is much less than GIN (1885k, shown on the OGB leaderboard). This shows that our UBoN is indeed a promising efficient alternative to message passing GNNs.

---

**Q2**. The complexity of eigenvalue decomposition could be high in case of large graphs. Could approximation schemes for eigenvalue decomposition be employed? What would be the impact of those schemes on the model’s performance?

**A2**. Usually, for graph-level tasks, the sizes of graphs are not very large, and the computation time for eigendecomposition is acceptable. For large graphs, there are indeed approximation schemes to speed up the computation. As mentioned in our paper, we only need the first $k$ eigenvectors as node features, so calculating all eigenvectors is not necessary. Papers such as [1] provide efficient algorithms for solving large top-$k$ graph eigenproblems, and many machine learning libraries also provide such functions (e.g., `eigh` in SciPy, `eigs` in MATLAB), which would be helpful to reduce the computation overhead for eigendecomposition.

We also provide the pre-processing time on larger datasets:

| Dataset | #Graphs | #Avg Nodes | Pre-processing time | Training time |
| --- | --- | --- | --- | --- |
| ogbg-molhiv | 41127 | 25.5 | 37.04 s | 10124.55 s |
| ogbg-moltoxcast | 8576 | 18.8 | 8.87 s | 7636.93 s |
| DD  | 1178 | 284.3 | 26.44 s | 3746.40 s |

It can be seen that even on datasets with larger number of graphs or nodes, the pre-processing time is relatively less than the training time, and the main training cost of our UBoN is simply an MLP network, which is very fast to train without any message passing steps.

References:

[1] Sgherzi, F., Parravicini, A., Siracusa, M., & Santambrogio, M. D. (2021, May). Solving Large Top-K Graph Eigenproblems with a Memory and Compute-optimized FPGA Design. In *2021 IEEE 29th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)* (pp. 78-87). IEEE.

---

**Q3**. Why does SE use the Laplacian matrix while EASE the normalized adjacency matrix? In the case of EASE, the same theoretical results could be obtained if the Laplacian matrix was employed.

**A3**. Yes, it is true. Since $\mathbf{L}^\mathrm{sym}=\mathbf{I}-\hat{\mathbf{A}}$, it is equivalent to using the eigendecomposition of the Laplacian matrix and the normalized adjacency matrix. Their eigenvectors are the same and their corresponding eigenvalues sum up to 1.

---

**Q4**. Several questions on function definition.

**A4**.

> In definition 1, what is the codomain of function $f$? It would be nice if the authors provided some examples of permutation invariant functions.

The codomain of function $f$ can be $\mathbb{R}$ (real value) or $\mathbb{R}^k$ (real vector), depending on the task. An example of permutation invariant function is a function that maps each graph to its number of vertices. This function is permutation invariant, since permuting the ordering of vertices does not change the number of vertices.

> What is a continuous graph invariant function? Does a function that maps each unique weighted graph to a different vector (and can thus solve graph isomorphism) belong to this set of functions?

A continuous graph invariant function $f$ is a continuous function on graphs that are invariant to the permutation of nodes, a.k.a. the permutation invariance property defined in Definition 1. The function for solving graph isomorphism indeed belongs to this class. Because our UBoN can approximate this function up to arbitrary precision, UBoN is thus capable of solving the graph isomorphism problem well.

For consistency and clarity, we have modified the name as permutation invariant function in the revision.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer MeG5 (2/2)
comment: **Q5**. How the UBoN model compares against 1-WL and its higher order variants, in terms of their ability to distinguish non-isomorphic graphs. It has been shown that the spectra of most matrices lie between 1-WL and 2-WL. I would like the authors to comment on that since the expressive power of UBoN is mainly empirically evaluated.

**A5**. As mentioned in the paragraph below Theorem 1, **because UBoN is universal, its expressive power is strictly stronger than any $k$-WL algorithms**. In fact, UBoN **has the ability to distinguish any pair of non-isomorphic graphs**, while $k$-WL and its higher order variants always fail on some of them.

---

**Q6**. I would not expect GCN to be in the list of baselines on the EXP dataset since GCN is known to be less powerful than 1-WL. I suggest the authors replace GCN with GIN which can achieve the same expressive power as 1-WL. Moreover, I would expect some baselines that are more expressive than 1-WL such as the ones of Table 2.

**A6**. We have added GIN and some other expressive models in our baselines. The results are as follows.

| Model | Accuracy (%) |
| --- | --- |
| GIN | 50.0 |
| PPGN | 50.0 |
| 1-2-3-GCN-L | 50.0 |
| 3-GCN | 99.7 ± 0.0 |
| UBoN | 99.8 ± 0.5 |

The accuracies of PPGN, 1-2-3-GCN-L, and 3-GCN are from [1]. UBoN still achieves the best performance among all models.

References:

[1] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. *arXiv preprint arXiv:2010.01179*.

---

**Q7**. Since UBoN-sign and UBoN-basis are more powerful than UBoN, I would expect them to outperform UBoN on all datasets. However, this is not the case as illustrated in Table 5. How can those results be explained? Does the same pattern hold for training performance as well?

**A7**. We are afraid there are some misunderstandings here. Both UBoN, UBoN-sign, and UBoN-basis are universal GNNs. Therefore, theoretically, they have the same (optimal) expressive power, in the sense that they can all approximate arbitrary permutation invariant functions.

The additional sign and basis invariance is to further ensure UBoN has certain invariance properties, which our experiments (Table 5) show that it is not actually necessary for applying UBoN to real-world graphs as they bring no clear benefits. This could be due to reasons from two aspects:

- **Data Property**. First, the two properties are not important in real-world graphs.
- **Generalization**. Second, further ensuring sign and basis invariance enlarges the input size and the model size, which could lead to **worse generalization, e.g., overfitting**. As mentioned in [1], the representation power and generalization are two aspects for a GNN to obtain good downstream performance.

References:

[1] Garg, V., Jegelka, S., & Jaakkola, T. (2020, November). Generalization and representational limits of graph neural networks. In *International Conference on Machine Learning* (pp. 3419-3430). PMLR.

---

**Q8**. The paper does not cite some related works. The proposed model is similar to that of [1] where the matrix of node attributes is concatenated with a matrix that contains structural node features and the resulting matrix is fed into DeepSets. However, the structural features employed in this paper (from the eigenvalue decomposition) make the proposed method sufficiently original. Furthermore, the authors provide some theoretical results with regards to UBoN’s expressive power.

**A8**. We thank Reviewer MeG5 for appreciating our theoretical results. We have added [1] to our related works. Both [1] and our work design positional encodings for node features. [1] focuses on the local neighborhood of each node, while our work focuses on utilizing the eigendecomposition of the adjacency matrix.

References:

[1] Dasoulas, G., Nikolentzos, G., Seaman, K., Virmaux, A., & Vazirgiannis, M. (2021, June). Ego-based entropy measures for structural representations on graphs. In *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 3210-3214). IEEE.

---

Thank you again for your detailed comments, and hope our response could address your concerns. Please let us know if you have additional questions.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer ok4x
comment: We thank Reviewer ok4x for appreciating our theory and the simplicity of the approach. We address your concerns in the following points.

---

**Q1**. Missing related work “Universal Function Approximation on Graphs”, (2020), Brüel-Gabrielsson should be cited and discussed.

**A1**. Following your suggestions, we have added this paper to our related works. This work and our work both design universal function approximators on graphs. The difference is that their work focuses on building universal networks on subgraphs, while our work achieves universality by augmenting node features.

---

**Q2**. What about other work that speeds up graph learning?

**A2**. There have been many works that aim to speed up graph learning in many ways, such as implementing simpler network structure [1,2], optimizing the training procedure [3], etc. [1] proposes a linear model that works well on node classification, but as the authors point out in their paper, such linear networks do not work well on graph-level tasks. [2] proposes an MLP model with NContrast loss that provides the model with structural information, but computing the NContrast loss is computationally expensive during training. In our work, the computation of eigendecompositions of our MLP model happens during pre-processing, thus it is more computationally efficient. We have added these discussions to the related work section.

References:

[1] Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., & Weinberger, K. (2019, May). Simplifying graph convolutional networks. In *International conference on machine learning*  (pp. 6861-6871). PMLR.

[2] Hu, Y., You, H., Wang, Z., Wang, Z., Zhou, E., & Gao, Y. (2021). Graph-MLP: node classification without message passing in graph. *arXiv preprint arXiv:2106.04051*.

[3] Kaler, T., Stathas, N., Ouyang, A., Iliopoulos, A. S., Schardl, T., Leiserson, C. E., & Chen, J. (2022). Accelerating training and inference of graph neural networks with fast sampling and pipelining. *Proceedings of Machine Learning and Systems*, *4*, 172-189.

---

**Q3**. What about the computational cost of spectral decomposition on large graphs?

**A3**. On larger graphs, there are indeed approximation schemes to speed up the computation of eigendecompositions. As mentioned in our paper, we only need the first $k$ eigenvectors as node features, so calculating all eigenvectors is not necessary. Papers such as [1] provide efficient algorithms for solving large top-$k$ graph eigenproblems, and many machine learning libraries also provide such functions (e.g., `eigh` in SciPy, `eigs` in MATLAB), which would be helpful to reduce the computation overhead for eigendecomposition.

We also provide the pre-processing time on larger datasets:

| Dataset | #Graphs | #Nodes | Pre-processing time | Training time |
| --- | --- | --- | --- | --- |
| ogbg-molhiv | 41127 | 25.5 | 37.04 s | 10124.55 s |
| ogbg-moltoxcast | 8576 | 18.8 | 8.87 s | 7636.93 s |
| DD  | 1178 | 284.3 | 26.44 s | 3746.40 s |

It can be seen that even on datasets with larger number of graphs or nodes, the pre-processing time is relatively less than the training time, and the main training cost of our UBoN is simply an MLP network, which is very fast to train without any message passing steps.

---

We thank Reviewer ok4x again for the comments. Please let us know if you have additional concerns.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer gwSj (1/2)
comment: We thank Reviewer gwSj for careful reading and detailed comments of our work, though there might be some misunderstandings of our theory.

---

**Q1**. The universality proved in Theorem 1 is trivial. The true universality we care is the model that obeying all symmetries of the input. Taking an example, one can call MLP over the original adjacency universal by ignoring the permutation invariance. Back to the universality mentioned in the paper, the model DOES not obeying the sign invariant and basis invariant in Theorem 1.

**A1**. We respectfully **disagree the claim that “*The true universality we care is the model that obeying all symmetries of the input*”**. Up to now, there are many studies on universal GNNs, however **many (practical) universal GNNs do not obey “all symmetries of the input”, e.g. [1,2].** For example, by adding random features to the input, RNI (random feature initialization) [1] **even** **break permutation invariance**. However, **these non-invariant techniques (like RNI)** **do show better expressive power and bring practical benefits in many real-world graphs,** and we believe they are meaningful explorations. The success of these techniques shows that **although desirable, it is *not necessary for a GNN to obey all symmetry, and sometimes we can trade some symmetries for better expressive power***. Indeed, our experiments on UBoN also show that 1) UBoN improves vanilla BoN a lot by being universal; and 2) further ensuring sign and basis invariance does not bring many benefits on real-world data.

**Comparison**. Our UBoN still has advantages over previous universal GNNs. First, it is the first universal GNN that does not require any graph convolution, making it an efficient method for practical use. Second, **UBoN obeys** **permutation invariance (Theorem 1), which is a rare property since many universal GNNs do not obey, neither the vanilla MLP you mentioned**. In RNI, they claim that they could further ensure permutation invariance by taking an expectation over all random features. **If this is acceptable, we can also easily ensure sign and basis invariance of UBoN by averaging over all random rotations of EASE**. Empirically, in Table 5, we also show that **further explicitly ensuring sign and basis invariance does not bring benefits on real-world data**. These facts show that **sign and basis invariance are not necessary desiderata in UBoN, although we can enforce it by taking an expectation over random rotations like RNI.**

References:

[1] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. arXiv preprint arXiv:2010.01179.

[2] Murphy, R., Srinivasan, B., Rao, V., & Ribeiro, B. (2019, May). Relational pooling for graph representations. In International Conference on Machine Learning (pp. 4663-4673). PMLR.

---

**Q2.** When taking sign and basis invariance into consideration, the universality is proved in Derek’s paper and is not the contribution of this paper.

**A2**. Theorem 3 and Theorem 4 are indeed Derek’s results, but here we mainly adopt as a direct extension of UBoN for ensuring sign and basis invariance. And the main message that we try to deliver is that sign and basis invariance are actually not important, as these extensions do not bring actual benefits. **The main contribution of this work is still to develop UBoN and prove its permutation invariance and universality in Theorem 1.**

---

**Q3**. With the adjacency matrix available, the eigenvalue information can be recovered from $\mathbf{AU}$ which is a single message passing step.

**A3**. Indeed it is. Still it requires a message passing layer to achieve this goal, while **the main goal of this work is to show that an MLP-like network alone is capable of universal approximation on graphs**. Exploring simplicity in neural network designs is an important and ever-occurring research topic in deep learning, with representative works like **ResNet, Transformer, and MLP-Mixer**, which leads revolutionary changes in the past years. In the graph literature, how to develop GNNs without message passing is also a challenging and widely-studied problem [1,2,3]. Therefore, we believe that our work is a meaningful exploration from a theoretical perspective, and it could potentially enlighten new research for alternative simple yet expressive paradigms for designing GNNs.

References:

[1] Hu, Y., You, H., Wang, Z., Wang, Z., Zhou, E., & Gao, Y. (2021). Graph-MLP: node classification without message passing in graph. *arXiv preprint arXiv:2106.04051*.

[2] Chen, L., Chen, Z., & Bruna, J. (2020). On graph neural networks versus graph-augmented mlps. *arXiv preprint arXiv:2010.15116*.

[3] Zhang, S., Liu, Y., Sun, Y., & Shah, N. (2021). Graph-less neural networks: Teaching old mlps new tricks via distillation. *arXiv preprint arXiv:2110.08727*.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer gwSj (2/2)
comment: **Q4**. Small TUDatasets are not meaningful in evaluating expressivity, and the larger datasets in OGB and Benchmarking GNN like ZINC should be used.

**A4**. Following your suggestions, we compare our UBoN with GIN and vanilla BoN on the following large scale graph-level datasets.

| Model | ogbg-molhiv | ogbg-moltoxcast | DD  | #params |
| --- | --- | --- | --- | --- |
| GIN | 75.58 ± 1.40 | 63.41 ± 0.74 | 75.3 ± 2.9 | 1885 k |
| BoN | 69.02 ± 2.30 | 58.02 ± 0.25 | 68.8 ± 3.5 | 16 k |
| UBoN | 73.41 ± 0.49 | 62.50 ± 0.32 | 76.1 ± 2.3 | 19 k |

The results show UBoN can indeed greatly improve the performance of vanilla BoN and reach comparable performance to GIN. On the other hand, the number of parameters in our model (19k) is much less than GIN (1885k, shown on the OGB leaderboard). This shows that our UBoN is indeed a promising efficient alternative to message passing GNNs.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer sg6T
comment: We thank Reviewer sg6T for the detailed comments on our theory, though we think that there are some misunderstandings of our proof.

---

**Q1**. The proof does not address the uniqueness of the resulting representations with respect to the graph isomorphism problem … [it] implicitly assumes, without proof, that the set representations from EASE correspond directly with graph isomorphism.

**A1**. We are afraid you misunderstood our method. In fact, the universality of UBoN does ***not*** require that two isomorphic graphs have the same (unique) EASE representations, i.e., the ***same inputs* of UBoN**. Instead, to tell isomorphism, we **only need the two graphs have the same *outputs*** in UBoN. Because UBoN can universally approximate a graph isomorphism mapping (*that map all isomorphic graphs to the same output value*) up to arbitrary precision, it can thus (approximately) solve the graph isomorphism problem. This universality property on graph isomer is also widely studied in previous universal GNNs [1,2]. 

Therefore, due to the universality of UBoN network, **even if the two isomorphic graphs have different EASE representations, they can also be assigned with the same output in UBoN and then be judged as isomorphic graphs**. 

As this concern seems to be your main reason for a strong reject for this work and we have shown that it is a misunderstanding of our method, we believe the current evaluation is quite unfair. We hope you could re-evaluate this work by looking into the details of the paper. If you would insist that our theory is wrong, please **point out any specific errors that we made in our proof**. We thanks for your time and look forward to an engaged discussion.

References:

[1] Kanatsoulis, C. I., & Ribeiro, A. (2022). Graph Neural Networks Are More Powerful Than we Think. *arXiv preprint arXiv:2205.09801*.

[2] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. *arXiv preprint arXiv:2010.01179*.

---

**Q2**. [The method implies two claims:] different sets imply that the graphs are non-isomorphic and identical sets imply isomorphism. In fact, only the first of the two statements is trivially true. Graphs having identical eigenvalues and eigenvectors does not imply they are isomorphic. In fact, were this to be the case, then graph isomorphism would not be a challenging problem.

**A2**. The explanation in **A1** above has explained that UBoN does not assume unique EASE representations for isomorphic graphs. Below, we further discuss the two claims you mentioned.

- The first claim that “different sets imply that the graphs are non-isomorphic” is **not** **true**, as even **the same graph can have two different EASE representations when we add apply a random rotation on the eigenvectors**. Nevertheless, as UBoN is permutation-invariant, the two different EASE representations will have **the same output in UBoN, which suggests that they are isomorphic graphs.**
- The second claim “identical sets imply isomorphism” is in fact **true**. Because we can reconstruct the entire adjacency matrix $\mathbf{A}$ from EASE ($\mathbf{A}=\mathbf{PP}^\mathrm{T}, \mathbf{P}=\mathbf{U\Lambda}^{1/2}$), the same sets imply that they have the same adjacency matrices, and of course they are isomorphism.

Then, we try to answer the final question: **does the universality of UBoN made graph isomorphism not a challenging problem?**

The fact that UBoN can *approximate* an NP-hard solution, does ***not*** mean that UBoN solves the NP problem in polynomial time. In fact, ***MLPs are known to be universal approximators to arbitrary functions [1], but it does mean that they can already solve any NP-hard problems in polynomial time***. The key here is that either of these NN approximators (including UBoN) do not offer exact solutions but only approximations. Therefore, finding the exact solution to graph isomorphism remains a challenging problem.

References:

[1] Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. *Neural networks*, *2*(5), 359-366.

---

We thank Reviewer sg6T for your constructive comments. Please let us know if you have additional questions. We are happy to address them during the rebuttal stage.

From: ICLR.cc/2023/Conference/Paper2733/Authors
title: Response to Reviewer 8PYU
comment: We thank Reviewer 8PYU for the comments on our theoretical results, though there might be some misunderstandings of our theory, as elaborated below.

---

**Q1.** Theorems 3 and 4 imply that there is a simple graph canonization based on the spectral decomposition of the Laplacian. This implies that graph canonization can be done in polynomial time

**A1.** We are afraid that you misunderstood our theory. The proposed decomposition (EASE) is ***not*** a graph canonization, and we do ***not*** require this property to prove our universality results. Similar to many universality results on graphs, such as [1,2], **we prove that the proposed UBoN can *approximate* any permutation-invariant function on graphs, up to arbitrary precision**. The fact that it can approximate an NP-hard solution, does ***not*** mean that our UBoN itself solves the NP problem in polynomial time. In fact, ***MLPs are known to be universal approximators to arbitrary functions [3], but it does mean that they can solve any NP-hard problems in polynomial time***. The key here is that either these NN approximators do not offer exact solutions but only approximations. Therefore, **your conjecture that we provide a poly-time graph canonization in this work is not true**.

References:

[1] Kanatsoulis, C. I., & Ribeiro, A. (2022). Graph Neural Networks Are More Powerful Than we Think. *arXiv preprint arXiv:2205.09801*.

[2] Abboud, R., Ceylan, I. I., Grohe, M., & Lukasiewicz, T. (2020). The surprising power of graph neural networks with random node initialization. *arXiv preprint arXiv:2010.01179*.

[3] Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. *Neural networks*, *2*(5), 359-366.

---

Given the wrong conjecture above is the only reason that you give for a strong reject of this work, we believe this evaluation is extremely unfair. It would be more objective if you could look into the details of this paper for a re-evaluation. If you would insist that our theory is wrong, please **point out any specific errors that we made in our proof**. We are looking forward to your reply.

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_8PYU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a new GNN based on bag of nodes where each node is associated with a spectral embedding and claim that this GNN is universal, i.e. can approximate any graph function.
strength_and_weaknesses: Theorems 3 and 4 imply that there is a simple graph canonization based on the spectral decomposition of the Laplacian. This implies that graph canonization can be done in polynomial time. This would be a major breakthrough. Unfortunately, I think that the proofs of Theorem 3 and 4 are not correct.
clarity,_quality,_novelty_and_reproducibility: The paper is well written.
summary_of_the_review: There is a major flaw in the main theoretical theorems of this paper.
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_gwSj
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper explores using MLP directly to encode a graph with the help of laplacian embedding and eigenvalues. It shows that it is stronger than 1-WL and empirically verified that. The paper also claim universality however this is kind of trivial and is not the universality the community cares. The paper also try to extend the method to sign invariant and basis invariant using the result of another paper. 
strength_and_weaknesses: **Strength**: The writing is clear. The presentation is easy to follow. The experiment demonstrates the improvement over message passing based GNN. The presented model is very simple, and it is interesting to see the performance comparsion to message passing based GNN.

**Weakness**:
1. The universality proved in Theorem 1 is trivial. The true universality we care is the model that obeying all symmetries of the input. Taking an example, one can call MLP over the original adjacency universal by ignoring the permutation invariance. Back to the universality mentioned in the paper, the model DOES not obeying the sign invariant and basis invariant in Theorem 1. And again, when taking sign and basis invariant into consideration, the universality is formally proved in Derek's paper, and it is not the contribution of this paper. So Theorem 1, Theorem 3, and Theorem 4, are not novel to the community. The presented model is very simple and it is good, however it is not good to misguide the community with calling it universal (it is kind of hype).

2. I checked all theorems and proofs, and I don't find much contribution in theory. All mentioned theorems have appeared in other papers. Derek's paper and [22 ICML, How Powerful are Spectral Graph Neural Networks].

3. The author mentioned that adding eigenvalue is another contribution over methods with only laplacian embedding. However it is not true. In fact with adjacency matrix A available, one only need to keep eigenvector U to be used, as the eigenvalue information can be recovered from AU which is a single message passing step (as used in Derek's paper).

4. The experimental design is really questionable. The community has agreed that these small TUDatasets are not meaning in evaluating expressivity, as these datasets are too small and too easy. If the author really want to show the power, one should use the larger dataset in OGB and Benckmarking GNN like ZINC. 

 
clarity,_quality,_novelty_and_reproducibility: Clarity is good. Quality is limited by the novelty. The paper doesn't have many new contribution, except designing a simple and widely used baseline. 
summary_of_the_review: To summarize, I think the paper's contribution is too limited. And the method designed is not useful in real-world, neither in theory. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_ok4x
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents eigenvalue-aware spectral embedding (EASE) — a positional encoding aimed at improving the expressive power of graph learning models. They prove that a MLP-based Bag-of-Nodes model together with EASE can achieve universal expressive power and preserve permutation invariances. 
strength_and_weaknesses: Basically the paper adds the e-values to spectral positional encodings and applies a MLP. The results are not significant improvements but nice to see performance is still competitive. However, the strength lies in the theory as well as in the simplicity of the approach. I think the weakness lies in comparisons to related work. I think they should cite and discuss "Universal Function Approximation on Graphs", (2020), Brüel-Gabrielsson. What about other work that speeds up graph learning? What about the computational cost of spectral decomposition on large graphs? I would love to see more discussion on this. 
clarity,_quality,_novelty_and_reproducibility: Very clear, high quality paper. Code should ideally be released for reproducibility. The novelty is limited, but still significant. 

summary_of_the_review: Overall great paper. It's clear and simple in a good way. I want more comparisons and discussion with related work. 

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_MeG5
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors propose UBoN, a model for dealing with graph-level learning tasks. First, the proposed method performs the eigenvalue decomposition of the normalized adjacency matrix. Then, it concatenates the attributes of the nodes of the graph with a combination of the eigenvector and the eigenvalue matrix. Next, the emerging matrix is fed to a DeepSets model which produces the output. The authors show that UBoN can achieve universal expressiveness. The model is evaluated on a synthetic dataset and it is shown that it indeed more expressive than 1-WL. It is also evaluated on real-world graph classification datasets where it achieves performance competitive to message passing GNNs.
strength_and_weaknesses: Strengths:

- The idea is relatively simple which a good thing in my opinion. UBoN treats each graph as a set of nodes where the nodes are represented by their attributes and by structural features extracted by the eigenvalue decomposition of the normalized adjacency matrix.

- The proposed UBoN model seems to be empirically strong. On EXP it almost achieves perfect accuracy, while it also outperforms the baselines on 4 out of the 5 graph classification datasets, while some of those baselines are very expressive.

- The theoretical results provided in the paper are interesting since they show that structural features can give rise to powerful models.

Weaknesses:

- My main concern with this paper is that the proposed model is only evaluated on relatively small and old datasets. The largest of those datasets contains less that 1200 samples. I would suggest the authors experiment with more recent larger datasets such as the OGB graph property prediction datasets which contain tens or even hundreds of thousands of samples.

- This point is related to the previous one. Since the proposed method needs to solve the eigenvalue decomposition problem, I expect its complexity to be high in case of large graphs (complexity is $\mathcal{O}(n^3)$ in practice). For the considered datasets, the average number of nodes is at most equal to $41.22$. I thus feel that the preprocessing time reported in Table 3 is a bit misleading. This time will be much higher in case of a dataset that contains large graphs such as DD, REDDIT-BINARY and REDDIT-MULTI. Could approximation schemes for eigenvalue decomposition be employed? What would be the impact of those schemes on the model's performance?

- Why does SE use the Laplacian matrix while EASE the normalized adjacency matrix? If I am not wrong, in the case of EASE, the same theoretical results could also be obtained if the Laplacian matrix was employed.

- In definition 1, what is the codomain of function $f$? It would be nice if the authors provided some examples of permutation invariant functions. What is a continuous graph invariant function? Does a function that maps each unique weighted graph to a different vector (and can thus solve graph isomorphism) belong to this set of functions?

- An interesting question that is not answered in the paper is how the UBoN model compares against 1-WL and its higher order variants, in terms of their ability to distinguish non-isomorphic graphs. It has been shown that the spectra of most matrices lie between 1-WL and 2-WL [1]. I would like the authors to comment on that since the expressive power of UBoN is mainly empirically evaluated.

- I would not expect GCN to be in the list of baselines on the EXP dataset since GCN is known to be less powerful than 1-WL [2]. I suggest the authors replace GCN with GIN which can achieve the same expressive power as 1-WL. Moreover, I would expect some baselines that are more expressive than 1-WL such as the ones of Table 2.

- Since UBoN-sign and UBoN-basis are more powerful than UBoN, I would expect them to outperform UBoN on all datasets. However, this is not the case as illustrated in Table 5. How can those results be explained? Does the same pattern hold for training performance as well?

- The paper does not cite some related works. The proposed model is similar to that of [3] where the matrix of node attributes is concatenated with a matrix that contains structural node features and the resulting matrix is fed into DeepSets. However, the structural features employed in this paper (from the eigenvalue decomposition) make the proposed method sufficiently original. Furthermore, the authors provide some theoretical results with regards to UBoN's expressive power.

[1] Rattan, G., & Seppelt, T., "Weisfeiler-Leman and Graph Spectra". arXiv preprint arXiv:2103.02972, 2021.\
[2] Xu, K., Hu, W., Leskovec, J., & Jegelka, S., "How powerful are graph neural networks?". In International Conference on Learning Representations (ICLR), 2019.\
[3] Dasoulas, G., Nikolentzos, G., Seaman, K., Virmaux, A., & Vazirgiannis, M., "Ego-based entropy measures for structural representations on graphs". In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3210-3214, 2021.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to read. The authors also provide enough details for the reader to reproduce the reported results. The quality of the paper is good, though I would like to see more experimental results. Although the novelty of the proposed method is not very strong, the authors provide some theoretical results which increase the paper's originality.
summary_of_the_review: Overall, this is an interesting paper which proposes a simple approach for dealing with graph-level machine learning tasks. The quality of the paper is good, and some novel theoretical results are presented. However, I have some concerns mainly with regards to the validation of UBoN. The complexity of the method might be prohibitive when applied to large graphs, while it is not clear whether the model is effective on recent large-scale graph property prediction datasets.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2733/Reviewer_sg6T
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper proposes universal bags of nodes (UBoN), a simple alternative to message passing graph neural networks that avoids the inefficiencies and 1-WL expressiveness limitations of message passing.  Specifically, UBoN consists of the following simple steps: 1) computing eigenvectors and eigenvalues of the input graph and appending them to the input node features, 2) passing node features through a node multi-layer perceptron (MLP), 3) aggregating all node representations through a readout function such as sum, and 4) processing the readout output through another graph-level MLP. The paper claims that, by using eigenvalue-aware spectral encoding (EASE), the UBoN model is universal for permutation-invariant functions over graphs. The paper then studies implications from using a subset of the eigenvalue components, and proposes solutions to introduce sign and basis invariance at the eigenvector level. Finally, the paper conducts an empirical analysis using the EXP dataset as well as other real-world datasets, showing that 1) UBoN can perform competitively, 2) is more expressive than 1-WL, and 3) also performs well with eigenvector selection and basis/sign invariance. 
strength_and_weaknesses: # Strengths: 
- The model is simple, efficient and expressive (However, please see weaknesses below)
- The empirical performance given the small number of parameters is encouraging.

# Weaknesses: 
- A fundamental weakness that undermines the paper is the claimed universality, which is incomplete, and most likely incorrect. Theorem 1 claims that UBoN is universal over graph structures by simply adding EASE encodings, that is, eigenvectors multiplied by their corresponding eigenvalues. The proof does not address the uniqueness of the resulting representations with respect to the graph isomorphism problem, but simply discusses how, by using the resulting features as a set, one can build a universal and invariant function approximation over sets, as in DeepSets. This, however, implicitly assumes, without proof, that the set representations from EASE correspond directly with graph isomorphism, i.e., different sets imply that the graphs are non-isomorphic and identical sets imply isomorphism. In fact, only the first of the two statements is trivially true. Graphs having identical eigenvalues and eigenvectors does not imply they are isomorphic. In fact, were this to be the case, then graph isomorphism would not be a challenging problem. Having identical eigenvalues is known as isospectrality, and is a necessary, but not sufficient condition for computing general graph isomorphism. Therefore, I strongly encourage the authors to complete the proof with the necessary correspondence (if correct).

-Alternatively, if this is not possible, I suggest the following course of action: The expressiveness of this approach must be studied anew in light of the power of spectral graph isomorphism checking. Then, the work should be wholly rewritten and presented in light of that power. 
clarity,_quality,_novelty_and_reproducibility: # Clarity: 
- The paper contains a fundamental flaw in the proof of the main theorem, but is otherwise easy to follow.
# Novelty: 
- Idea is of limited novelty, but this is overshadowed by the incorrect expressiveness claim. 
# Quality: 
- Unfortunately, poor.
# Reproducibility: 
- No concerns
summary_of_the_review: Unfortunately, the paper makes an unsubstantiated universality claim for its proposed UBoN method, as it implicitly claims that adding EASE encodings is sufficient for universality, when it is known that isospectrality is not sufficient to conclude isomorphism in the general case, implying that some non-isomorphic graphs may be mapped to the same outputs by UBoN. This is a fundamental flaw in a key argument made in the paper, and thus there is no option but to reject this paper at the moment. 
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject