From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper proposes a new WL-like expressive power hierarchy for graph isomorphism testing, and derives a new GNN architecture based on these principles. While the reviewers appreciated the new hierarchy, they questioned the novelty wrt related works on k-hop CNNs. Furthermore, the experimental results were not very impressive. The AC believes that the paper presents a direction somewhat different from the majority of GNN literature that might be interesting and worth being presented at the conference. 
justification_for_why_not_higher_score: not sufficiently novel
justification_for_why_not_lower_score: interesting idea
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author Responses
comment: We appreciate the reviewer's comments and follow-up question. For this work, we consider the major contributions are on the theoretical developments (i.e., a new graph isomorphism algorithmic hierarchy and a theory about the equivalence between connected and general isomorphism graphs), and the proposed G3N model is one instance for illustrating the key ideas of these theoretical developments. For this reason, we conducted experiments on ZINC and MolHIV datasets to show that the G3N model performs comparably with some other GNNs without an intention to claim our work as SOTA. 

Following the reviewers' comments, we checked the experimental results of existing works on ZINC and refined our result in the updated version but missed MolHIV. We admit our negligence and would like to thank you for pointing this out. We have hereby updated the results for MolHIV below. We consider the result is comparable with other GNNs but not SOTA. According to the code of DEEP LRP, GSN, and CIN, these models reported performance based on pre-selected subgraph patterns, same as for ZINC. 

|Dataset| MolHIV (test) | MolHIV (val) |
|:----:       |:----:         |:----:  |  
|GCN  | 0.7606 $\pm$ 0.0097 | 0.8204 $\pm$ 0.0141 |
|GIN  | 0.7558 $\pm$ 0.0140 | 0.8232 $\pm$ 0.0090 |
|GraphSNN  | 0.7851 $\pm$ 0.0170 | 0.8359 $\pm$ 0.0096 |
|PNA | 0.7905 $\pm$ 0.0132 | -  |
|DGN | 0.7970 $\pm$ 0.0097 | - | 
|DEEP LRP*  | 0.7687 $\pm$ 0.0180 | 0.8131 $\pm$ 0.0088|
|GSN*  | 0.7799 $\pm$ 0.0100 | **0.8658 $\pm$ 0.0084**|
|CIN* |**0.8094 $\pm$ 0.0057** | - |
|G3N-(2,3) | 0.7900 $\pm$ 0.0134 | 0.8359 $\pm$ 0.0061|

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_rk5R
title: Follow-Up Question
comment: I thank the authors for their effort and response.

- Regarding the relation to K-hop GNNs, the added discussion resolves my concerns, and I think that together with the other reviewers' comments, the paper is better placed in the literature now.

- Regarding experimental results - I wonder how the authors chose which results to list on the Molhiv dataset table. In the [OGB benchmark](https://ogb.stanford.edu/docs/leader_graphprop/#ogbg-molhiv) leaderboard, some works outperform the current method, e.g., CIN which is listed in the ZINC results table, although the authors place their work as SOTA. 

I generally lean toward raising my score, mainly since I appreciate the paper's theoretical contribution in constructing a strict and flexible hierarchy. Nevertheless, I want to be sure that the experimental results are not misrepresented.

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author Responses
comment: Thanks for the suggestion. 

We would be very happy to mark the revised part. Indeed, we were wondering whether we would be allowed to do this when submitting the revised version. Now it seems that we cannot upload a PDF any more on Openreview. Please let us know if there is a way we can send it through. Thank you very much.

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_QGtL
title: some suggestions
comment: Can you mark the revised part in blue? Just for a better checking.

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to Reviewer YCJN 2/2
comment: (continue with Author responses to Reviewer YCJN 1/2)

> Q7: *In the actual model, authors inject isomorphism types into node features (which is different than operating over isomorphism types) ...*

The model encodes isomorphism types at the node level without pre-selecting any application-dependent subgraph patterns. By applying a sum aggregation on node features in a subgraph, this encoding allows to further capture the cardinality of nodes in each subgraph. Then, the model performs a linear transformation w.r.t positional types to learn the importance of different isomorphism types at different positions within the neighbourhood of nodes. 

> Q8: *...The theorem statements are not very specific: It should be stated what is fixed, and how the other parameter ranges. It is also unclear whether the statements are over the class of all graphs, or just over graph pairs. The WL hierarchy is an infinite hierarchy: Is the same true here?*

Thanks for the suggestions. We revised the theorem statements to improve the clarity. The theorem statements are over the class of all graphs. To show the strictness of the proposed hierarchy, graph pairs as counterexamples suffice. 

Similar to k-WL, our $\mathscr{N}(t,d)$-WL is also an infinite hierarchy in which expressivity increases when $t$ and $d$ increase. For specific graphs,  when $t$ and $d$ equal to the diameters of graphs, the upper bound expressivity is reached. In such cases, whether the $\mathscr{N}(t,d)$-WL algorithm can distinguish two non-isormorphic graphs is reduced to the classical graph isomorphism (GI) problem. This is because, given a graph $G$, each $d$-hop neighbourhood subgraph is the same as $G$ when $d \geq dia(G)$; induced subgraphs of $t$ vertices are also the same as $G$ when $t\geq dia(G)$, where $dia(G)$ refers to the diameter of $G$. 


> Q9: *WL algorithms can be used to analyse node-level, or link-level properties, and the same applies here, but it is unclear to me ...*

Thanks for the suggestion. We agree with the reviewer that the same applies to our $\mathscr{N}(t,d)$-WL algorithms that they can be used to analyse node-level or link-level properties. This direction can be further extended to analysing subgraph-level properties and building connections to expressivity of GNN models.

For the $k$-WL algorithms, despite fruitful connections of these algorithms to logic and descriptive complexity theory, as discussed in [2, 4], a complete description of all subgraph patterns whose counts and occurrence are $k$-WL invariant is only available for $k=1$. There is no clear understanding for higher-order $k$-WL invariants yet [4]. Even for the case $k=3$, such a characterisation is an open problem left to be explored [2]. In fact, traditionally, the $k$-WL algorithm is commonly used as a combinatorial tool in graph isomorphism testing and as noted in [3] little work has been done to study the applicability of $k$-WL to recognition of graph properties. Thus, for both $k$-WL and $\mathscr{N}(t,d)$-WL algorithms, this is an interesting direction for further research. 

We added a discussion on the above direction into the updated version (see Section 6).


> Q10: *The runtime of these class of algorithms is also prohibitive, and using a small $t$ seems the only way around the issue...*


Different from the global nature of high dimensional $k$-WL, $\mathscr{N}$-WL is local, independent from the size $t$ of induced subgraphs. This locality is controlled by $d$, which is generally much smaller than the diameter of a graph. Thus, $\mathscr{N}$-WL is much more efficient than $k$-WL when performing on graphs with diameters that are greater than $d$ values, i.e., $dia(G)>>d$, where $dia(G)$ refers to the diameter of a graph $G$.

We added an additional experiment to empirically analyse and compare the time complexity of our $\mathscr{N}$-WL algorithm and the classical $k$-WL algorithm on four datasets ZINC, MolHIV, NCI1, and IMDB-B. The results and discussion are included in Appendix E.7 and Figures 10-13. 

References:

* [1] Feng et. al. How Powerful are K-hop Message Passing Graph Neural Networks (NeurIPS 2022)

* [2] Arvind et al. On weisfeiler-leman invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences, 2020

* [3] Fuhlbrück et al. The Weisfeiler-Leman algorithm and recognition of graph properties, Theoretical Computer Science, 2021. 

* [4] Sandra Kiefer, Power and Limits of the Weisfeiler-Leman Algorithm, 2020

* [5] Morris et al. SpeqNets: Sparsity-aware Permutation-equivariant Graph Networks, ICML, 2022

* [6] W.L. Kocay. Some New Methods in Reconstruction Theory, Combinatorial Mathematics IX, 1982

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to Reviewer YCJN 1/2
comment: We thank the reviewer for the constructive feedback. We respond to your comments below.

> Q1: *The paper is hard to follow: ... the proposed class of algorithms enumerate all subgraphs of order $t$ within a $d$-hop neighbourhood, and this should be clearly stated.*

We revised the introduction to improve the clarity of the paper. We also added additional experiments and discussions to show the importance of $t$. See more details in our response to Q4 below.

> Q2: *The paper is not very well-placed in the literature. Authors have an extensive related work (a bulk of it is in the appendix, which is not a great practice in my opinion), but there are no concrete connections to other well-known concepts.*

The reason why we include an extensive related work in Appendix B rather than in the main paper is due to space constraints. There are a number of active research lines studying the expressivity of GNNs. Including all of them into the main paper would be difficult and would require to remove other important parts out for space.

We added a new section in Appendix B.2 to discuss the connections with the $k$-WL algorithms and its variants [5]. We also added a discussion on $K$-hop GNNs [1] in Appendix B.1.

> Q3: *...This paper builds on this idea to use positional info in the induced subgraphs, but the connections are not explored/explained.*

We added a discussion to explain the connections in the updated version. See more details in "Positional types for induced subgraphs" under "Author responses to common questions".

> Q4: *It is not very clear whether one gains more by using the induced subgraphs, as compared to just using multi-hop aggregation*

The size $t$ of subgraphs also plays an important role in expressivity, both theoretically and in practice. To show the importance of $t$, we added the following into the updated version:

* We added a table in Appendix A which contains the detailed information for indistinguishable pairs of simple graphs of eight vertices under varying configurations of $t$ and $d$ values. It shows that, when $t=1$, increasing $d$ from $2$ to $6$ cannot help distinguish more pairs. To distinguish all pairs of simple graphs of eight vertices, $\mathscr{N}(1,1)$-WL needs to have at least $t\geq 3$ and $d\geq 2$. Thus, both $t$ and $d$ are important for the expressivity of models.

* We also added an additional ablation study on an OGB dataset MolTOX21. The results below show that increasing $t$ consistently improve the performance (Table 9, Appendix E.4).

|\#Layers     |               |3              | 4            | 5                  |
|:----:       |:----:         |:----:         |:----:           |:----:           |
|$d=1$        |$t=1$          |0.7363 $\pm$ 0.006 |0.7438 $\pm$ 0.008 |0.7398 $\pm$ 0.009|
|             |$t=2$          |0.7402 $\pm$ 0.003 |0.7565 $\pm$ 0.006 |0.7482 $\pm$ 0.006|
|             |$t=3$          |**0.7487 $\pm$ 0.005**   |**0.7649 $\pm$ 0.007**    |**0.7531 $\pm$ 0.005** |
|$d=2$        |$t=1$          |0.7365 $\pm$ 0.013  |0.7400 $\pm$ 0.009 |0.7553 $\pm$ 0.011|
|             |$t=2$          |0.7426 $\pm$ 0.009   |0.7514 $\pm$ 0.008 |0.7603 $\pm$ 0.009|
|             |$t=3$          |**0.7496 $\pm$ 0.019**   |**0.7636 $\pm$ 0.009**   | **0.7657 $\pm$ 0.006** |
|$d=3$        |$t=1$          |0.7443 $\pm$ 0.017  |0.7511 $\pm$ 0.007 |0.7494 $\pm$ 0.001|
|             |$t=2$          |0.7531 $\pm$ 0.005  |0.7593 $\pm$ 0.018  |0.7577 $\pm$ 0.008|
|             |$t=3$     |**0.7586 $\pm$ 0.006**    | **0.7695 $\pm$ 0.007**  |**0.7608 $\pm$ 0.007** |


> Q5: *...There are many well-known connections between k-WL hierarchy and logics, which precisely inform us about the formal properties of these algorithms*

We agree that studying the formal properties of the $\mathscr{N}$-WL algorithms would need to explore the connections between the $\mathscr{N}$-WL hierarchy and logics. However, due to the differences in algorithmic designs (see Appendix B.2), comparing the expressive power of $k$-WL and $\mathscr{N}$-WL directly is non-trivial. A possible way to go forward is to characterize $\mathscr{N}$-WL in terms of a guarded fragment of first-order logic and then compare the expressive power of these two methods in terms of their logical characterization. This is an interesting direction to explore in future.

> Q6: *There are many local variants of k-WL algorithms which are proposed with similar intuition ... The connections need to be discussed in depth, clarifying whether the proposed model has advantages over local WL algorithms.*

We added a new section in Appendix B.2 to compare with k-WL and its variants. It is worthy to note that $\mathscr{N}^c$-WL can improve efficiency without compromising expressivity (Theorem 3.7). We establish this theorem upon Kocay's work on the Algebra of Subgraphs, which deals with relations holding among subgraphs of a graph [6]. We believe that this theorem itself is a significant contribution.

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to Reviewer e9AE
comment: Thank you for the reviews and suggestions. We respond to your comments below.

> Q1: *there is a parameter $d$ in $\mathscr{N}$-WL. But there is again $\forall d\geq 1$ as the condition of ''... $f_{pos}(S_i)\neq f_{pos}(S_j)$''. Is the second $d$ not related to the first $d$?*

They are related. The parameter $d$ in $S_{(u,t,d+1)}$ and $S_{(u,t,d)}$ also relates to the parameter $d$ in $\mathscr{N}(t,d)$-WL. Specifically, $S_{(u,t,d)}$ refers to the set  of all induced subgraphs of order $t$ within the $d$-hop neighbourhood of a node $u$ and $S_{(u,t,d+1)}$ refers to the set of all induced subgraphs of order $t$ within the $d{+}1$-hop neighbourhood of a node $u$.  

Intuitively, the condition $\forall d\geq 1$ and the statement of $f_{pos}(S_i)\neq f_{pos}(S_j)$ in Equation 2 require that induced subgraphs appearing in the $d{+}1$-hop neighbourhood of a node $u$ but not in the $d$-hop neighbourhood should have different positional types from induced subgraphs appearing in its $d$-hop neighbourhood. This applies for each $d\geq 1$.
 
> Q2: *Is $f_{pos}$ unique or universal for any target node $u$?... if the answer to the above question is ''not related'', then it seems $f_{pos}$ is coloring everything distinctly because the target node can be arbitrary.*

There is no requirement on the uniqueness of $f_{pos}$ with respect to nodes. However, $f_{pos}$ can only assign positional types to induced subgraphs in a way that is permutation invariant and also must satisfy the statement described under strong hierarchy (Equation 2). The definition of $f_{pos}$ in our work is intended to characterise two general conditions for positional types that are required at the minimum to establish the proposed hierarchy: (1) permutation invariant; (2) satisfying the condition described under strong hierarchy (Equation 2).


> Q3: *how is $J$ obtained in the model in Eq(4)?*

Let $S$ be an induced subgraph. $f_{pos}(S)$ is defined to be the set of shortest-path distances from the nodes in $S$ to the current node $u$, i.e., $f_{pos}(S)=\{\rho(u,v)|v\in V(S)\}$, where $\rho$ refers to the shortest-path distance between node $u$ and node $v$. For example, if $S$ has three nodes $V(S)=\{v_1,v_2,v_3\}$ with $\rho(u,v_1)=1$, $\rho(u,v_2)=1$ and $\rho(u,v_3)=2$, then the positional type $f_{pos}(S)=\{1,1,2\}$ for the induced subgraph $S$ in this case. We added the implementation details in Appendix D.

See more details in "Positional types for induced subgraphs" under "Author responses to common questions".


> Q4: *The compared baselines in the ZINC experiment are not enough, because it only includes PPGN as the GNN beyond 1-WL. For instance, GSN is missing, which is cited and discussed by the authors for many times. It would be great to see the comparison with more expressive GNNs to show the sufficiency and efficiency of the proposed model.*

Thanks for the suggestions. We added three baselines: DEEP LRP [1], GSN [2], and CIN [3] in the ZINC experiments and updated the experimental results. 

See a detailed discussion in "ZINC experiments'' under "Author responses to common questions''.

References:

* [1] Chen et al. Can graph neural networks count substructures? NeurIPS, 2020

* [2] Bouritsas et al. Improving graph neural network expressivity via subgraph isomorphism counting, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

* [3] Bodnar et al. Weisfeiler and Lehman go cellular: CW networks, NeurIPS, 2021.

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to Reviewer QGtL
comment: Thanks for your insightful comments and acknowledging our theoretical contributions. We respond to your comments below.

> Q1: *Although the proposed hierarchy has increased expressivity with $d$ and $t$. It is not clear the upper bound expressivity of the method. And the connect to $k$-WL is not clear also.*    
    
 This is a great point. Thanks for pointing this out.  We added the following discussion in the updated version (see Section 3.1). 

With increasing $d$ and $t$ values, for specific graphs, the upper bound expressivity would be reached when both $t$ and $d$ equal to the diameter of a graph. In such cases, whether the $\mathscr{N}(t,d)$-WL algorithm can distinguish two non-isormorphic graphs is reduced to the classical graph isomorphism (GI) problem. This is because, given a graph $G$, each $d$-hop neighbourhood subgraph is the same as $G$ when $d \geq dia(G)$; induced subgraphs of $t$ vertices are also the same as $G$ when $t\geq dia(G)$, where $dia(G)$ refers to the diameter of a graph $G$. 
    
   In terms of connecting to $k$-WL, as shown by Cai, Fürer, and Immerman in their seminal paper [1],  for any $k\in \mathbb{N}$, there exists a pair of non-isomorphic graphs that cannot be distinguished by the $k$-WL algorithm. Similarly, in our work, we show the constructions (presented in Appendix C.1 and C.2) that, for any $t\in \mathbb{N}$ and $d\in \mathbb{N}$, there exists a pair of non-isomorphic graphs that cannot be distinguished by $\mathscr{N}(t,d)$-WL. 
    
> Q2: *The proof of theorem 3.7 is questionable.*
    
We develop this theorem based on Kocay's work on the Algebra of Subgraphs, which deals with relations holding among subgraphs of a graph [2].
    
For clarity, we updated the proof of Theorem 3.7 and added two lemmas (i.e., Lemmas 4-5) in Appendix C.3 along with the proof details. In addition to these, we further added an example (i.e., Example 1) in Appendix C.3 to illustrate the key ideas of the proof.

> Q3: *The runtime analysis shows that the method is faster over sparse graph, but for denser graph this can be very slow.*

We agree with the reviewer. Nonetheless, this is a common issue for GNN methods and sampling methods can be used to address this issue.

> Q4: *When $d$ is large, the computational cost can be closer to k-WL (its set version) for extracting all t-tuples. The main difference to $k$-WL will be that the intermediate representation is size $n$ instead of size $n^t$.*

Since $\mathscr{N}$-WL considers local neighbourhood subgraphs, when $d$ is large, local neighbourhood subgraphs become closer to the whole graph and,  accordingly, the computational cost of $\mathscr{N}$-WL is closer to $k$-WL. However, the gap of computational cost is significant between $\mathscr{N}$-WL  and $k$-WL when the diameters of graphs are relatively larger than $d$.

We added an experiment to empirically analyse the computational cost of $\mathscr{N}$-WL, in comparison with $k$-WL, in terms of different $d$ and $t$ values. The results show that $\mathscr{N}$-WL is much more efficient than $k$-WL when performing on graphs with diameters that are greater than $d$. The performance gap between $\mathscr{N}$-WL and $k$-WL is reduced when increasing $d$ or graphs have small diameters.

The results and a detailed discussion are included in Appendix E.7. 


> Q5: *The experimental result over real-world datasets is not very good, this may suggests the expressivity upper bound of this method is limited.*
    
We added the baselines and updated the experimental results for ZINC. See more details in "ZINC Experiments'" under "Author responses to common questions''.

> Q6: *...The paper [Zhao et al. NeurIPS 2022] improved k-WL by using ≤k sets and studied its expressivity theoretically ... I suggest the author to have a check.*

We added a new section in Appendix B.2 to compare the $k$-WL algorithms and its variants, including the work by [3,4,5]. Generally, all these works extend the $k$-WL algorithms by either reducing the number of $k$-tuples to be coloured (e.g., restricting to only a subset of $k$-tuples  satisfying the condition on the minimal number of connected components). The work in [3] restricts this to sets with at most $k$ nodes and at most $c$ connected components. Our work in this paper is designed in a way that is quite different from these works. See a detailed discussion and comparison in Appendix B.2.


References:
    
   * [1] Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 1992
    
   * [2] W.L. Kocay. Some New Methods in Reconstruction Theory, Combinatorial Mathematics IX, 1982
   
   * [3] Zhao et al. A Practical, Progressively-Expressive GNN. NeurIPS, 2022.
   
   * [4] Morris et al. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings, NeurIPS, 2020.
   
   * [5] Morris et al. SpeqNets: Sparsity-aware Permutation-equivariant Graph Networks, ICML, 2022

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to Reviewer rk5R
comment: Thanks for your constructive comments. We respond to your comments below.

> Q1:  *Relation to K-hop GNNs -[1] analyses the expressive power of K-hop GNNs which basically, similar to this work, aggregate information from the k-hop neighborhood of each node and use this information to update the node's color at each iteration...I find the idea in KP-GNNs very similar to the addition of the positional types in the current work.*

The aim of our work is not to propose a $K$-hop GNN. Instead, our work aims to provide a comprehensive framework for characterising the expressive power of different GNNs, including $K$-hop GNNs. For example, standard $K$-hop message-passing GNNs with the shortest path distance kernel [1] can be characterized as our $\mathscr{N}(1,d)$-WL with $d=K$, where each node has its shortest-path distance to the target node as a positional type. KP-GNN is an instance of $\mathscr{N}(t,d)$-WL in our work with $d=K$, $t$ referring to the size of peripheral subgraphs, and isomorphism types corresponding to subgraph patterns used by KP-GNN.

We added a discussion on connections to K-hop GNNs in Appendix B.1. Also, see more details in "Connections to $K$-hop GNNs" under "Author responses to common questions''.

> Q2: *Experimental Results -The experimental results, as in many GNN papers, justify the theoretical statements but do not exhibit any significant improvement in real-life datasets. Furthermore, I find the comparison on the ZINC dataset rather lacking (see )*

We added the baselines and updated the experimental results for ZINC. We noticed that the baseline results in Table 5 of the paper [1] are different from the results provided in the original papers. For example, GSN and PNA have the original results 0.163 $\pm$ 0.004 and 0.188 $\pm$ 0.004, respectively. For a fair comparison, in our experiment, we consider these results provided in the original papers.

See a detailed discussion in "ZINC experiments'' under "Author responses to common questions''.


> Q3: *The name of the paper is quite general and does not reflect the properties of the proposed hierarchy....*

Thanks for the suggestion. We agree that the suggested name reflects our work more precisely. We have changed it in the updated version.

References:

* [1] Feng et. al. How Powerful are K-hop Message Passing Graph Neural Networks, NeurIPS, 2022.

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Summary of the revision
comment: We thank all the reviewers for the constructive comments, which help us to improve the quality of our work. Below, we summarise the main changes in the updated version:

* We revised the introduction to clarify our technical contributions and key ideas of our work (see Section 1).
* We added a discussion on indistinguishable pairs of graphs from Graph8c (see Appendix A and Table 5).
* We added a discussion on $K$-hop message-passing GNNs in the related work (see Appendix B.1).
* We added a new section to discuss the connections with k-WL and its variants (see Appendix B.2 and Table 6).
* We added a discussion on the upper bound expressivity of the proposed hierarchy (see Section 3.1).
* We added two lemmas (Lemma 4 and Lemma 5) for the proof for Theorem 3.7, and added a detailed example to illustrate the key ideas (Example 1) (see Appendix C.3).
* We added a discussion and implementation details for positional types of induced subgraphs (see Appendix D).
* We added and revised experiments on real-world datasets - TU datasets and ZINC dataset (see Section 5.2).
* We added baselines for experiments on ZINC dataset and MolHIV dataset (see Section 5.2).
* We moved the ablation study from Section 5.3 to Appendix E.4, and added an additional experiment on MolTOX21 dataset (see Appendix E.4 and Table 10).
* We added an experiment to empirically compare the computational cost of $\mathscr{N}$-WL and $k$-WL algorithms on real-world datasets (see Appendix E.7 and Figures 10-13).
* We added a discussion on future work (see Section 6).

From: ICLR.cc/2023/Conference/Paper4131/Authors
title: Author responses to common questions
comment: We thank all the reviewers for their valuable comments. Below, we first address some common questions. Then, we address the individual comments from each reviewer.

**Connections to $K$-hop GNNs**

The main idea of $k$-hop GNNs is to aggregate information from not only direct neighbours of a node but also neighbours within $K$ hops of a node. To further improve the expressive power, the authors of [1] proposed  KP-GNN, which leverages the information of peripheral subgraphs into $K$-hop message-passing GNNs.


Our work provides a comprehensive theoretical framework for characterising the expressive power of different $K$-hop message-passing GNNs. For example, standard $K$-hop message-passing GNNs with the shortest path distance kernel [1] can be characterized as our $\mathscr{N}(1,d)$-WL with $d=K$, where each node has its shortest-path distance to the target node as a positional type. KP-GNN is an instance of $\mathscr{N}(t,d)$-WL in our work with $d=K$, $t$ referring to the size of peripheral subgraphs, and isomorphism types corresponding to subgraph patterns used by KP-GNN.

Our main contributions in this work are not a $k$-hop GNN. Instead, we make two fundamental theoretical contributions: 

1. We develop a new class of graph isomorphism algorithms which exhibit a natural and strict hierarchy of expressivity for graph neural networks. This hierarchy is designed in a way different from $k$-WL, and the proof on the strictness of the hierarchy is non-trivial.

2. We establish a new theorem on the equivalence of experssivity between induced connected subgraphs and induced subgraphs in general which enables to leverage graph sparsity for improving efficiency but without compromising on expressivity.

We revised Section 1 to clarify our technical contributions and added discussions on $K$-hop GNNs /$k$-WL variants and their connections to our work in Appendix B.1 and Appendix B.2.  


**Positional types for induced subgraphs**

In our work, the definition of positional types for induced subgraphs (via the function $f_{pos}$) is intended to characterize two general conditions that are required at the minimum to establish the proposed hierarchy: (1) permutation invariant; (2) satisfying the condition described under strong hierarchy (Equation 2 in the updated version).

The shortest path distance kernel defined in K-hop GNNs [1] can be considered as an instance of our positional types, which is defined only for nodes (nodes correspond to induced subgraph of $t=1$ in our work). 

In general, there are many possible ways to instantiate our function $f_{pos}$ for positional types. Let $S$ be an induced subgraph. Below are two simple ways of defining $f_{pos}$:

 1. $f_{pos}(S)$ is defined to be the set of shortest-path distances between the nodes in $S$ to the current node $u$, i.e., $f_{pos}(S)=\{\rho(u,v)|v\in V(S)\}$, where $\rho$ refers to the shortest-path distance between node $u$ and node $v$. For example, if $S$ has three nodes $V(S)=\{v_1,v_2,v_3\}$ with $\rho(u,v_1)=1$, $\rho(u,v_2)=1$ and $\rho(u,v_3)=2$, then the positional type $f_{pos}(S)=\{1,1,2\}$ for the induced subgraph $S$ in this case.
2. $f_{pos}(S)$ is defined to be the maximum value of shortest-path distances between the nodes in $S$ and the target node $u$, i.e., $f_{pos}(S)=max(\{\rho(u,v)|v\in V(S)\})$. In this case,  the positional type for the induced subgraph $S$ depicted in Case (1) becomes $f_{pos}(S)=2$ because $max({1,1,2})=2$.

In our implementation, we use the first approach. We added the implementation details in Appendix D in the updated version.


**ZINC experiments**

We added three baselines: DEEP LRP [2], GSN [3], and CIN [4] in the ZINC experiment and also added an experiment for ZINC with edge features. It is worthy to note that

1. DEEP LRP, GSN and CIN pre-select application-dependent subgraph patterns and then inject the counts of these subgraphs into the message-passing scheme. Our model do not pre-select subgraph patterns. Instead, we consider all isomorphism types and let the model to learn useful subgraph patterns.

2. KP-GNN [1] and other baselines in Table 5 of [1] inject the edge structural features in the message passing scheme. In our previous ZINC results, we do not take edge features into account.

We updated the experimental results for ZINC dataset. Among the baselines which do not pre-select applicant-dependent subgraph patterns, our G3N performs best on ZINC both with edge features and without edge features.

References:

* [1] Feng et al. How Powerful are K-hop Message Passing Graph Neural Networks, NeurIPS, 2022.

* [2] Chen et al. Can graph neural networks count substructures? NeurIPS, 2020

* [3] Bouritsas et al. Improving graph neural network expressivity via subgraph isomorphism counting, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

* [4] Bodnar et al. Weisfeiler and Lehman go cellular: CW networks, NeurIPS, 2021.

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_YCJN
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the expressive power of graph neural networks from a different angle than the classical $k$-WL hierarchy. Authors argue that $k$-WL is not very well-suited for developing new graph neural network architectures since (i) these algorithms are already computationally demanding with $k\geq 2$, (ii) this hierarchy is not ``granular enough'' to capture graphs that lie between $1$-WL and $3$-WL, and most importantly (iii) $k$-WL algorithms rely on computations over $k$-tuples (and associated neighbourhoods), which are not local, and this does not align with the initial motivation of graph neural networks which aggregate information locally. Based on these (and similar) observations, authors propose a new hierarchy of graph isomorphism tests, coined as Neighbourhood-WL ($N$-WL)
algorithms. They prove that this is a proper hierarchy and introduce Graph Neighbourhood Neural Network (G3N) based on
the $N$-WL algorithms, and report empirical findings.
strength_and_weaknesses: Strengths:

- The study of expressive power of GNNs is important, and there is a value in looking beyond the WL hierarchy in designing novel models.
- Looking at multi-hop neighbourhoods in aggregation is at the core of this work, which is well-understood concept with potential benefits/

Weaknesses:

- The paper is hard to follow: For example, it is not immediately clear what the parameter $t$ exactly stands for in $N(d,t)$ and why it is important. It is stated relatively early that this is the size of the induced subgraphs, but there is no justification as to why this parameter needs to be controlled. In my understanding, this is due to computational inefficiency: the proposed class of algorithms enumerate all subgraphs of order $t$ within a $d$-hop neighbourhood, and this should be clearly stated. 

- The paper is not very well-placed in the literature. Authors have an extensive related work (a bulk of it is in the appendix, which is not a great practice in my opinion), but there are no concrete connections to other well-known concepts: To elaborate on this:

   -  $N(d,t)$ algorithms operate over $d$-hop neighbourhoods, which are defined using shortest path distances. Authors hint to the fact that that the class of $N(d,t)$ algorithms may not be a natural choice, since increasing $d$, while fixing $t$ does not necessarily yield more expressive power. They propose to use the positional information in the respective hops, which leads to a more intuitive class of algorithms: In my opinion, this is the right way of using the shortest path kernel within the algorithm. This paper builds on this idea to use positional info in the induced subgraphs, but the connections are not explored/explained. 

   -  It is not very clear whether one gains more by using the induced subgraphs, as compared to just using multi-hop aggregation (which also has strictly more expressive power than 1-WL and generally incomparable to the hierarchy). The presented ablation study confirms the role of $d$, but the same cannot be easily claimed for $t$.

   -  Authors are understandably critical about the $k$-WL hierarchy, but this is not always well-calibrated. I suggest authors tone these statements differently. There are many well-known connections between $k$-WL hierarchy and logics, which precisely inform us about the formal properties of these algorithms.

   -  There are many local variants of $k$-WL algorithms which are proposed with similar intuition: instead of aggregating over all $k$-tuple neighbourhoods, these algorithms only look at $k$-tuples within a local neighbourhood, which is not far away from the intuitions presented in this paper. The connections need to be discussed in depth, clarifying whether the proposed model has advantages over local WL algorithms.


- In the actual model, authors inject isomorphism types into node features (which is different than operating over isomorphism types), quoting the authors: ``However, we found that using different learnable linear transformations for all possible isomorphism
and positional types leads to much slower training. ... To remedy this problem, we instead only use different weight matrices for
positional types only, and inject the isomorphism type information as additional features to node embeddings."

- The given review time is not sufficient to check the proofs in detail. The theorem statements are not very specific: It should be stated what is fixed, and how the other parameter ranges. It is also unclear whether the statements are over the class of all graphs, or just over graph pairs. The WL hierarchy is an infinite hierarchy: Is the same true here? 

- WL algorithms can be used to analyse node-level, or link-level properties, and the same applies here, but it is unclear to me, e.g., under which conditions $(G_1,u)$ can be distinguished from $(G_2,v)$. I understand if this is left as future work, but it is important to highlight.

- The runtime of these class of algorithms is also prohibitive, and using a small $t$ seems the only way around the issue. I am not convinced that this new class of algorithms suggests a promising direction. 



clarity,_quality,_novelty_and_reproducibility: There is room for improvement in the presentation (please refer to my earlier comments). The main idea is clear but the proposed class of algorithms are somewhat new but they not very novel - they are closely related to local version of WL algorithms, WL sub-graph kernel, and shortest path kernel, which are not discussed.
summary_of_the_review: I agree with many points of the paper, but I am not convinced that this class of algorithms is the way forward. It is also unclear to me whether, or to what extent, this study improves our understanding of the field.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_e9AE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work tries to propose a new class of graph isomorphism testing algorithm, called Neighbourhood WL ($\mathscr{N}$-WL), which stores complex information in low dimension, instead of using high-order tuples in $k$-WL. It develops two Neighbourhood WL algorithms with parameters ($t$,$d$) and theoretically discusses whether the algorithms gain expressivity with parameters increased. Then, to reduce complexity, it further develops a connected-subgraph version of $\mathscr{N}$-WL and its GNN implementation. The proposed GNN model is evaluated on a number of synthetic and real-world experiments.
strength_and_weaknesses: Pros:
1. The discussion of related works in Section 2 and Appendix B is comprehensive.
2. The theoretical arguments of weak-hierarchy-related $\mathscr{N}^-$-WL and strong-hierarchy-related $\mathscr{N}$-WL seem to be nice, with sufficient proof and counter-examples for general settings of parameters ($t$,$d$).
3. The development from $\mathscr{N}$-WL to its connected-subgraph version $\mathscr{N}^c$-WL is great, accompanied with clear reasoning in Section 3.2.

Concerns:
1. I am a little confused by the definition of $f_{pos}$ under ''Strong Hierarchy''. Here are a series of questions:
    * there is a parameter $d$ in $\mathscr{N}$-WL. But there is again $\forall~ d\ge 1$ as the condition of ''... $f_{pos}(S_i)\neq f_{pos}(S_j)$''. Is the second $d$ not related to the first $d$? 
    * Is $f_{pos}$ unique or universal for any target node $u$? If unique, it seems very complicated to parameterize the representation of $J_d$ in Eq(4). If $f_{pos}$ is universal: moreover, if the answer to the above question is ''not related'', then it seems $f_{pos}$ is coloring everything distinctly because the target node can be arbitrary.
    * again, how is $J_d$ obtained in the model in Eq(4)?
2. The compared baselines in the ZINC experiment are not enough, because it only includes PPGN as the GNN beyond 1-WL. For instance, GSN is missing, which is cited and discussed by the authors for many times. It would be great to see the comparison with more expressive GNNs to show the sufficiency and efficiency of the proposed model.
clarity,_quality,_novelty_and_reproducibility: The reproducibility is good.

For clarity, please see the above concerns.

For quality, theorems and their proof generally seem to be in good shape, but I need to get the above concerns clarified to check the quality again.
summary_of_the_review: For now, I would like to recommend ''below acceptance threshold'' due to the mentioned concerns. I am willing to raise my score if the concerns are addressed.

=== Acknowledgement ===

Due to time limit, I only read the main text and the proof of Theorem 3.1. Other parts in Appendix were not thoroughly checked.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_QGtL
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a new hierarchy of expressive isomorphism test algorithm and GNN model, with combining k-WL tuple idea and rooted subgraph idea from subgraph GNNs. More specifically, the author still work on subgraph GNNs with each node representing the encoding of d-hop rooted subgraph around the node. Unliking 1-WL encoding of the d-hop rooted subgraph, the author proposes to apply t-set extraction for the d-hop rooted subgraph which should be much more powerful than 1-WL encoding. The author proved strictly increased expressivity with respect to increasing t and d, under the help of positional information of every t-set. Furthermore, the author claim that one only need to consider (≤)t-set whose induced subgraph are connected, and it does not lose any expressivity. The corresponding GNN model is also designed, and the author conducts extensive experiments over several widely used benchmark datasets for evaluating expressivity. 
strength_and_weaknesses: **Strength**: 
1. Going beyong k-WL hierarchy is an important direction, as k-WL is not practical with k > 3.
2. The designed hierarchy combines recently hot subgraph GNNs and k-WL, which is novel and an interesting angle.
3. The author proved strictly increasing expressivity with respect to increasing d hop and t size, which is a great contribution in theory. 
4. I personally find that theorem 3.7 is the most interesting one, however I'm not fully convinced that the theorem is correct, as I cannot fully  follow their proofs. (This theorem can reduce a lot t-sets, which can greatly improve runtime. It's hard to believe that this won't lose expressivity. )

**Weaknesses**:
1. Although the proposed hierarchy has increased expressivity with d and t. It is not clear the upper bound expressivity of the method. And the connect to k-WL is not clear also. It's possible that its maximum expressivity is limited.  
2. The proof of theorem 3.7 is questionable. 
3. The runtime analysis shows that the method is faster over sparse graph, but for denser graph this can be very slow.
4. When d is large, the computational cost can be closer to k-WL (its set version) for extracting all t-tuples. The main difference to k-WL will be that the intermediate representation is size n instead of size n^t. 
5. The experimental result over real-world datasets is not very good, this may suggests the expressivity upper bound of this method is limited. 

**Missing Reference**:
Studying an improved expressivity hierarchy has been explored. The paper [Zhao et al. NeurIPS 2022] improved k-WL by using ≤k sets and studied its expressivity theoretically. The theorem 8 in [Zhao et al. NeurIPS 2022] is similar to the paper's Lemma 3.6 and has connection to Theorem 3.7. They also used connected components. I suggest the author to have a check. 

**Additional Suggestion**:
As mentioned above, the proposed hierarchy is under the rooted subgraph WL presented in [Zhao et al, ICLR 2022]. The difference would be improving encoding of rooted subgraph with higher-order sets. The original paper uses 1-WL to encode rooted subgraph, which has overall expressivity upper bounded by 3-WL [Frasca et al. NeurIPS 2022]. It would be interesting to know that assuming we have the most expressive rooted subgraph encoder, what is the expressivity upper bound of the rooted subgraph WL? This upper bound would also be the upper bound of the proposed hierarchy. The author needs to study the upper bound of the proposed hierarchy to see whether it can achieve universality. 

[Zhao et al. NeurIPS 2022] A Practical, Progressively-Expressive GNN, https://arxiv.org/pdf/2210.09521.pdf.   
[Zhao et al. ICLR 2022] From stars to subgraphs: Uplifting any GNN with local structure awareness.   
[Frasca et al. NeurIPS 2022] Understanding and extending subgraph gnns by rethinking their symmetries.   
clarity,_quality,_novelty_and_reproducibility: The paper is clear written and not hard to follow. And the paper is focusing on an important direction of designing progressively expressive GNN not restricted by k-WL. The ideas and theorems presented are novel, given all theorems are assumed to be correct. 
summary_of_the_review: The author proposes an interesting design of color updating isomorphism test algorithm, and study it theoretically. It would be further improved if its upper bound is studied or the connection to k-WL is much more clear. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4131/Reviewer_rk5R
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper introduces a new family of isomorphism testing algorithms based on the WL tests hierarchy called Neighborhood-WL ($\mathcal{N}-$WL). Similarly to the WL test, the proposed algorithms iteratively refine a node coloring to determine whether two graphs are isomorphic. The difference from the WL test is that the node coloring is updated by aggregating the colors of subgraphs of size $t$ in the node’s $d$-hop neighborhood. The authors show that the parameters $(t,d)$ induce a strict hierarchy of isomorphism testing algorithms and further suggest and prove equivalence to a more computationally efficient variant aggregating a connected heredity neighborhood. Inspired by the $\mathcal{N}$-WL hierarchy, the authors propose a corresponding GNN design that can match the expressive power of $\mathcal{N}$-WL tests.


strength_and_weaknesses: ### 1. Strengths

- The paper proposes a new isomorphism testing hierarchy. 
- The authors provide a thorough theoretical characterization of the suggested hierarchy - expressivity, connection to WL tests, and complexity. 
- The paper introduces a flexible and general GNN architecture design based on the $\mathcal{N}$-WL hierarchy.
- An extensive experimental evaluation and ablation study is performed.

### 2. Weaknesses

#### 	1. **Relation to K-hop GNNs** - 
[1] analyses the expressive power of K-hop GNNs which basically, similar to this work, aggregate information from the k-hop neighborhood of each node and use this information to update the node's color at each iteration. In [1] they show that K-hop GNNs' expressive power is bounded by the 3-WL test. They further suggest a variant of K-hop GNNs to increase the expressive power, called KP-GNNs. I find the idea in KP-GNNs very similar to the addition of the positional types in the current work. 

#### 	2. **Experimental Results** - 
The experimental results, as in many GNN papers, justify the theoretical statements but do not exhibit any significant improvement in real-life datasets. Furthermore, I find the comparison on the ZINC dataset rather lacking (see Table 5 in [1]).

#### 	3. **Name of the paper** - 
The name of the paper is quite general and does not reflect the properties of the proposed hierarchy. I would suggest changing it to something of the flavor "Neighborhood WL hierarchy of expressivity for graph neural networks"

[1] Feng et. al. [How Powerful are K-hop Message Passing Graph Neural Networks](https://arxiv.org/abs/2205.13328 ) (NeurIPS 2022) 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Statements and theorems are proven and proofs are generally written in a clear way. 


summary_of_the_review: The paper provides a well-defined new hierarchy of isomorphism testing hierarchy. I find this construction elegant, and the generality of the suggested framework is appealing. My main concern regarding this work is the lack of discussion of the relation to K-hop GNNs that share many ideas with this work. Furthermore, [1] somewhat overlaps with this work, and an elaborate discussion on that is also required. 

I rate this paper as marginally below the acceptance threshold but would reconsider it upon clarifying and addressing my concerns. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold