Paper: An Efficient Subgraph GNN with Provable Substructure Counting Power (ICLR-2024)

Reviewer comment (Evaluation):
The experimental results are not very compelling. In particular, real-world experimental results for QM9 are strong in part, but performance on other tasks is substantially worse. Moreover, results on OGBG-HIV, ZINC (reported in the appendix), which ideally should also be in the main paper given that efficiency is reported on these, are not strong. Just to be clear: it is completely acceptable not to achieve SOTA results across the board, particularly against specialized approaches. However, I do expect some analysis of results (just like in QM9), leading to real-world scenarios where ESC-GNN is a clear and obvious candidate for use and would achieve the best results. This comment also applies to the synthetic experiments. I therefore ask the authors to revisit their experimental section and modify their analysis towards establishing a well-defined use case for their approach.

Author response:
Thank you for your valuable feedback and insightful suggestions. Your comments have been diligently considered, and we will integrate your suggestions into our revised manuscript.


**Importance of substructures.** We have incorporated an ablation study to show the importance of substructures in real-world tasks. The substructures, as detailed in Table 1, include 3-6 cycles, tailed triangles, chordal cycles, 4-cliques, 4-paths, and triangle-rectangles. We first report the number of these substructures at graph-level and node-level. Specifically, we report the average number of these substructures per graph (denoted as "graph") and the average number of these substructures that pass each node (denoted as "node"). The statistics are shown below.

|OGBG-HIV |3-cycle | 4-cycle | 5-cycle | 6-cycle | tailed triangle | chordal cycle | 4-clique | 4-paths | triangle-rectangles | 
|  :----: |  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
|graph |  2.69e-2 |  3.69e-2 | 7.00e-1 | 2.28 | 1.02e-1 | 4.84e-3 | 2.43e-5 | 68.38 | 9.36e-3|
| node |  3.16e-3 | 5.79e-3 | 1.37e-1 | 5.37e-1 | 4.01e-3 | 3.79e-4 | 7.63e-6 | 5.40 | 3.67e-4 |

|ZINC |3-cycle | 4-cycle | 5-cycle | 6-cycle | tailed triangle | chordal cycle | 4-clique | 4-paths | triangle-rectangles | 
|  :----: |  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
|graph |  6.41e-2  | 1.60e-2 | 8.52e-1 | 1.80 | 8.79e-2 | 0 | 0 |  55.15 | 0 |
| node | 8.30e-3 | 2.76e-3 | 1.84e-1 | 4.67e-1 | 3.80e-3 | 0 | 0 | 4.76 | 0 |


Note that 6-cycles are commonly observed within these two graphs. On average, a graph contains approximately two 6-cycles, and there is an average incidence of 0.5 6-cycles that pass a node. This observation underscores the significance of 6-cycle, whose presence may potentially be attributed to the existence of benzene rings.

We then evaluate the significance of these substructures. In particular, we use a base GIN framework as the backbone network, and adopt the count of substructures that pass each node as the augmented node features. In the table, “all” denotes adding all these substructures as the augmented node feature. Due to the time limit, we only run the experiment twice and report the mean result.

| Dataset | Baseline |3-cycle | 4-cycle | 5-cycle | 6-cycle | tailed triangle | chordal cycle | 4-clique | 4-paths | triangle-rectangles | all |
|  :----: |  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| ZINC |0.149| 0.145 | 0.150 | 0.143 | 0.115 | 0.146 | 0.152 | 0.149| 0.167 |0.149 | 0.100 |
| OGBG-HIV |73.89| 73.48 | 74.67 | 74.47 | 73.60 | 73.69  | 71.98 | 73.56| 68.40| 53.18| 71.47 |

**Analysis on the real-world benchmarks.** 

We observe that in most cases, the number of substructures does not boost the performance either on ZINC or on OGBG-HIV. The only exception is that the number of 6-cycles boosts the performance on ZINC (maybe due to the information of benzene rings), and the number of all these substructures boosts the performance on ZINC. We present potential illustrations for the observation:

Firstly, in these molecule graphs, it is imperative to incorporate not only structural but also semantic information to enhance the model. For instance, it is well-established that molecular fingerprints can significantly elevate performance on the OGBG-HIV dataset (Refer: https://ogb.stanford.edu/docs/leader_graphprop/#ogbg-molhiv). In particular, the fingerprints contain the number of specific molecule structures, encompassing both the graph structures of molecules and the semantic information of atom types and bond types.

Secondly, the general distance information contains more structural information than the number of specific substructures, thus contributing more to real-world performance. For instance, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle. 

Based on the observation, we extract further information from the distance encoding by refining the original method. In our original implementation, we solely employ an embedding layer for the extraction of distance information. During the rebuttal stage, we integrate a 2-layer MLP, supplemented with two batch normalization layers and two ELU activation layers, subsequent to the embedding layer. As shown in the table below, the implementation of this enhanced model structure on both ZINC and OGBG-HIV datasets has yielded a significant amplification in performance, providing empirical evidence that the distance encoding contains valuable information which includes but is not limited to the number of substructures.

|dataset |ZINC | OGBG-HIV | 
|  :----: |  :----: | :----: | 
| ESC-GNN| 0.086$\pm$0.003 | 78.62$\pm$1.06|  
| ESC-GNN (new) | 0.078$\pm$0.004 |79.21$\pm$0.84|

Reviewer comment (Experiments):
The paper's approach to subgraph counting is specialized towards common patterns, and only establishes relatively simple results. This is not a major weakness in itself (and is common in subgraph GNN literature), particularly given the complexity of general (induced) subgraph counting. However, the paper would be more interesting / compelling if it were to discuss more general sub-structure detection (more explicitly than via a connection to k-WL). To this point, the authors can strengthen the work by conducting case studies on real-world datasets to establish the importance of detecting cycles/paths/cliques. This would nicely complement the existing ablation studies in the appendix, and provide a meaningful explanation of when ESC-GNN is useful. As it stands, a main concern with subgraph GNNs is their over-specialization to pre-defined graph structures, and so any results / experiments to show more general structure detection / strong performance beyond the pre-designed use case, would substantially strengthen this paper.

Author response:
**Some comments on the proposed encoding.** Prior to delving into the analysis of general structure detection, it is important to point out that the proposed distance encoding is not specially designed for substructure counting. Rather, it should be considered as a catalyst for future research endeavors that develop an efficient (pre-computed) encoding scheme preserving the representation power (not limited to the counting power) of subgraph GNNs.



**More general structure detection.** The number of substructures with $n$ nodes will grow exponentially with the increasing of $n$. In particular, each of the $n$ nodes can be potentially connected to any of the remaining $n-1$ nodes, yielding a total of $\frac{n(n-1)}{2}$ possible edges. Given that each edge either exists or does not exist, there are $2^{\frac{n(n-1)}{2}}$ possible substructures in total. 

This enumeration includes multiple instances of isomorphic substructures. The maximum occurrence of any isomorphism type within this enumeration does not exceed $n!$. Therefore, the count of distinct isomorphism types of substructures with $n$ nodes is at least $\frac{2^{\frac{n(n-1)}{2}}}{n!}$. As a consequence, considering only the popular substructures can be a more practical choice.

Still, we can extend the theoretical results to a more general extent: **ESC-GNN can count any connected substructures with no more than 4 nodes.**

These substructures include edges (with 2 nodes), 2-paths, 3-cycles, (with 3 nodes), 3-paths, 4-cycles, tailed triangles, 4-cliques, and the “cyclic graph with a diagonal” (denote as $C$) (with 4 nodes).

It is obvious that ESC-GNN can count edges, and we have proven that ESC-GNN can count 2-paths, 3-cycles, 3-paths, 4-cycles, and 4-cliques. We now prove that ESC-GNN can count tailed triangles and $C$.

**Counting of $C$.** Assume that the number of $(1, 1)$ nodes as $k$. Recall that the $(1,1)$ nodes denote the nodes whose distances to the two target nodes are both $1$. The number of $C$ that passes the two target nodes is $\frac{k(k-1)}{2}$

**Counting of tailed triangles.** The tailed triangle is formed by a triangle and a tailed node, and there are generally two situations for the tailed node: (1) the tailed node is exclusively linked to a single target node; (2) the tailed node is connected to both target nodes. The latter scenario aligns identically with $C$, which can be computed by ESC-GNN. Addressing the first situation, assume the number of $(1,1)$ nodes is $k$, the number of $(1, 2)$ nodes is $m$, and the number of $(2,1)$ nodes is $n$. Then the number of situation one is exactly $k(m+n)$.

Reviewer comment (Theory):
I think the major weakness is that it seems that the model loses permutation equivariance due to the order of the encodings in $s_{uv}$. Specifically, consider the node-level distance encoding: for the subgraph rooted at edge $uv$ we have a distance histogram for $v$ and a distance histogram for $u$. Those two are concatenated (along with the other encodings) in $s_{uv}$. But which of the two distance histograms should be the first in the concatenation (that for $u$ or that for $v$)? If the edge is undirected no ordering should be preferred so choosing according to the node id leads to a choice that is not permutation equivariant.

Author response:
**A1.** Actually, our model satisfies permutation equivariance (PE). For your question, the information for node $u$ will be first in the concatenation. This is because our distance encoding is based on $k$-tuples which rely on the sequence of nodes rather than undirected edges that do not preserve the node sequence. Therefore, $s_{uv}$ and $s_{vu}$ can be different since $(u,v)$ and $(v,u)$ are different 2-tuples. 

We provide an intuitive illustration of why ESC-GNN satisfies PE. When computing the node representation for $u$, ESC-GNN aggregates the information from its neighbors $\\{v | v \in N(u)\\}$. In this aggregation, the model sums over all $s_{uv}$ instead of $s_{vu}$. The approach ensures that the sequence remains consistent and unaffected by any permutations.

Reviewer comment (Theory):
The second main weakness is that Proposition 4.5 does not seem to follow from previous work, and it is not proven in the paper. I think that the claim on page 5: “Previous works (Geerts, 2020; Frasca et al., 2022) show that for m ≥ 2 … a subgraph GNN rooted at k-tuples with backbone GNN as powerful as m-WL can be implemented by (m + k)-IGN.” is not true. Indeed it was shown only for $k=1$ and $m=2$. Therefore Proposition 4.5 is not immediate. I think it should be related to Proposition 2 in Qian et al 2022, which proves the same for any $k$ and $m=1$.

Author response:
**A2.** Thanks for your suggestion very much. We will add a formal proof in the revised manuscript, and provide a brief proof below. Huang et al. (2023) show that the distance information within subgraphs preserves the same representation power as the node marking policy, i.e., gives the rooted nodes an identity marking. We will then show that the subgraph GNN based on node marking policy can be implemented by $(k+m)$-IGN. (the word “implement” follows from Definition 3 in (Frasca et al.)).

First, the node marking policy can be implemented by the $(k+m)$-IGN following the proof of Lemma 4 in (Frasca. et al.). Assume that for a graph $G$, its input to the $m$-IGN is a $n^m * c_1$ matrix $X$, where $n$ is the number of nodes, and $c_1$ is the feature dimension. For the corresponding subgraphs, the input to the $(k+m)$-IGN is a $n^{(m+k)} * c_2$ matrix $H$, where $c_2$ is the feature dimension. For the $k$-tuple $(v_1, v_2, \dots, v_k)$, we have $H[v_1, v_2, \dots, v_k, v_i, \dots, v_i] = X[v_i, \dots, v_i]$  $\bigoplus$  $\mathbb{1}_{i, k}$ 
for each $i \in \\{1, \dots, m\\}$. In the equation, 

$\mathbb{1}_{i, k}$  denotes the one-hot $k$-dimensional vector being $1$ in dimension $i$ and 0 elsewhere and $\bigoplus$ denotes the concatenation between vectors. For other entries, $H[v_1, v_2, \dots, v_k, :, \dots, ] = X$ $\bigoplus$ 

$\mathbb{0}_{k}$ (the space is left since markdown will produce a wrong formula without the space), where $\mathbb{0}_k$  

denotes a zero vector of dimension $k$. Since the subgraph GNN models the permutation equivariant operations within $n^{m+k}$, it can be implemented by the $(m+k)$-IGN. An example can be found in Lemma 5 in (Frasca et al.)). Specifically, in the message passing period of (Frasca et al.), the node representation $X_{iii}$ can be updated by $X_{iij}$ and $X_{ijj}$, the node representation $X_{iji}$ can be updated by $X_{iji}$ and $X_{iii}$, and $X_{ijk}$ can be represented by $X_{ijk}$ and $X_{ijj}$. In a more general situation, we can replace the first node $i$ with the $k$-tuple, and the latter two nodes with the $m$-tuple that can implement the operation of $m$-WL (Geerts, 2020). As for the message aggregation period, the subgraph pooling followed by a global pooling can be represented by a non-adjacent-to-adjacent pooling followed by a global summation. Therefore the subgraph GNN can be implemented by $(m+k)$-IGN, and thus is bounded by $(m+k)$-WL.

Reviewer comment (Theory):
Please expand on the order of the two node-level distance encoding, as well as on Proposition 4.5, as explained in the Weaknesses.

Author response:
**A1.** Actually, our model satisfies permutation equivariance (PE). For your question, the information for node $u$ will be first in the concatenation. This is because our distance encoding is based on $k$-tuples which rely on the sequence of nodes rather than undirected edges that do not preserve the node sequence. Therefore, $s_{uv}$ and $s_{vu}$ can be different since $(u,v)$ and $(v,u)$ are different 2-tuples. 

We provide an intuitive illustration of why ESC-GNN satisfies PE. When computing the node representation for $u$, ESC-GNN aggregates the information from its neighbors $\\{v | v \in N(u)\\}$. In this aggregation, the model sums over all $s_{uv}$ instead of $s_{vu}$. The approach ensures that the sequence remains consistent and unaffected by any permutations.

**A2.** Thanks for your suggestion very much. We will add a formal proof in the revised manuscript, and provide a brief proof below. Huang et al. (2023) show that the distance information within subgraphs preserves the same representation power as the node marking policy, i.e., gives the rooted nodes an identity marking. We will then show that the subgraph GNN based on node marking policy can be implemented by $(k+m)$-IGN. (the word “implement” follows from Definition 3 in (Frasca et al.)).

First, the node marking policy can be implemented by the $(k+m)$-IGN following the proof of Lemma 4 in (Frasca. et al.). Assume that for a graph $G$, its input to the $m$-IGN is a $n^m * c_1$ matrix $X$, where $n$ is the number of nodes, and $c_1$ is the feature dimension. For the corresponding subgraphs, the input to the $(k+m)$-IGN is a $n^{(m+k)} * c_2$ matrix $H$, where $c_2$ is the feature dimension. For the $k$-tuple $(v_1, v_2, \dots, v_k)$, we have $H[v_1, v_2, \dots, v_k, v_i, \dots, v_i] = X[v_i, \dots, v_i]$  $\bigoplus$  $\mathbb{1}_{i, k}$ 
for each $i \in \\{1, \dots, m\\}$. In the equation, 

$\mathbb{1}_{i, k}$  denotes the one-hot $k$-dimensional vector being $1$ in dimension $i$ and 0 elsewhere and $\bigoplus$ denotes the concatenation between vectors. For other entries, $H[v_1, v_2, \dots, v_k, :, \dots, ] = X$ $\bigoplus$ 

$\mathbb{0}_{k}$ (the space is left since markdown will produce a wrong formula without the space), where $\mathbb{0}_k$  

denotes a zero vector of dimension $k$. Since the subgraph GNN models the permutation equivariant operations within $n^{m+k}$, it can be implemented by the $(m+k)$-IGN. An example can be found in Lemma 5 in (Frasca et al.)). Specifically, in the message passing period of (Frasca et al.), the node representation $X_{iii}$ can be updated by $X_{iij}$ and $X_{ijj}$, the node representation $X_{iji}$ can be updated by $X_{iji}$ and $X_{iii}$, and $X_{ijk}$ can be represented by $X_{ijk}$ and $X_{ijj}$. In a more general situation, we can replace the first node $i$ with the $k$-tuple, and the latter two nodes with the $m$-tuple that can implement the operation of $m$-WL (Geerts, 2020). As for the message aggregation period, the subgraph pooling followed by a global pooling can be represented by a non-adjacent-to-adjacent pooling followed by a global summation. Therefore the subgraph GNN can be implemented by $(m+k)$-IGN, and thus is bounded by $(m+k)$-WL.

Reviewer comment (Theory):
On page 7, the claim “As shown in Proposition 5.1, ESC-GNN is less powerful than subgraph MPNNs rooted at 2-tuples” does not seem correct, as according to Proposition 5.1 they can be as powerful as subgraph MPNNs. Please clarify.

Author response:
**A3.** For example, subgraph MPNNs rooted at 2-tuple can count 5-cycles (Huang et al. 2023), while ESC-GNN cannot. Together with Proposition 5.1, ESC-GNN is less powerful than subgraph MPNNs rooted at 2-tuples. We will clarify it in the revised manuscript.

Reviewer comment (Theory):
Does Theorem 4.4 hold for both induced and non-induced substructures?

Author response:
**A4.** It holds both for induced subgraph counting and subgraph counting. We will clarify it in the paper.

Reviewer comment (Experiments):
Why do you focus on node-level tasks in the experiments? I understand that node-level implies graph-level but the contrary is not true. Since you focus on graph-level tasks in the theoretical part, I don't understand why you test on node-level tasks. Furthermore, I noticed there is an additional counting experiment on ZINC in the appendix, but why is it limited to cycle counting?

Author response:
**A5.** The experiments follow the setting of (Huang et al.). We choose it instead of the graph-level counting task since it is more challenging, i.e., we can compute the number of substructures within the graph by the weighted sum of the number of substructures that pass each node. However, the reverse does not necessarily hold. Although our proof mostly focuses on graph-level counting, it can be potentially transferred to node-level counting. We briefly illustrate it below.

Our proof has shown that the 2-tuple level counting of the mentioned substructures can be done. For a substructure $s$, assumes the number of $s$ that passes the 2-tuple $(u,v)$ as $n_{uv}$, and the number of automorphism of the 2-tuple starting at node $u$ as $p$, i.e., the number of 2-tuples starting from $u$ that has the same isomorphic type as $(u,v)$ in $s$. Then the number of $s$ that passes node $u$ is $\sum_{v \in N(u)}n_{uv} / k$. Take the counting of 3-cycles $s = \\{u, v, w\\}$ that passes $u$ as an example, $k$ is 2 since in $s$, $(u, v)$ and $(u, w)$ have the same isomorphic type as $(u,v)$ ($(u,w)$, resp.). Considering that there is a cycle that passes $(u, v)$ ($(u, w)$, resp.), the number of 3-cycles that passes $u$ is $(1+1)/2=1$.

**A6.** We report the number of these substructures in ZINC. Specifically, we report the average number of these substructures per graph (denoted as "graph") and the average number of these substructures that pass each node (denoted as "node"). The statistics are shown below.

|ZINC |3-cycle | 4-cycle | 5-cycle | 6-cycle | tailed triangle | chordal cycle | 4-clique | 4-paths | triangle-rectangles | 
|  :----: |  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
|graph |  6.41e-2  | 1.60e-2 | 8.52e-1 | 1.80 | 8.79e-2 | 0 | 0 |  55.15 | 0 |
| node | 8.30e-3 | 2.76e-3 | 1.84e-1 | 4.67e-1 | 3.80e-3 | 0 | 0 | 4.76 | 0 |

We observe that ZINC may not be optimally suited for substructure counting tasks. A significant limitation is (1) the absence of certain substructures, such as 4-cliques, chordal cycles, and triangle-rectangles; (2) many substructures seldom exist in ZINC, such as the 3-cycles and 4-cycles. Therefore we put it in the appendix and do not report the performance on graphlets.

Reviewer comment (Presentation):
The time comparison on ZINC and OGB is presented in the main paper without reporting the results on those datasets in the main paper. Please move the results on these datasets in the main paper.

Author response:
**A7.** Thanks for your suggestion. It is temporarily not added due to space limits. We will add it to the main paper in the revised manuscript.

Reviewer comment (Experiments):
Why do the results on ZINC not include the standard deviation or average across seeds? And why do you use a graph transformer? What are the results with a GNN as a backbone model?

Author response:
**A8.** Actually, the reported result is the mean MAE score across ten individual runs. We will add the std score in the revised manuscript. Specifically, ESC-GNN reaches an MAE score of 0.086$\pm$0.003 on ZINC with the graph transformer as the backbone network ((now an MAE score of 0.078$\pm$0.004, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU)). In addition, it reaches an MAE score of 0.096$\pm$0.006 with GIN as the backbone network. Note that the proposed encoding can significantly boost the performance for both backbone models.

The reason for choosing the graph transformer as the backbone is based on empirical evidence. Previous works have shown that densely connected models such as the graph transformers (Ramp´aˇseketal.,2022.) or global information such as the laplacian eigenvectors (Lim et al. 2022) can boost the performance on ZINC. Since laplacian-based eigenvectors may violate permutation equivariance, we adopt the graph transformer without laplacian-based positional encoding as the backbone model.

Reviewer comment (Evaluation):
The results in Table 1 are not strong enough even if one considers the scalability gains due to hand-crafted embeddings.

Author response:
**A2.** First, our framework can count substructures with more than 4 nodes. For example, stars of arbitrary size and substructures like the tailed 4-cliques. While models such as PPGN (Maron et al.2019) can count up to 3-clique and up to 5-path, NGN (Zhang \& Li, 2021) can count up to 3-clique and up to 3-path, I$^2$-GNN (Huang et al. 2023) can count up to 4-clique and up to 5-path, they still make great contributions to the community.

In addition, the proposed encoding is a general distance encoding, and not designed to explicitly count these substructures. It somewhat contains more information than the number of specific substructures. For example, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle.


Furthermore, in terms of empirical evidence. There are works, such as the GSN (Bouritsas et al) that explicitly enhance GNNs with the number of substructures. This includes some substructures that are beyond the counting power of our model and other subgraph GNNs.  However, when evaluated on real-world datasets, GSN achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), GSN records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). This empirically shows the superiority of the proposed encoding.

Reviewer comment (Evaluation):
The drop in performance as one goes to 5-cycles and 6-cycles is quite severe, indicating poor generalization.

Author response:
**A2.** First, our framework can count substructures with more than 4 nodes. For example, stars of arbitrary size and substructures like the tailed 4-cliques. While models such as PPGN (Maron et al.2019) can count up to 3-clique and up to 5-path, NGN (Zhang \& Li, 2021) can count up to 3-clique and up to 3-path, I$^2$-GNN (Huang et al. 2023) can count up to 4-clique and up to 5-path, they still make great contributions to the community.

In addition, the proposed encoding is a general distance encoding, and not designed to explicitly count these substructures. It somewhat contains more information than the number of specific substructures. For example, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle.


Furthermore, in terms of empirical evidence. There are works, such as the GSN (Bouritsas et al) that explicitly enhance GNNs with the number of substructures. This includes some substructures that are beyond the counting power of our model and other subgraph GNNs.  However, when evaluated on real-world datasets, GSN achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), GSN records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). This empirically shows the superiority of the proposed encoding.

Reviewer comment (Novelty):
The paper may not provide any substantial research with potential for impact, mainly because of the hand-crafted nature of the proposed models based on ad-hoc theoretical arguments which have little value in the general case and are useful only in extremely specific instances (subgraphs of size at most 3 to 4).

Author response:
**A2.** First, our framework can count substructures with more than 4 nodes. For example, stars of arbitrary size and substructures like the tailed 4-cliques. While models such as PPGN (Maron et al.2019) can count up to 3-clique and up to 5-path, NGN (Zhang \& Li, 2021) can count up to 3-clique and up to 3-path, I$^2$-GNN (Huang et al. 2023) can count up to 4-clique and up to 5-path, they still make great contributions to the community.

In addition, the proposed encoding is a general distance encoding, and not designed to explicitly count these substructures. It somewhat contains more information than the number of specific substructures. For example, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle.


Furthermore, in terms of empirical evidence. There are works, such as the GSN (Bouritsas et al) that explicitly enhance GNNs with the number of substructures. This includes some substructures that are beyond the counting power of our model and other subgraph GNNs.  However, when evaluated on real-world datasets, GSN achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), GSN records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). This empirically shows the superiority of the proposed encoding.

Reviewer comment (Experiments):
Have the authors compared their results to more efficient subgraph-enhancement algorithms such as Qian (2022)?

Author response:
**A1.** As you mentioned, the subgraph sampling strategy accelerates subgraph GNNs while losing the theoretical counting power/expressiveness. To assess the practical implications of this trade-off, we conduct an empirical evaluation of the performance and efficiency of OSAN on ZINC and OGBG-HIV, and report the results in the table below. In the table, “Preprocess” denotes the time (second) to preprocess the data (generate subgraphs), and “Run” denotes the total time (second) to train and test. For the configuration of the subgraph sampling strategy, we adopt the official config file from https://github.com/chendiqian/OSAN/blob/master/configs/ogbg-molhiv/node_select/sel25_subgraph3_imle.yaml (OGBG-HIV) and https://github.com/chendiqian/OSAN/blob/master/configs/zinc/imle_with_esan_model/khop3_subgraph_10.yaml (ZINC).

We observe that despite employing a subgraph sampling strategy, OSAN is still not faster than ESC-GNN. This is because ESC-GNN only needs to run message passing on the whole graph, while OSAN needs to run message passing on a collection of subgraphs (even if the number of subgraphs is restricted). In addition, ESC-GNN and even MPNN outperform OSAN on both datasets, showing that the subgraph sampling strategy not only compromises theoretical counting power but also empirically diminishes the representational power of the model. The evidence from these comparisons underscores the superior efficiency and effectiveness of ESC-GNN in handling graph data.



| ZINC | MAE| Preprocess| Run | 
|  :----: |  :----: | :----: | :----: |
| MPNN |  0.163$\pm$0.004  | 6.2 | 1945.0 | 
| OSAN | 0.168$\pm$0.006|  182.8 | 8913.2 |
| ESC-GNN | 0.086$\pm$0.003 | 362.4 | 2872.2 |




| OGBG-HIV |AUCROC| Preprocess| Run | 
|  :----: |  :----: | :----: | :----: |
| MPNN |  77.07$\pm$1.49  | 2.7 | 6296.8 | 
| OSAN | 75.96$\pm$1.34 | 6.6 | 7980.1 |
| ESC-GNN | 78.62$\pm$1.06 | 1782.5 | 6301.0 |

Reviewer comment (Evaluation):
In Table 1, the column for "3-cycles" has all successful entries less than 0.001. ESC-GNN shows an error of 0.0074, yet it is in the same bracket. Can you explain how the cut-off of 0.01 for MAE was chosen?

Author response:
**A4.** First, we note that MPNNs and graph transformers (with positional encodings) cannot successfully count 3-cycles, therefore, “the column for "3-cycles" has all successful entries less that 0.001” is not true. In addition, the evaluation criterion follows (Huang et al. 2023). It is reasonable since we can directly use a rounding function to obtain the ground truth with such a low MAE score. Furthermore, we have proven in Section 5 that ESC-GNN can theoretically count 3-cycles.

Reviewer comment (Reproducibility):
"In conclusion, subgraph GNNs rooted at k-tuples with backbone GNN as powerful as m-WL can reach a similar counting power to (m + k)-WL while being much more efficient." What values of k and m do you use for experiments?

Author response:
**A5.** $k$ and $m$ are both set to 2.

Reviewer comment (Novelty):
Section 4.2: "Subgraph GNNs have long been used to count substructures. Existing works mainly focus on counting certain types of substructures, e.g., walks (You et al., 2021) and cycles (Huang et al., 2023) and do not relate subgraph GNNs with substructure counting in a holistic perspective." Does an incremental extension to cliques/paths of size <=4 really make your framework holistic?

Author response:
**A2.** First, our framework can count substructures with more than 4 nodes. For example, stars of arbitrary size and substructures like the tailed 4-cliques. While models such as PPGN (Maron et al.2019) can count up to 3-clique and up to 5-path, NGN (Zhang \& Li, 2021) can count up to 3-clique and up to 3-path, I$^2$-GNN (Huang et al. 2023) can count up to 4-clique and up to 5-path, they still make great contributions to the community.

In addition, the proposed encoding is a general distance encoding, and not designed to explicitly count these substructures. It somewhat contains more information than the number of specific substructures. For example, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle.


Furthermore, in terms of empirical evidence. There are works, such as the GSN (Bouritsas et al) that explicitly enhance GNNs with the number of substructures. This includes some substructures that are beyond the counting power of our model and other subgraph GNNs.  However, when evaluated on real-world datasets, GSN achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), GSN records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). This empirically shows the superiority of the proposed encoding.

Reviewer comment (Theory):
The bulk of the paper advocates higher-order GNNs but then the proposed method is the application of a standard GNN on an augmented graph? There is a bit of a mismatch between theory and the proposed method.

Author response:
**A1.** In Section 4, we show that subgraph GNNs are nearly as powerful as $k$-WL in terms of counting substructures, and the key reason is the integration of distance information within the subgraph framework. However, a notable limitation of subgraph GNNs is their suboptimal efficiency. To address the limitation, in Section 5, we accelerate them by pre-computing the distance information and enhancing MPNNs with the information. We then theoretically (Section 5) and empirically (Section 6) evaluate the proposed model and observe that it preserves the representation power of subgraph GNNs. In other words, the proposed model can be viewed as an efficient subgraph GNN which successfully preserves the representation power of classic subgraph GNNs.

Reviewer comment (Novelty):
The proposed method seems very related to approach by Bouritsas et al and Barceló et al in which subgraph information is used (isomorphism, homomorphism) alongside classical GNNs.

Author response:
**A2.** It is important to note that (Bouritsas et al) and (Barceló et al) explicitly encode the number of substructures to enhance GNNs. The approach heavily relies on domain-specific knowledge and the desired substructures may significantly vary across different tasks. In contrast, we encode the more general distance information of subgraphs, which enables plain GNNs to count many substructures without the need to manually determine which substructures to include. In addition, the general distance information contains more structural information than the number of specific substructures. For instance, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle. The superiority is also justified by empirical evidence. 

Regarding the real-world dataset, (Bouritsas et al) achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), (Bouritsas et al) records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). These results highlight the superiority of our proposed model.

Reviewer comment (Theory):
It is unclear what theoretical justifications of the proposed encoding method.

Author response:
**A3.** The proposed encoding is directly motivated by the theoretical results in Section 4, where we prove that the distance information within subgraphs is the key to boosting the counting power of subgraph GNNs. Building upon this theoretical foundation, we design the proposed encoding, as detailed in Section 5, and prove that it preserves essential structural information. This includes the ability to count substructures and differentiate between non-isomorphic graphs, thereby preserving the representation power of subgraph GNNs. It accelerates subgraph GNNs since it does not need to run message passing among all subgraphs.

To provide an intuitive example, consider the task of counting 3-cycles. Subgraph GNNs rooted on nodes can first compute the number of triangles that pass each node $u$ as the number of edges whose both endpoint nodes are with distance “1” to $u$. It then aggregates the numbers of all nodes and computes the number of 3-cycles within the graph. Notice that the distance information is actually contained in our proposed encoding.


In terms of the fingerprint (e.g., the one that boosts the performance on OGBG-HIV), we note that this fingerprint encompasses numerous molecular structures, a collection of which are identifiable (or countable) by subgraph GNNs. While it is feasible to incorporate these molecular structures within our proposed encoding framework, we do not include the information in our initial experiments since such information is excessively domain-specific, which could potentially limit the general applicability of the proposed model.

Reviewer comment (Theory):
What is the rationale behind the structural encoding presented in section 5. What guarantees does it give? Or other encoding methods possible? (a la molecular finger printing).

Author response:
**A3.** The proposed encoding is directly motivated by the theoretical results in Section 4, where we prove that the distance information within subgraphs is the key to boosting the counting power of subgraph GNNs. Building upon this theoretical foundation, we design the proposed encoding, as detailed in Section 5, and prove that it preserves essential structural information. This includes the ability to count substructures and differentiate between non-isomorphic graphs, thereby preserving the representation power of subgraph GNNs. It accelerates subgraph GNNs since it does not need to run message passing among all subgraphs.

To provide an intuitive example, consider the task of counting 3-cycles. Subgraph GNNs rooted on nodes can first compute the number of triangles that pass each node $u$ as the number of edges whose both endpoint nodes are with distance “1” to $u$. It then aggregates the numbers of all nodes and computes the number of 3-cycles within the graph. Notice that the distance information is actually contained in our proposed encoding.


In terms of the fingerprint (e.g., the one that boosts the performance on OGBG-HIV), we note that this fingerprint encompasses numerous molecular structures, a collection of which are identifiable (or countable) by subgraph GNNs. While it is feasible to incorporate these molecular structures within our proposed encoding framework, we do not include the information in our initial experiments since such information is excessively domain-specific, which could potentially limit the general applicability of the proposed model.

Reviewer comment (Novelty):
The proposed method uses handcrafted features (as part of encoding). How does it related to the work by Bouritsas et al in which edges carry counts of subgraphs?

Author response:
**A2.** It is important to note that (Bouritsas et al) and (Barceló et al) explicitly encode the number of substructures to enhance GNNs. The approach heavily relies on domain-specific knowledge and the desired substructures may significantly vary across different tasks. In contrast, we encode the more general distance information of subgraphs, which enables plain GNNs to count many substructures without the need to manually determine which substructures to include. In addition, the general distance information contains more structural information than the number of specific substructures. For instance, we have proven that the number of 4-cycles can be computed using the distance information. Conversely, deducing distance information solely from the count of 4-cycles is impracticable, given the multiplicity of configurations that can form a 4-cycle. The superiority is also justified by empirical evidence. 

Regarding the real-world dataset, (Bouritsas et al) achieves an MAE score of 0.115 on ZINC, whereas ESC-GNN attains an MAE score of 0.086 (now an MAE score of 0.078, after adding an MLP layer after the initial embedding layer. A detailed analysis is available in “**Analysis on the real-world benchmarks**” in our response to Reviewer aWFU). Similarly, utilizing the same backbone (GIN), (Bouritsas et al) records an AUCROC score of 77.99 on OGBG-HIV, whereas ESC-GNN demonstrates a higher AUCROC score of 78.62 (now an AUCROC score of 79.21). These results highlight the superiority of our proposed model.