Paper: Counting Graph Substructures with Graph Neural Networks (ICLR-2024)

Reviewer comment (Presentation):
Writing quality: The presentation and the layout of the paper need to be improved substantially. Equation spacing/referencing is not correct with editing hints unremoved. The paper is hard to follow. A clear introduction/navigation at the beginning of each section is lacking. This is particularly the case in the methodology part, which confuses the reader.

Author response:
We would like to thank the reviewer for their suggestion. We have substantially improved the paper presentation in the revised manuscript. We would be happy to implement further changes that the reviewer may suggest after reviewing our revised manuscript.

Reviewer comment (Experiments):
Experiments: The experiments show that the proposed GNN has good capability of substructure counting. However, this may not lead to better performance on downstream tasks. There are many benchmark datasets commonly used for node-level and graph-level tasks. Testing on more datasets will better demonstrate how counting substructures can help improve the performance of other tasks.

Author response:
We would like to thank the reviewer for their comment. We want to point out that we are testing our approach in 5 different datasets. Two synthetically generated datasets, to test the main scope of the paper (counting substructures), and 3 real benchmark datasets (ZINC, REDDIT-B, REDDIT-M) to assess the performance and generality of our approach in downstream tasks. We are also adding experiments with the full ZINC dataset in the revised version and would be happy to include additional datasets that the reviewer suggests. However, we believe that the current set of experiments is sufficient to show the effectiveness and generality of our approach. Note that the contribution of our work is not only practical but has a strong analytical component which we prefer to maintain.

Reviewer comment (Experiments):
For the regression task on ZINC, the method proposed in ‘Graph Neural Networks with Local Graph Parameters’ accepted by NeurIPS 2021 seems to have a better result, which is not included in this paper.

Author response:
Thank you for bringing this work to our attention, which we cite and include in the revised version of the paper. However, it is not true that this method achieves better performance. In fact, the best performance for the regression task on ZINC, which is reported in ‘Graph Neural Networks with Local Graph Parameters’ is MAE 0.1353 ± 0.01, whereas our proposed approach achieves MAE 0.110 ± 0.005.

Reviewer comment (Experiments):
The comparison to other baselines on cycle counting and ZINC is insufficient.

Author response:
Note that our approach is generic and we try to keep the comparisons between more general approaches for fairness. Nevertheless, we have compared the performance of our approach against 25 different baselines in our experiments. We are also including 5 more baselines from ‘Graph Neural Networks with Local Graph Parameters’ on ZINC in the revised manuscript. However, we would be happy to include any further comparisons/discussions that the reviewer may suggest.

Reviewer comment (Writing):
I feel like the sentence in the abstract "However, their ability to count substructures, which play a crucial role in biological and social networks, remains uncertain" may be confusing in the sense that, the counting ability of regular MPNN is pretty clear as shown in [1].

Author response:
We agree with the reviewer that [1] has provided significant insights into the counting ability of message-passing GNNs. [1] studies the counting capabilities of message-passing GNNs with either constant or symmetric (structural) input signals. In that case, the counting capacity of message-passing GNNs is upper bounded by the 1-FWL test. However, our approach shows that random input signals can better exploit the power of message-passing operations and improve their counting capacity beyond 1-FWL. As a result, the ability of message-passing GNNs to count substructures is indeed very clear with symmetric input signals, thanks to [1], however when the input is not symmetric this research question is still open. 

To avoid possible confusion we rephrase this part to 'However, there are still open questions regarding their ability to count and list substructures, which play a crucial role in biological and social networks."

Reviewer comment (Experiments):
Many powerful, expressive, and well-adopted GNN baselines are missed. For instance, GNN-AK[1], GrphGPS[2], CIN [3], ....

Author response:
We would like to thank the reviewer for bringing these works to our attention. We have cited and added discussions on your suggested papers in the revised manuscript.

Reviewer comment (Theory):
Proposition 5.2 show that Moment-GNN can improve expressivity over 1-FWL. What about the upper-bound on its' expressive power? Is it bounded by 3-WL like subgraph GNNs [1]?

Author response:
We would like to thank the reviewer for motivating us to look deeper into the expressivity of our approach. In our new results, we can prove that message-passing GNNs with random inputs can count 8-node cycles and 4-node cliques. These operations however do not admit a closed-form expression with purely Hadamard product operations but they also involve diagonals of the tensor Tucker model. As a result, the expressivity of message-passing GNNs with random inputs and the expectation operator in a hidden layer is not bounded by the 3-WL test. However, it could be the case that Moment-GNN (as defined in (14) in the revised manuscript) is indeed bounded by the 3-WL test, but we haven't proven it yet. Please see the general response and Section 6 in the revised manuscript for more details.

Reviewer comment (Writing):
The word "equation" is redundant in many places.

Author response:
Thank you for your suggestion. We have significantly improved the paper writing in the revised version. Unfortunately the word "equation" was generated automatically from the ICLR template whenever the command \eqref was used, which explains the repetition. We have addressed this issue in the revised version.

Reviewer comment (Writing):
What is the role of the characteristic function in page 3? Why is it unrelated to x?

Author response:
The short answer is that input x is a random vector which can be fully described by its characteristic function, the same way a random vector can be fully described by its density function (e.g., Gaussian function). The characteristic function can also be viewed as the Fourier Transform of the density function. Since x can be completely described by its characteristic function it is also completely related to x.

We understand that there is a confusion regarding input x and its properties and therefore have updated the first part of Section 3, which now reads:

The goal of our paper is to answer the research question in 1.1 and characterize the expressive power of message-passing GNNs in terms of counting important substructures of the graph. To do that we study  Equation 2, when the input is a random vector $\mathbf{x}\in\mathbb{R}^N$. Our work focuses on the ability of message-passing GNN operations to generate expressive features, therefore the input $\mathbf{x}$ should be completely uninformative, i.e., structure and identity agnostic. To ensure that $\mathbf{x}=\left[x_1,x_2,\dots,x_N\right]^T$ is agnostic with respect to the structure of the graph, we assume that $x_i$, $i=1,\dots,N$ are independent random variables. To ensure that $x_i$s are identity agnostic, we assume that they are identically distributed and satisfy $\mathbb{E}\left[ x_i\right]= 0$ $\mathbb{E}\left[ x_i^p\right]= 1$, for $i=1,\dots,N$ and $p\geq 2\in\mathbb{Z}$. Overall, the elements of $\mathbf{x}$ are independent and identically distributed (i.i.d.), with joint characteristic function $\phi_{\mathbf{x}}\left(t_1,\dots,t_N\right)=\prod_{i=1}^N \left(e^{jt_i}-jt_i\right)$, and the moments of the joint distribution of $\mathbf{x}$ satisfy: 

$\mathbb{E}\left[\mathbf{x}\right]=\mathbf{0},\quad
    \mathbb{E}\left[\mathbf{x}\mathbf{x}^T\right]=\mathbf{I},\quad
   \mathbb{E}\left[\mathbf{x}\circ\mathbf{x}\circ\mathbf{x}\right]=\underline{\mathbf{I}},\quad
    \mathbb{E}\left[\mathbf{x}\circ\mathbf{x}\circ\dots\circ\mathbf{x}\right]=\underline{\mathbf{I}}$

Reviewer comment (Theory):
The authors only proved positive results for the expressive power of GNNs using high-order moments, which are incomplete. Does the GNNs fail without high-order moments? In other words, are the theoretical results tight? For example, can the GNNs count 6-cycle using only 2-order moment, and can the GNNs count 7-cycle using only 3-order moment?

Author response:
The point the reviewer is raising is important and is worth further clarification. Our results are sufficient but not necessary. For example, we cannot guarantee that 7-node cycles cannot be computed using the 3rd-order moments. However, it is important to note that the question we are addressing in this paper is not whether high-order moments of certain graph filters can count substructures, but whether message-passing GNNs can count substructures. High-order moments are not the goal but rather the tool we are using to analyze the counting abilities of GNNs.

For instance, a 2-layer GNN with square nonlinearities implicitly or explicitly computes 4th-order moments, and a 3-layer GNN with square nonlinearities computes 6th-order moments. To make things more interesting, GNNs with analytic pointwise nonlinearities, such as the hyperbolic tangent, can linearly combine infinitely high-order information in their output with just a single layer. As a result, the question the reviewer is posing is indeed very interesting, but more from a graph theory perspective, rather than GNN analysis.

Reviewer comment (Theory):
Theorem 4.1 may not be meaningful, as it uses a GNN to fit the number of connected components of a *single* graph, which is just equivalent to fitting a constant if understood correctly.

Author response:
We respectfully disagree with the reviewer's comment. Theorem 4.1 demonstrates that message-passing GNN operations are able to generate powerful node embeddings that, when summed across nodes, enable the computation of the number of connected components in a graph. Considering that message-passing GNNs with constant or structural inputs are provably unable to generate this information, we find the significance of Theorem 4.1 noteworthy.

However, we agree with the reviewer that our other theorems prove more powerful results. In particular, we establish that message-passing GNNs can learn how to count cycles and quasi-cliques for any graph. As a result, our other theorems not only characterize the expressive power of GNNs but also their generalization ability.


In summary, Theorem 4.1 is crucial in describing the expressive power of GNNs and is applicable to graphs observed during training. On the other hand, all the other theorems characterize both the expressive and generalization power of GNNs and are also applicable to graphs that were not observed during training. Parts of this discussion appear in the revised manuscript.

Reviewer comment (Reproducibility):
The proposed method relies crucially on taking the expectation in the final layer. How do you implement this in your experiments? Will a large number of samples be needed?

Author response:
In fact the proposed approach **does not** necessarily rely on taking the expectation in the final layer. One can either implement the architecture with the expectation or use the equivalent closed-form expression of Equation 14. In our implementation, we employ the equivalent closed-form expression. Since we are using a single Moment GNN layer followed by GIN layers, the exact expressions are described in Equation 65 of the Appendix. Detailed discussion about this topic can be found in Proposition 5.1, Remark 5.1, and Section 7.1 of the revised manuscript.

Reviewer comment (Reproducibility):
From the results, the GNNs can even count 8-cycles perfectly, but the current theory did not prove it. What is the number of moment order required in your experiments?

Author response:
Please see the general comment.

Our new results, which are presented in Section 6 of the revised manuscript, prove that message-passing operations can actually count 8-node cycles. These results however do not directly apply to the implementation we used in our experiments. In the experiments, the moment order is a hyperparameter which is selected between 3-5.