Paper: Non-Redundant Graph Neural Networks with Improved Expressiveness (ICLR-2024)

Reviewer comment (Experiments):
The necessity of k-redundant Neighborhood Tree (k-NT) seems week. In Table 2, experiments on EXP-class, the performance seems to be always higher when k is smaller. Removing all redundant nodes seems to be the best choice, why use k as an selection?

Author response:
**W1** Since Theorem 1 applies only to $1$-NTs, it is currently conjectured that on graph level $0$-NTs are as expressive as $1$-NTs. However, since it has not been proven yet, the claim of being strictly more expressive than $1$-WL holds only for $1$-NTs. We show that it does not make sense to choose a very high $k$, because this introduces redundancy again, up until the point where the $k$-NTs are equal to the unfolding trees.

Reviewer comment (Experiments):
Experiments seems to be not sufficient enough to support the authors claim. For example in the abstract, authors claimed that the paper experimentally shows the method alleviates over-squashing. They have shown the results for synthetic datas in Table 2, but they are no experiments for real-world datasets to show this (such as experiments on long-range graph benchmark).

Author response:
**W2** Please refer to the global reply regarding further experiments and the LRGB.

Reviewer comment (Experiments):
In the introduction section, the authors mentioned PathNNs and RFGNN as closely related works. Also in table 3, the authors highlighted the best results from polynomial time complexity in bold. However, it seems that they are no comparison with any methods having polynomial time complexity other than linear.

Author response:
**W3** Please refer to the answer to **Q2** below.

**Q2** RFGNN does not have a polynomial but exponential time complexity. To the best of our knowledge, no methods currently exist that address the problem of reducing redundancy in graph neural networks and running in polynomial time.

Reviewer comment (Evaluation):
For experiment results in Table 1, 3, authors highlighted the best results with polynomial time complexity methods, emphasizing that DAG-MLP has advantages in time. What is the time complexity of DAG-MLP in terms of big-O notation? Also, is there any inference time comparison for the inference time of each method (GIN, SPN, PathNN, DAG-MLP)?

Author response:
**Q1** Please refer to the global reply regarding the time complexity of DAG-MLP.

Reviewer comment (Experiments):
Following weakness #4, is there are more baselines to compare with the paper method having a polynomial time complexity? What about the results of RFGNN mentioned for related works?

Author response:
**Q2** RFGNN does not have a polynomial but exponential time complexity. To the best of our knowledge, no methods currently exist that address the problem of reducing redundancy in graph neural networks and running in polynomial time.

Reviewer comment (Theory):
This work argues to alleviate over-squashing based on the results from RFGNN (Chen et al., 2022). However, as the authors argue, their proposed GNN architecture is different from RFGNN. Hence the logic is incomplete, i.e., it is not clear whether the proposed architecture alleviates over-squashing based on the same logic as RFGNN.

Author response:
**Comparison to RFGNN**
  * We elaborate on the relation to RFGNN regarding the ability to avoid over-squashing in the global reply. Our analysis also shows how redundancy is harmful to GNN tasks and that our approach provides advantages in this respect compared to MPNNs and RFGNN. We believe that these arguments lift your concerns.
  * RFGNN claims to be fully expressive (solve graph isomorphism) and has exponential runtime. This is a huge drawback. Additionally, it does not address the second type of redundancy (computational redundancy), which we also address in our submission.
  * Please refer to our global reply regarding time complexity.

Reviewer comment (Theory):
Upon reading Appendix A, the authors seem to claim that RFGNN introduces more redundancy compared to the proposed work. Since there is no clear explanation of how redundancy is harmful to GNN tasks, it is hard for me to understand the benefit of the proposed DAG-MLP.

Author response:
**Comparison to RFGNN**
  * We elaborate on the relation to RFGNN regarding the ability to avoid over-squashing in the global reply. Our analysis also shows how redundancy is harmful to GNN tasks and that our approach provides advantages in this respect compared to MPNNs and RFGNN. We believe that these arguments lift your concerns.
  * RFGNN claims to be fully expressive (solve graph isomorphism) and has exponential runtime. This is a huge drawback. Additionally, it does not address the second type of redundancy (computational redundancy), which we also address in our submission.
  * Please refer to our global reply regarding time complexity.

Reviewer comment (Theory):
The authors do not compare the expressive power of DAG-MLP compared to RFGNN. One might argue that RFGNN might be more expressive than the proposed DAG-MLP at the cost of introducing more redundancy.

Author response:
**Comparison to RFGNN**
  * We elaborate on the relation to RFGNN regarding the ability to avoid over-squashing in the global reply. Our analysis also shows how redundancy is harmful to GNN tasks and that our approach provides advantages in this respect compared to MPNNs and RFGNN. We believe that these arguments lift your concerns.
  * RFGNN claims to be fully expressive (solve graph isomorphism) and has exponential runtime. This is a huge drawback. Additionally, it does not address the second type of redundancy (computational redundancy), which we also address in our submission.
  * Please refer to our global reply regarding time complexity.

Reviewer comment (Experiments):
Overall, I think TUDataset is not good enough for evaluating the performance of DAG-MLP in practical scenarios. Especially, to validate the ability of DAG-MLP to alleviate over-squashing, I strongly suggest the long-range graph benchmark (Dwivedi et al., 2022) to run the proposed DAG-MLP.

Author response:
**Weak experiments**
  * Please refer to the global reply regarding experiments on LRGB.

Reviewer comment (Experiments):
The list of baselines is not comprehensive enough to check whether if performance improvement of the proposed DAG-MLP is practically relevant.

Author response:
Thank you for considering our response. We agree that the empirical running time is of importance. However, reliable comparisons are also difficult to achieve, e.g., because implementations are optimized to a different extent.

We are happy to consider DReW and other contemporaneous work in our comparison. However, we have to ensure that the same setup is used or reproduce the results for a fair comparison. For this reason, we compared to SJLR. However, our main focus is to show that our technique improves the message passing. We are not claiming to outperform rewiring techniques in general.

Reviewer comment (Evaluation):
How does the actual running time of DAG-MLP compare with the baselines in the considered experiments? I think this is an important criterion since the main (and possibly the only) benefit of DAG-MLP over RFGNN is the computational complexity.

Author response:
**Questions**
See answer to point 2 in “Weak experiments”. We also address this in our global reply. We will state the difference in computational complexity more clearly in the final version.

Thank you for considering our response. We agree that the empirical running time is of importance. However, reliable comparisons are also difficult to achieve, e.g., because implementations are optimized to a different extent.

Reviewer comment (Experiments):
k seems to be a hyper-parameter that would need to be tuned in practice.

Author response:
$k$ as a hyper-parameter: Setting $k$ to $1$ (or even $0$) should in general be the best solution. With higher $k$ values, redundancy is introduced again, up until the point where the $k$-NTs are equal to the unfolding trees. We do not expect NTs with $k$ greater than $1$ to be more expressive than $1$-NTs, therefore this parameter does not have to be optimized (other than maybe choosing between $0$ and $1$). If we could prove that $0$-NTs are as expressive on graph-level as $1$-NTs, then $0$-NTs could be used in any case for graph-level tasks.

Reviewer comment (Theory):
Even though DAG-MLPs are provably more expressive than the Weisfeller-Lehmann method, they are not proven to be fully expressive (i.e. distinguish any non-isomorphic graphs).

Author response:
Not fully expressive/Example: Indeed, DAG-MLP is not fully expressive. There are non-isomorphic graphs that it cannot distinguish (for example the $4\times 4$-Rooks and Shrikhande graph). There is always a trade-off between running time and expressivity. Currently, no polynomial time algorithms for solving graph isomorphism are known.

Reviewer comment (Theory):
The computational complexity of the proposed architecture and algorithms are not analysed but form an integral part of the contribution.

Author response:
We analyzed the size of the DAG, as well as the time to build it in our submission in Section 4.2. This makes up the crucial part of the running time for our method.  Please also refer to the global reply regarding the computational complexity.

Reviewer comment (Theory):
How expressive are the proposed DAG-MLPs? It seems like there could exist non-isomorphic graphs that cannot be distinguished by DAG-MLP. What would be an example?

Author response:
Not fully expressive/Example: Indeed, DAG-MLP is not fully expressive. There are non-isomorphic graphs that it cannot distinguish (for example the $4\times 4$-Rooks and Shrikhande graph). There is always a trade-off between running time and expressivity. Currently, no polynomial time algorithms for solving graph isomorphism are known.

Reviewer comment (Experiments):
How does the expressive power compare with baseline methods?

Author response:
Please refer to the global reply regarding the expressivity of the baseline methods.

Reviewer comment (Evaluation):
What is the computational complexity of building and evaluation DAG-MLPs? What are their memory requirements?

Author response:
We analyzed the size of the DAG, as well as the time to build it in our submission in Section 4.2. This makes up the crucial part of the running time for our method.  Please also refer to the global reply regarding the computational complexity.

Reviewer comment (Theory):
The main weakness is the computational cost, which requires O(nm) space where n is the number of nodes and m is the number of edges. This brings a significant limitation to the applicability of the proposed method, even for moderate-sized graphs.

Author response:
**W1** We agree that the increase in running time is a drawback of our approach. However, our method has significantly lower computational complexity than comparable approaches. We elaborate on this further in the global reply.

Reviewer comment (Experiments):
The experimental result only shows occasional marginal improvements over some baselines and only on a few datasets. This is not enough to demonstrate the effectiveness of the proposed method.

Author response:
**W2** We compare our approach to the most recent state-of-the-art methods in Table 3. Since these methods have been shown to outperform the baselines, we believe that this is appropriate. In [1] there are results for other methods, but Path-NN outperforms them in most cases. We provide additional experimental results supporting the advantages of our method in the global reply.

Reviewer comment (Theory):
One main motivation for the proposed method is to address over squashing, but there is no theoretical analysis of the proposed method to address it.

Author response:
**W3** Thank you for asking how our approach formally relates to over-squashing. We referred to the analysis in [2] without making the relation explicit in our manuscript. We have elaborated this in the global reply showing that our approach can formally be related to over-squashing and improves in this respect over existing methods. We believe that including these arguments will strengthen our manuscript.

Reviewer comment (Experiments):
What is the largest graph size that the proposed method can handle?

Author response:
**Q1** This depends on the properties of the graphs and on the hardware used. Giving a general answer to this question is generally difficult (also for standard GNNs).

Reviewer comment (Evaluation):
What is the preprocessing time for the proposed method?

Author response:
**Q2** In practice, this again depends on the hardware used. On a 64 CPU our non-optimized implementation performs the preprocessing for the 600 protein graphs of the dataset ENZYMES (33 nodes and 62 on an average) in 7.8 seconds for 1-NTs. We believe that the theoretical time complexity is more meaningful (although it does not take our compression techniques into account) and refer to the global reply.