Paper: Maximally Expressive GNNs for Outerplanar Graphs (ICLR-2024)

Reviewer comment (Experiments):
Authors only compare the CAT with the non-CAT version on MPNN. I believe more baseline models need to be included for completeness. It would be great to see pre-processing cost + training/inference cost + performance comparison on different baseline methods like subgraph-based GNNs and high-order GNNs.

Author response:
Please refer to the global response regarding additional experiments. We intend to include at least one additional baseline and include these metrics.

Reviewer comment (Experiments):
Authors claim CAT can help alleviate the problem of over-squashing and improve the performance of long-range tasks. However, no evaluations are performed. I believe some experiments should be conducted, like datasets in Long-Range Benchmarks [1].

Author response:
In the current submission, we focus on maximizing the expressivity on outerplanar graphs. In Table 3 we provide first evidence that $\text{CAT}$ typically reduces the diameter and the effective resistance of the graphs. However, as $\text{CAT}$ is not designed specifically for this, we see further investigation into whether $\text{CAT}$ can help reduce oversquashing and long range tasks in general as future work.

Reviewer comment (Evaluation):
Can the authors explain why the CAT achieved worse performance than the original one in MOLLIPO and MOLTOX21? Especially in MOLLIPO, the gap is significant.

Author response:
Unfortunately, we do not have insights on why $\text{CAT}$ performs worse on these datasets. Overall it is difficult to say why certain models do not perform well on certain datasets and it often requires more insight in both the dataset and the task at hand, as well as the strengths and weaknesses of the model. Our goal was not to achieve state-of-the-art performance on all datasets. Instead we show that $\text{CAT}$ in most cases increases the performance for commonly used datasets with outerplanar graphs (ZINC and the $8$ MOL- datasets from OGB).

Reviewer comment (Novelty):
**Scholarship**: The fundamental of this paper is to annotate graphs with labels to enhance the discriminative capacity of standard GNNs, which is widely studied under different node labelling approaches and a more thorough related work coverage is essential. Most importantly, there is a recent paper "Dimitrov et al, PlanE: Representation Learning on Planar Graphs, NeurIPS 2023" which introduces a *complete algorithm on the class of all planar graphs*. This paper strictly subsumes the result presented in the current submission.

Author response:
**Weakness Scholarship:**  
We address [1] in an answer to all reviewers, but we would be happy to address any further comments you may have. We would like to emphasize that while we focus on the smaller group of outerplanar graphs, our pre-processing works in linear time.

Reviewer comment (Theory):
**Technically incorrect statements**: There are many hand-wavy and sometimes incorrect statements. First of all, authors should cite the paper "Kiefer et al, The Weisfeiler-Leman Dimension of Planar Graphs is at most 3, LICS 2017" or its journal version and present the result correctly: The result states that 3-WL is complete on planar graphs (and not just on outerplanar graphs). Moreover, authors also confuse the dimension counts of WL: "...any GNN which matches the expressivity of 3-WL, such as 3-IGN (Maron et al., 2019) or 3-GNN (Morris et al., 2019), is capable of solving our main goal of distinguishing all outerplanar graphs." This is incorrect since Kiefer et al's result uses the classical WL algorithm also referred as the folklore 3-WL, so neither 3-IGN nor 3-GNNs have folklore 3-WL power. It is also open whether folklore 2-WL would suffice for planar isomorphism testing.

Author response:
**Weakness Technically Incorrect Statements:**  
> Technically incorrect statements: There are many hand-wavy and sometimes incorrect statements. 

We are happy to clarify any issues and provide more details if some statements were unclear. Please point us to additional statements you find hand-wavy or even incorrect, besides the one regarding the WL-dimension.

Regarding the latter statement, we have to disagree. There seems to be a small misunderstanding between the WL-dimension of outerplanar graphs and the WL-dimension of planar graphs. We purposely cited Kiefer's PhD thesis [2] instead of [3], as only her thesis contains a result specifically for outerplanar graphs. In particular, Corollary $7.40$ [2] states that the WL-dimension of outerplanar graphs is $2$. Note that what Kiefer calls WL is in the GNN literature typically called FWL (folklore WL). This means that $2$-FWL is necessary (i.e., $1$-WL / color refinement is not enough) and sufficient ($3$-FWL is not required) for outerplanar graphs. Furthermore, 2-FWL is equivalent to what the GNN community calls 3-WL. By the known equivalence of, e.g., $3$-WL-GNNs / $3$-IGN to $3$-WL and hence $2$-FWL, we see that the two mentioned GNNs are the $k$-IGN / $k$-WL-GNNs variants with minimum $k$ that one would need to distinguish all outerplanar graphs. Note that they have a runtime of roughly $\mathcal{O}(n^3)$.

By contrast, [3] shows that the WL-dimension of planar graphs is at most $3$ (and it is still open whether it could be $2$). Hence $3$-FWL and $4$-WL are sufficient to distinguish all planar graphs. By the previous explanation, this would require (as long as it remains open whether the WL-dimension is $2$ or $3$) to use $4$-IGNs / $4$-WL-GNNs, with an impractical runtime of roughly $\mathcal{O}(n^4)$.

Reviewer comment (Novelty):
**Significance and novelty**: The paper's technical contribution is limited. Leveraging 1-WL result in combination with CAT is interesting but relatively straightforward. My biggest concern is that there are complete neural models on planar graphs, which are also very scalable. The other technical contributions amount to arguing about shortening the propagation distance in the graph for better information flow. I find this analysis somewhat weak, because this effect can be trivially achieved by adding a virtual node or alike. It appears tangential to the study.

Author response:
**Weakness Significance and Novelty:**

> My biggest concern is that there are complete neural models on planar graphs, which are also very scalable.

To the best of our knowledge, there are no linear-time complete models for outerplanar graphs except $\text{CAT}$. PlanE requires quadratic time preprocessing (see global response). 

> The other technical contributions amount to arguing about shortening the propagation distance in the graph for better information flow. I find this analysis somewhat weak, because this effect can be trivially achieved by adding a virtual node or alike. It appears tangential to the study. 

This can indeed be trivially achieved with a virtual node or with other graph rewiring methods. However, the main goal of $\text{CAT}$ was to improve expressivity. As $\text{CAT}$ adds nodes and edges to the graph, we wanted to analyze how this affects other graph properties important for learning.  As such, we believe that showing that $\text{CAT}$ does not increase the diameter arbitrarily (and actually decreases diameter and average effective resistance in practice) is a useful result. After all, our analysis shows that it is **not** necessary to employ methods like virtual nodes in conjunction with $\text{CAT}$.

Reviewer comment (Presentation):
The presentation can be improved. For example, maybe the authors can consider presenting two molecules in Figure 5 before at the beginning of Section 3. It might be better to explain the intuition of Theorem 1 and Theorem 2 using specific examples (like Figure 5).

Author response:
**Weakness 1.** Improve presentation:

We will improve the presentation as suggested and motivate the two theorems better.

Reviewer comment (Theory):
It might be better if the authors can discuss more about the limitations and the potential of the mathematical techniques -- For example, does Theorem 2 hold true for general planar graphs, why or why not?

Author response:
**Weakness 2.** Discuss more (theoretical) limitations. Does Theorem 2 hold true for general planar graphs, why or why not?  

Currently, we have not investigated whether planar graphs in general can be distinguished by $\text{CAT}$. However, we conjecture that WL+$\text{CAT}$ cannot, since the biconnected components in general planar graphs do not necessarily have a unique Hamiltonian cycle. 
However, $\text{CAT}$ can also be applied to non-outerplanar graphs and potentially increase the expressivity. Additionally, we have first ideas how to generalize $\text{CAT}$ to distinguish all ($k$-outer)planar graphs (see answer to Reviewer DwdD, Weakness 3.).

Reviewer comment (Novelty):
Discussions of related work could be more comprehensive. For example, I found that this paper is highly related to the recent paper [1]. Both works share a common foundation rooted in biconnectivity, specifically concerning biconnected components and block cut trees, as well as a focus on distance metrics, encompassing the shortest path distance and resistance distance. Since the motivation of the two papers is also similar (both pointing out the importance of planar graphs in practice and giving molecular graphs as counterexamples), I feel that giving an in-depth discussion (perhaps within the introductory section of your paper), could further justify the paper's significance and provide a more comprehensive context for the reader.

Author response:
Thank you for your remarks regarding [1]. Yes, we are happy to extend our discussion of [1] in the related work in the final version of our submission. Most importantly note that all GNNs discussed in [1] have a time complexity of roughly $\mathcal{O}(n^2)$, while our approach has linear time complexity for both pre-processing and message passing. Additionally, we would like to point out that merely identifying block and cut vertices (and edges etc.) is not enough to distinguish outerplanar graphs. For example, the two non-isomorphic biconnected outerplanar graphs in our Figure 1 (as undirected graphs) have the same block-cut-vertex-trees (actually just one node). 

Moreover, it is worth noting that [1] proposes an architecture that relies on a transformer backbone, while $\text{CAT}$ is a graph transformation (i.e., a simple pre-processing), which allows for maximally expressive MPNNs (and any GNN that has at least WL expressivity) on outerplanar graphs.
We address the relation to the PlanE paper in our answer to all reviewers and will include a discussion thereof in the related work section of our final submission.

Reviewer comment (Writing):
The proposed CAT transformation is somehow complicated, in particular for the general case (Definition 2). I think it may be beneficial to discuss more about the intuition of Definition 2 in Appendix. Moreover, it introduces a lot of additional nodes and edges and even changes an undirected graph to a directed one. While it has indeed shown that the separation power is improved, significantly changing the structure of the input graph may result in some drawbacks.

Author response:
**Weakness 2.** Intuition:
We will add a thorough description of the intuition of $\text{CAT}$ to the appendix and include further examples.

> While it has indeed shown that the separation power is improved, significantly changing the structure of the input graph may result in some drawbacks.  

One possible drawback we considered was that $\text{CAT}$ might reduce graph connectivity. That is why we included an empirical analysis (Table 3) plus some theoretical guarantees (Proposition 1). We prove that even in the worst case $\text{CAT}$ never increases the diameter by more than 7. In the empirical analysis we actually observe that connectivity (in terms of average pairwise effective resistance and diameter) decreases in almost all cases. Could you please clarify which other drawbacks are meant here?

Reviewer comment (Experiments):
While the experimental results showed that an MPNN with CAT can outperform vanilla MPNN, such baselines are actually quite weak. I am not fully convinced to what extent the proposed approach is superior in practice.

Author response:
**Weakness 4.** Experiments:  
> While the experimental results showed that an MPNN with CAT can outperform vanilla MPNN, such baselines are actually quite weak. I am not fully convinced to what extent the proposed approach is superior in practice.

We will provide additional experiments and comparisons to a stronger baseline (see global response).
While MPNNs are no longer considered state of the art, we disagree that they are quite weak. For example, our GIN on MOLHIV outperforms 7 out of 8  models proposed in [3]. Our current experiments indicate that $\text{CAT}$ improves the predictive performance of MPNNs. We expect that $\text{CAT}$ can improve the performance for other GNNs.

Reviewer comment (Theory):
Can you illustrate more about how your approach may/may not be generalized to $k$-outerplanar graphs?

Author response:
**Weakness 3 / Question**: "Can you illustrate more about how your approach may/ may not be generalized to $k$-outerplanar graphs?"

The main property of outerplanar graphs that we use is that their biconnected components have unique and efficiently computable Hamiltonian cycles. For more general graph families, we would need to decompose the graph into subgraphs with unique Hamiltonian cycles or perhaps iterate over all Hamiltonian cycles. Each of the Hamiltonian subgraphs could then be processed by something similar to $\text{CAT}^*$. How to perform this decomposition is a highly non-trivial question. Moreover, there might be different decompositions of the same graph and so we would have to enumerate all possible decompositions, if we want to remain permutation-invariant.

We chose $k$-outerplanar graphs as an example for future work, as the small percentage of non-outerplanar graphs in molecular datasets is known to be $k$-outerplanar for a small $k$ (typically, $2$-$4$, see e.g. [4]). Also note that any planar graph is $k$-outerplanar for some $k$. Furthermore, $k$-outerplanar graphs, once embedded into the plane in a $k$-outerplanar way, allow per definition for a decomposition into the $k$ many ``shells''. Similarly one could inspect the _book embedding_ of a $k$-outerplanar graph, which effectively decomposes the graph into up to $3k$ outerplanar subgraphs (i.e. _pages_) [2]. As for typical molecules $k$ is a very small constant, we expect the number of different such decompositions to be manageable. We regard this as a promising future research direction.

Reviewer comment (Novelty):
Lacking clarifications and discussions of related work. In [1], the authors developed a framework for the whole planar graphs class, which seems very relevant to this work. It is highly recommended to (1) clarify the novelty and originality of this work against [1]; (2) add discussions on the relations, scopes and any other aspects of these two works for improved quality.

Author response:
**Weakness 1.** (Theoretical) comparison to PlanE:
We address the PlanE paper in detail in our answer to all reviewers and will include it in the related work in the final submission.

Thank you again for your review! We hope that we could resolve all of your questions. In particular, we discussed PlanE and added experiments on a larger dataset with over 250k molecules (ZINC 250k) in our global reply above.