Paper: WL-Tree: a New Tool for Analyzing Graph Neural Networks (ICLR-2024)

Reviewer comment (Theory):
The proposed method is not well defined. The formulations in Equations 3-5 are not consistent. In Equation 3, an anchored subgraph is defined in terms of a set of walks but Equation (5) defines an anchored subgraph as a set of pairs of nodes and walks. Further, the definition of $\dot{\cup}$ is not clearly presented. Also, why $dist(i,j)$ is only less than $\ell$ but $dist(i,k)$ is less than or equal to $\ell$?

Author response:
Equation 3 is the definition, and equation 5 is derived from equation 3. The idea of equation 5 is as follows: if i is the anchor of an anchored subgraph, then the anchored subgraph can be viewed as the union of 1) i and i's neighbors and 2) anchored subgraphs surrounding i's neighbors. 

In the edge set of equation (4), we have $\{(j, k) \in E: \mathrm{dist}(i, j) < \ell, \mathrm{dist}(i, k) \le \ell \}$. A better writing should be $\{(j, k) \in E: \mathrm{dist}(i, j) \le \ell, \mathrm{dist}(i, k) < \ell \}$, which is consistent with our usage of $j$ in the node set. It means that: if $(j, k)$ is included in the edge set of $S_i^\ell$, then it must in a length-$\ell$ walk starting from $i$, then the distance between $i$ and one end needs to be less than $\ell$.

Reviewer comment (Theory):
The proposed WL-trees differ from the computational tree structures of message-passing GNNs mainly in disallowing the revisit of the parent nodes. The authors claim that the proposed WL-trees are equivalent to the 1-WL algorithm in terms of node coloring. This does not seem correct. Consider a counter-example, where G is a graph consisting of two triangles and H is a cycle of length 6. These two graphs cannot be distinguished by 1-WL, but would have different WL-trees proposed in the paper.

Author response:
We will have the same WL tree, which has the following pattern: each node has two children.

Reviewer comment (Theory):
The tree structures underlying message-passing GNNs and their connection to 1-WL have been well studied in the literature. It is unclear why the proposed WL-trees can provide a more fine-level analysis of the expressiveness of node representations learned by message-passing GNNs. In particular, the proposed WL-trees are not equivalent to 1-WL (see the above point [2]).

Author response:
As we have shown in our work, a WL-tree is equivalent to a WL-color. We have clarified W2 as well. In this regard, our WL-trees provide a new way of analyzing graph structures behind node representations. It is easier to use because 1) it connects well with the original graph structure, and 2) it is easy to visualize.

Reviewer comment (Evaluation):
In what kinds of scenarios will the proposed algorithm 1 be useful?

Author response:
The algorithm is able to enumerate graph structures that give the same WL tree/color. When we want to analyze these structures, e.g. checking which structures a GNN cannot distinguish, then we can use the algorithm to do so. In fact, we use the algorithm to check how many graph structures can possibly be confused by GNNs in section 6.

Reviewer comment (Experiments):
For section 6, what are the justifications for selecting CLIP and Nested GNN? There are a large number of GNN models developed in the literature. I don't see why these two particular GNN models are selected for analysis.

Author response:
Because these two GNNs can be best explained by our WL-trees. We plan to examine the expressiveness of more GNNs in our future work.

Reviewer comment (Theory):
Theorem 12 and Theorem 13 look confusing. Why is max used in Equation 8? Is the notation $G(j,h)$ defined? Also, the expressive power of Nested GNN goes beyond 1-WL, but the WL-trees proposed in the paper are claimed to be equivalent to 1-WL. So why does Theorem 13 state that there is a bijective mapping between WL-trees and their node embeddings calculated by Equation 10?

Author response:
The max operation is taken over all possible colorings $\bar{c}$. It is from the CLIP-2 method -- we just formalize the method. 

Both the two GNNs first color graph nodes with extra information (random colors or local message-passing) and then run a vanilla GNN to get node representations. So we can use our WL trees to quantify the abilities of their vanilla GNNs using extra node colors.

Reviewer comment (Novelty):
Although WL-tree seems to be a new concept, I did not quite get what perspective from which it is important compared to existing 1-WL algorithm results. It is known that message passing GNN, such as GIN, are capturing the rooted subtree around each node, which is exactly the structure captured by 1-WL, as shown in Figure 1 of [1]. As defined in Section 4, the WL-tree proposed in this paper is also such a rooted subtree. The only difference is that the rooted tree in this paper does not include the parent of a node as its child. It is not clear to me why this difference is important and how it brings significant differences compared to existing understanding. (Please correct me if I am understanding wrongly or incompletely.)

Author response:
There are at least two benefits if we do not include the parent in the child. First, it is easier to discuss a node's WL color in a tree -- our formulation directly uses the original tree structure without adding new nodes. Second, it is easier to enumerate graph structures behind a WL color. With our formulation, we only need to merge some nodes to get a graph structure. If a node's parent is also included in the child list, then the situation is much more complex.

Reviewer comment (Novelty):
Also, it is unclear what advantages or new understandings can be inspired by this new concept. Could you summarize what findings we can get from this new analysis tool?

Author response:
First, with the new tool, it is easier to discuss which graph nodes have the same color/node representations. For example, showing that a GNN with a fixed depth cannot distinguish a graph node from a particular tree node. Second, it is easier to explain how previous GNNs enhance node representations. Random node colors give a particular tree pattern (some nodes much take the same color), so that the graph structure can be separated from others that have the same WL color. 

As we mentioned in our submission, this new tool is fundamentally no different from WL colors but it makes some analysis clearer, particularly when we can visualize examples with WL trees. As a comparison, it is more abstract to discuss WL colors and logic to reach the same conclusion.

Reviewer comment (Experiments):
The experiments only show a simple analysis of two existing GNN models. What new model designs this new concept can lead to? This is not obvious from the reading. I think the experimental section can include deeper analyses or include a model inspired by the introduced WL-tree tool.

Author response:
In our experiment, we use our new method to enumerate graph structures corresponding to a WL tree/color. We show that advanced GNN methods are able to eliminate a large number of possible graph structures by including extra node colors. But they do not do much better in terms of distinguishing graph structures **in the dataset**: a WL tree/color may correspond to many graph structures, but if only one of these structures is in the dataset, then it still can be distinguished by a vanilla GNN. Our experiment explains why we don't see a significant increase in performance when we use stronger GNNs on some datasets.