Paper: InterpGNN: Understand and Improve Generalization Ability of Transdutive GNNs through the Lens of Interplay between Train and Test Nodes (ICLR-2024)

Reviewer comment (Theory):
Although the authors derive PAC-Bayesian bound for GNNs in the transductive setting and show how the interplay between training and testing sets influences the generalization ability, I fail to see the strong connection between the theoretical analysis and the proposed method. The proposed method seems to simply adopt the idea of the self-attention mechanism from the transformer and apply it to the graph. It's not clear to me how the proposed method enhances the generalization for the distant nodes.

Author response:
We wish to clarify that our method extends beyond merely applying the self-attention mechanism to graphs. Unlike traditional graph transformers that computes node-to-node attention, our approach calculates attention between nodes and a memory component. This memory stores crucial node features and then broadcasts this information to all nodes, acting as a bridge for transferring information across distant nodes. This is particularly significant for large-scale graphs in mini-batch training, where the memory enables efficient information flow between out-of-batch nodes, thus enhancing the overall connectivity and information dissemination in the graph neural network.

Reviewer comment (Theory):
In equation 7, why do you need max operation since $L-r+1>0$ always holds?

Author response:
The max operation is indeed a typo and will be removed.

Reviewer comment (Evaluation):
In Figures 1 and 2, the group 2 usually have the higher accuracy and smaller generalization error. Do you have any explanation for this observation?

Author response:
The accuracy observed on real dataset is not entirely equivalent to the generalization error discussed in Theorem 1. A generalization bound is a theoretical limit on a model's expected performance with unseen data, whereas test accuracy is an empirical measure obtained by evaluating the model on a fixed test dataset. While smaller generalization bounds suggest potential for higher accuracy, as indicated by the downward trend in Figure 1, they do not guarantee higher accuracy. The actual accuracy depends on factors like model complexity, training data quality and quantity, and the model's alignment with the data distribution.

Additionally, regarding Figure 1 (test nodes grouped by interplay, as analyzed in our theorem), Group 2 only shows higher accuracy than Group 1 across all models on the CiteSeer dataset. In contrast, on the Cora dataset, all four baseline models perform better in Group 1 than in Group 2. This variation can be attributed to the differing levels of homophily in these datasets. Specifically, CiteSeer exhibits weaker homophily, meaning that adjacent nodes are less likely to share the same label. The homophily of a node $i$ is defined as the ratio of neighbor nodes with the same label:

$$ h_{i}=\frac{|\{j \in \mathcal{N}_ {i}: y_{i}=y_{j} \} |}{d_{i}} $$

Thus, the overall homophily of the dataset is:

$$ h = \frac{\sum_ {i \in V} h_{i}}{|V|} $$

The homophily of Cora dataset is 0.81 and the homophily of CiteSeer dataset is 0.74.

Reviewer comment (Experiments):
In the experiment on comparison with GNN variants using Dummy nodes, what is the method (denoted as Dummy nodes in Figure 7) used for comparison? This experiment is confusing.

Author response:
Dummmy nodes are introduced in [1] to capture global graph properties and boosting graph structure learning. Dummy nodes are initialized with a zero vector and conneted to all other nodes in the graph. Dummy nodes can then aggregate information from all nodes. 

The reason we compared our method with dummy nodes is that our memory in InterpGNN-GW acts somewhat like a dummy node. But instead of gathering information from all nodes like a dummy node does, our model uses attention to pick and store features from important nodes into the memory. Also, in mini-batch training, dummy nodes cannot get information from nodes in all batches, which our method can do by using the global workspace.

[1] Liu, Xin, et al. Boosting graph structure learning with dummy nodes. ICML 2022.

Reviewer comment (Novelty):
The idea of the Global Workspace is intriguing, but it has already been extensively explored in previous research mentioned by authors, diminishing the novelty of this work.

Author response:
We are grateful for the reviewer's interest in the Global Workspace concept. Indeed, while the general idea of Global Workspaces has been explored in various fields, our application within the realm of graph analysis introduces novel contributions: 

In previous approaches, the Global Workspace primarily served to ensemble different specialized modules. These modules, often pre-trained neural networks, are designed for diverse functions such as sensory perception (including visual or auditory classification, object segmentation), long-term memory storage, reinforcement learning  agents. The interaction of these modules' outputs with the workspace facilitated the extraction of higher-level features. 

In contrast, our method utilizes the Global Workspace to store critical node features, addressing specific challenges in graph analysis. Our primary focus is on enhancing the interplay between distant nodes and facilitating inter-batch node information exchange. This approach effectively mitigates the issue of excessive node aggregation often encountered in node-pair Graph transformers. Additionally, considering that GNNs often employ mini-batch training methods on large-scale graphs, leading to a loss of information between different batches, our method facilitates cross-batch information exchange via global workspace, thereby boosting performance on large-scale graphs.

Reviewer comment (Writing):
Some concepts lack a comprehensive explanation, such as W_{r,k} in Eq. (10). Further refinement is needed for these elements, including a clearer explanation of the dimensions of linear layers, which would enhance the clarity of the paper.

Author response:
We apologize for the confusion caused by a typo in Eq. (10) of our paper. The correct equation should be:

$$
\widehat{\mathbf{H}_ c}=\operatorname{softmax}\left(\frac{\mathbf{H}_ c \mathbf{W}_ {r, q}\left(\mathbf{M}^ {t+1} \mathbf{W}_ {r, k}\right)}{\sqrt{d_ {k}}}\right) \mathbf{M}^ {t+1} \mathbf{W}_ {r, v}
$$

We now provide a detailed explanation of the parameters in Eq. (9), Eq. (10), and Eq. (11). The node feature matrix is denoted as $X \in \mathbb{R}^ {N\times d}$, where $N$ is the number of nodes, and $d$ is the dimension of node features. An arbitrary GNN transforms these features into $H \in \mathbb{R}^ {N\times n _f}$, and we obtain the positional embedding $H _{pe} \in \mathbb{R}^{N\times n _p}$. These are concatenated to form the input to the global workspace, which consists of $n _m$ memory slots, each a vector of length $l _m$, represented as $\mathbf{M} = [m _1, \cdots, m _{n _m}]$.

When writing node features to the memory with Eq. (9), keys and values are obtained by multiplying the node embedding with two different linear layers, i.e., $\mathbf{K} _w=\mathbf{H} _c \mathbf{W} _{w,k}$ and $\mathbf{V} _w=\mathbf{H} _c \mathbf{W} _{w,v}$, where $\mathbf{W} _{w,k}$ and $\mathbf{W} _{w,v}$ have input dimensions of $(n _f + n _p)$ and output dimensions of $d _k$ and $d_v$, respectively. The subscript $w$ indicates these values and linear layers are used when writing node features to the Global workspace. The query $\mathbf{Q_w}$ is obtained by multiplying the memory with a linear layer $\mathbf{W} _{w,q}$, whose input dimension is $l_m$ and output dimension matches that of the keys' linear layer. The resulting matrix multiplication of keys and query gives the node-memory attention scores $\mathbf{Q} _w \mathbf{K} _w ^\top \in \mathbb{R}^ {N\times n _m}$. After applying Softmax to these attention scores, they are multiplied with $\mathbf{V} _w$ to select important nodes for updating the memory.

Eq. (10) involves similar linear layers as Eq. (9), with the difference being the swapping of positions between memory and node embedding, to reload contents from the memory back to the nodes. Therefore, the output dimensions of $\mathbf{W} _{r,k}$, $\mathbf{W} _{r,v}$, and $\mathbf{W} _{r,q}$ remain $d_k, d_v, d_k$, respectively, while their input dimensions change to $l_m, l_m, n_f+n_p$.

In Eq. (11), an additional graph convolution operation is performed, where $\mathbf{A} \in \mathbb{R} ^{N\times N}$, $\widehat{\mathbf{H} _c}  \oplus \mathbf{H} _c \in \mathbb{R} ^{N\times (n_f+n_p)}$, and $\mathbf{W} _o \in \mathbb{R} ^{(n_f+n_p)\times K}$, with $K$ being the number of classes.

Reviewer comment (Experiments):
Results provided in table 3 and 4 are limited.

Author response:
For small graphs, it is indeed true that the improvements by our model are not as prominent. This is primarily due to the limited size of these datasets, which inherently makes it challenging for any model to demonstrate significant improvements. However, it is worth noting that our method still outperforms the baseline models on small datasets. On the other hand, the performance enhancement of our method is  notable on large graphs. The reason for this improvement is due to the out-of-batch node communication via global workspace.


Additionally, following Reviewer mAhi's suggestion, we conducted tests in a non-rigorous inductive learning setting. In this scenario, where the model doesn't have access to test node features during training, our method effectively broadcasts information from the training phase to test nodes. Our methods significantly mitigates the performance drop observed in other models when transitioning from a transductive to an inductive learning setting. Our method can achieve close results to those in the transductive setting. For a comprehensive overview of experiments on non-rigorous inductive setting, please refer to the A5 to Reviewer mAhi.

Reviewer comment (Presentation):
In Figure 2, the criteria for grouping based on node degree are somewhat confusing. For example, what do the numerical values on the x-axis in the figure represent? I assume these values are simply categories for grouping, rather than actual node degrees. I believe the authors could provide a clearer explanation of the grouping criteria.

Author response:
In Figure 2, test nodes are sorted by node degree in descending order and evenly divided into five groups. The x-axis represents these groups, with lower indexes indicating higher node degrees. Specifically, Group 1 (smallest index) contains nodes with the highest degrees, and Group 5 (largest index) comprises those with the lowest.

Reviewer comment (Theory):
There is a discrepancy between theory and practice. The ''transductive'' or ''semi-supervised'' setting the authors adopt is the same as that in [1], where the randomness come from the labels of nodes. However, in real-world applications, including the experiments of this paper, both nodes features and their labels are fixed, and the randomness come from the random partition of training and test nodes [2,3,4]. Therefore, I think that the authors should use the results in literatures [5,6], e.g., Corollary 7 in [6] to derive similar results as Lemma 1. This can be done by incorporating the techniques in works [7,8]. Otherwise, the authors should conduct some synthetic experiments that treating labels as random variables to verify their theoretical results.

Author response:
**Q1: The randomness should come from the random partition of training and test nodes instead of labels. Otherwise, the author should conduct some synthetic experiments.**
**A1**: In response to the reviewer's comments on the use of randomness in our approach, we provide the following clarifications and results:

Firstly, PAC-Bayes, as a method for analyzing generalization error, fundamentally assumes that with probability $1-\delta$, data are sampled from a fixed but unknown distribution, and that model parameters are also drawn from a distribution. Hence, randomness originates from the sampling of data and model parameters. Generalization error is measured through the divergence between the model's performance on prior (before see training data) and posterior probabilities. This assumption is the most commonly used in PAC-Bayes bounds, and these bounds have been experimentally validated on real datasets [9-13]. Without loss of generality, assuming data is sampled from a true distribution $(x,y)\sim D$, we fix $x$ and assume labels are sampled from a real probability distribution $y\sim Pr(x|G)$, following [1]. Works [2,3,4] do not use the PAC-Bayes framework, hence they operate under different assumptions than our theorem.

Secondly, in the context of PAC-Bayes analysis, the randomness coming from the labels of nodes is equivalent to randomness coming from the partition of training and test nodes, since it is assumed that data are sampled from a true distribution. Since the partition follows a uniform distribution, adding a uniform distribution on the data distribution remains invariant. In [6], although the analysis is for transductive learning and assumes randomness from partitioning, the bounds are derived by exhaustively enumerating the number of classification errors $K$ across all $N$ samples (training + test), i.e., $\text{bounds} = \max _{K=0 \ldots N} (\cdot)$, where $N$ is the total number of samples. Then the error of training set follows a hypergeometric distribution.  The maximum generalization error necessarily occurs in the scenario where all classifications are incorrect, which results in trivial bounds.

Thirdly, following your suggestion, we conducted experiments on a synthetic dataset. We used the Stochastic block model implemented with PyTorch Geometric to generate a graph dataset. The node features of each block are sampled from normal distributions, with the centers of clusters being vertices of a hypercube. Due to the memory limitation of 128GB, we could not generate large-scale datasets with the CBM. We created a graph with 6 categories of nodes, 1000 nodes each, where the probability of connections within the same category was set at 0.02 and 0.01, and the probability of connections between different categories was set at 0.001. The other settings remained consistent with those in the paper, and the experimental results are as follows:

| same-class prob | 0.02 | 0.01 |
| --- | --- | --- |
| GCN | $29.11 \pm 0.25$ | $26.51 \pm 0.29$ |
| ClusterGCN | $30.19 \pm 0.21$ | $25.76 \pm 0.33$ |
| NodeFormer | $30.93 \pm 0.31$ | $27.12 \pm 0.20$ |
| InterpGCN | $31.89 \pm 0.18$ | $27.87 \pm 0.27$ |

Reviewer comment (Experiments):
The authors claim that the proposed module is plug-and-play, which means that it can be applied on any GNNs. However, they only equip GCN with this module. Performing more experiments that applying this module to other type of GNNs is encouraged.

Author response:
**Q2: Performing more experiments that applying this module to other type of GNNs is encouraged.**

**A2**: In response to your suggestion to demonstrate the plug-and-play nature of our proposed module by applying it to various types of Graph Neural Networks (GNNs), we have expanded our experiments beyond GCN. It is important to note that in our paper, we have already replaced GCN with GnnAutoScale (GAS). Following this, we have integrated our module with other GNN architectures including GAT, GCNII, and SGC. The results of these experiments on various datasets are as follows:

|  | Reddit | Aminer-CS | Amazon2M |
| --- | --- | --- | --- |
| InterpGAT-GW | 91.97 $\pm$ 0.29 | 66.87 $\pm$ 0.34 | 85.81 $\pm$ 0.17 |
| InterpGCNII | 92.43 $\pm$ 0.17 | 67.66 $\pm$ 0.35 | 87.58 $\pm$ 0.28 |
| InterpSGC | 93.33 $\pm$ 0.24 | 68.41 $\pm$ 0.27 | 88.79 $\pm$ 0.22 |
| InterpGCN | 93.35 $\pm$ 0.20 | 68.76$\pm$ 0.31 | 89.11 $\pm$ 0.14 |
| InterpGAS | 92.75 $\pm$ 0.23 | 69.01$\pm$0.22 | 88.59$\pm$ 0.19 |

Additionally, following the suggestion of Reviewer wkbK, we conducted tests on Heterophily graph datasets. We embedded our global workspace module into heterophily graph neural networks, which resulted in performance improvements. Please see details in A8 to reviewer wkbK.

Reviewer comment (Evaluation):
The experiments show that the proposed module could also reduce structural unfairness. Can you provide a further explanation for this phenomenon? If the reason is the interplay between training and testing nodes induced by attention mechanism, I guess that graph transformer models can also reduce structural unfairness.

Author response:
**Q3: Explanation for reducing structural unfairness. If the reason is the interpaly between training and testing nodes induced by attention mechanism, graph transformers models can also reduce structural unfairness.**

**A3**: The reduction in structural unfairness in our model is attributed to the enhanced interplay between training and testing nodes facilitated by the workspace. Similarly, graph transformer models are indeed capable of reducing structural unfairness through their attention mechanisms. The generalized error bound we derived is not exclusively limited to our method; rather, we aim to use Theorem 1 to explain the structural unfairness phenomenon in message-aggregating GNN. This also theoretically analyzes why methods that strengthen interplay, such as deep GCN and implicit connections (e.g., graph transformers), can enhance generalization performance.

Reviewer comment (Evaluation):
Graph Transformer models (e.g., GraphGPS) could also enhance the interplay between training and testing nodes more or less. Why these methods perform inferior to the proposed method?

Author response:
**Q4: Graph Transformer models (e.g., GraphGPS) could also enhance the interplay between training and testing nodes more or less. Why these methods perform inferior to the proposed method?**
**A4**: You are indeed correct that graph transformer models can enhance the interplay between training and testing nodes. The proposed method, InterpGCN-GW, surpasses graph transformer models primarily due to two reasons:

**Out-of-Batch Interplay**: InterpGCN-GW can facilitate inter-batch interplay. Graph transformers on large-scale graph datasets typically compute in-batch attention, which limits access to out-of-batch nodes. In contrast, the workspace in our method can be broadcast to nodes across all batches, enhancing interplay among inter-batch nodes and providing global consistency.

**Filter unnecessary nodes to be aggregated**: InterpGCN-GW requires aggregating a significantly smaller number of nodes compared to graph transformers. Graph transformers with node-pair attention tend to aggregate information from all nodes, potentially incorporating noise and diminishing the impact of important nodes. Our method selectively stores important node embeddings in the memory through attention, effectively acting as a condensation or filter of nodes.

Reviewer comment (Experiments):
"results concerning graph structure is missing". Homophily/Heterophily is one of the most important properties of the graph structure, but is missed in this paper [1-4]. See [4] for a study with PAC-Bayesian generalization bound.

Author response:
**A1**:  Thank you for pointing out the need to discuss heterophily graphs. Theorem 1 in our work can be extended to include heterophily graphs with an additional assumption. The primary distinction between homophily and heterophily graphs lies in whether connected nodes belong to the same category. In our theoretical derivation, we assume that the true labels are solely dependent on features, meaning nodes with similar features are likely to be in the same class. Formally, for two nodes $i$ and $j$ with labels $k$, the difference in their label probabilities is bounded by their feature difference: 

$$ |P(y_i=k|x_i) - P(y_j=k|x_j)| \leq C|x_i - x_j|, $$

where $C$ is a constant determined by the true distribution. In heterophily graphs, we can add a heterophily coefficient related to node distance to this assumption: 

$$ |P(y_i=k|x_i) - P(y_j=k|x_j)| \leq C \cdot h_r^{(uv)} \cdot |x_u - x_v|, $$

where $r$ is the distance between nodes $i$ and $j$. In homophily graphs, $h_1$ is small, indicating a tendency for neighboring nodes to be in the same class. Conversely, in heterophily graphs, $h_1$ is larger, implying neighboring nodes are likely to belong to different classes. Similar to our defined L-hop interplay between training set $S$ and test set $U$, we can define L-hop heterophily as:

$$ \mathcal{H}_ L  = \frac{ \sum_{r=1}^L (L-r+1) \cdot |h_r^{i,j}|}{|S| \cdot |U|}, $$

where $i \in S$, $j \in U$. In the final generalization error, an additional term on $\mathcal{H}_L$ is included. Taking the dating network dataset from [6] as an example, adjacent nodes are more likely to belong to different classes. Here, due to typically small interplay values, L-hop heterophily $\mathcal{H}_L$ has a more significant impact on generalization error than L-hop interplay $(1-I_L)$. For non-adjacent nodes, as heterophily is usually not strong (as discussed in R2), L-hop interplay remains the dominant factor.


Regarding the PAC-Bayesian generalization bound in [4], a similar approach to heterophily is introduced. However, [4] makes strong assumptions about graph datasets, limiting the extendibility of the derived generalization error bounds. Their generalization error is based on that graph is generated with a contextual stochastic block model (CBSM), assuming features of different classes originate from distinct normal distributions, with different connection probabilities $p$ and $q$ for inter-class and intra-class nodes, respectively. The final generalization error bound based on [7] includes the homophily ratio difference between training and test sets.

Reviewer comment (Theory):
"more information interaction between test and training nodes can lead to smaller generalization error" — I doubt the correctness of this claim on heterophilic graphs, see [3].

Author response:
**A2**: The statement regarding the impact of information interaction on generalization error can be more accurately phrased as follows: "More information interaction between neighboring test and training nodes can lead to a smaller generalization error for homophily graphs, while more information interaction between non-neighboring test and training nodes can lead to a smaller generalization error for heterophily graphs." This concept is supported by a survey on heterophily GNNs, which states, "On heterophilic graphs, nodes with high structural and semantic similarities might be farther away from each other," and "The representation ability of heterophilic GNNs can be significantly improved by capturing important features from distant but informative nodes," as quoted from [8].


These elucidate why our method  by writing important nodes into the memory and broadcasting them to all nodes, exhibits superior performance on heterophily graphs (experimental results in R8).  Similarly, many GNNs aggregate information from non-neighboring neighbors. For instance,  MixHop [9], H2GCN [10] propose aggregating information from higher-order neighbors at each message passing step. Similarly, UGCN [11] utilizes two-hop networks for message passing, while GGCN [12] employs cosine similarity to send signed neighbor features under constraints of relative node degrees. GPR-GNN [13] uses learnable weights to adaptively combine representations from each layer via the Generalized PageRank (GPR) technique.

The discussion in [3] does not conflict with these statements.  [3] still focuses on the influence of neighboring nodes' label distribution, which necessitates considering the inter-class distinguishability of neighbors.  In the example provided in [3], if two nodes have 2 and 4 neighboring nodes with labels (2, 3) and (2, 2, 3, 3), they would still be classified similarly after aggregation. This supports the necessity of aggregating non-neighboring node features, consistent with our method's approach.

Reviewer comment (Novelty):
Compared to the graph transformers based on inner-batch pair-wise attention, the global workspace can provide global consistency. The idea of building global consistency is interesting.

Author response:
**A3**: We appreciate your recognition of the capability of our methods for ensuring global consistency.

Reviewer comment (Novelty):
"Implicit connection" [5] establishes a graph rewiring method to rebuild some implicit connection between nodes.

Author response:
**R4**: We will include the graph rewiring method in our related works section.

Reviewer comment (Writing):
"the labels of training set $S_Y$, the features of both training and test nodes $X = [S_X,U_X]$". The notations are weird. I suggest "$S_Y$" → "$Y_S$", "$X = [S_X,U_X]$" → "$X = [X_S,X_U]$".

Author response:
**A5**: Thank you for the feedback on notations; we will make the suggested corrections in our revised version.

Reviewer comment (Theory):
"premise that embedding exchange among connected nodes leads to closer output logits" — This premise is invalid for heterophilic graphs. See examples in [1-3].

Author response:
**A6**: To clarify, the output logits referred to are the outputs of the final layer of the graph neural network. Consider a simple graph neural network with only two nodes, $i$ and $j$. In the case where $i$ and $j$ are not connected, the difference in their outputs is $|\sigma(x_iW) - \sigma(x_jW)|$, where $\sigma$ denotes the activation function, and $ W $ is the weight matrix. However, if $i$ and $j$ are connected, their  output after embedding exchange becomes $|\sigma\left(\frac{x_u+x_v}{2}W\right) - \sigma\left(\frac{x_u+x_v}{2}W\right)|$, which simplifies to 0. This demonstrates that embedding exchange among connected nodes indeed leads to closer output logits. In the context of heterophilc graphs, we only need to refine our thorem to reflect that closer outputs among neighboring nodes can lead to larger generalization error on heterophilc graphs, as indicated in **A1**.

Reviewer comment (Theory):
"Theorem 1 interpret the structure imbalance phenomenon." How does Theorem 1 interpret the structure imbalance phenomenon? You need to elaborate it in the main paper. You only test on homophilic datasets (Cora, CiteSeer, PubMed). To make claims about grouping effect, you need also to test on heterophilic datasets.

Author response:
**A7** If the test set nodes have more connections with the training set nodes, this results in a larger $L$-hop interplay value, leading to a smaller generalization bound value. For instance, consider two groups selected from a dataset: Group 1 ($U_1$) comprising nodes directly adjacent to the training set, and Group 2 ($U_2$) consisting of nodes with no pathways to the training set. In a three-layer graph neural network, the 3-hop interplay for $U_1$, denoted as $\mathcal{I}_2^{(1)}$, will be at least $\frac{2}{|U_1|}$. In contrast, the 2-hop interplay for Group 2 will be 0. Therefore, the generalization error upper bound for Group 1 is smaller than that for Group 2.

We conducted grouping tests using GCN on heterophilic datasets including Actor, Chameleon, and Squirrel. We select 20 nodes from each class for the training set, 500 for validation set, and 1000 for test nodes. After repeating the experiments 100 times, the average results are as follows:

| Group index | Actor | Chameleon  | Squirrel |
| --- | --- | --- | --- |
| 1 | 22.3 | 51.5 | 33.3 |
| 2 | 21.5 | 47.4 | 32.0 |
| 3 | 22.6 | 45.9 | 31.0 |
| 4 | 22.6 | 45.1 | 32.2 |
| 5 | 23.2 | 39.4 | 28.1 |

As observed, in Chameleon and Squirrel datasets, different groups show a tendency to decline in performance correlating with the decrease in interplay.

Reviewer comment (Experiments):
Ablation study on memory and the gate control of memory doesn't show the effectiveness of memory and the gate control. The proposed models don't show statistically better performance than SOTA. Again, the model needs to be tested on heterophilic graphs.

Author response:
**A8**: In response to the reviewer's comments regarding the ablation study on memory and gate control, as well as the performance comparison with state-of-the-art (SOTA) models, especially on heterophilic graphs, our clarifications and findings are as follows:

1. Firstly, it's important to clarify that our ablation study of Memory is not about removing the memory part but initializing it at each training epoch. Memory still can  store node features. A performance drop is observed if we completely remove the memory part, i.e., compare ClusterGCN with InterpGCN-GW. The gate control aspect follows the implementation from previous global workspace studies, and we agree that its contribution to the graph global workspace is relatively small.

2. Our method shows modest improvements on small-scale graphs but large benefits on large-scale graphs. This advantage stems from the ability of our global workspace component to facilitate information exchange with nodes in other batches during the training and test phase. Following Reviewer mAhi's suggestion, we also tested our model in a non-rigorous inductive learning setting, where the model does not access test node features during training, yet our methods can broadcast information from the training phase to test nodes. In this scenario, other models exhibit a substantial performance drop compared to transductive learning, while our method achieves results similar to those in the transductive setting. For a detailed presentation of these results, please refer to **A5** to Reviewer mAhi.





3. We have conducted experiments on heterophily graph datasets, using official Github implementations with default hyperparameters for the baselines. The results are as follows:


|                 | Actor          | Squirrel        | Chameleon      |
| --------------- | -------------- | --------------- | -------------- |
| GCN             | 28.92 $\pm$ 1.67 | 57.21 $\pm$ 1.66  | 62.12 $\pm$ 1.74 |
| NodeFormer      | 29.93 $\pm$ 1.78 | 49.12  $\pm$ 1.22 | 58.73 $\pm$ 1.84 |
| InterpGCN-GW    | 31.57 $\pm$ 1.29 | 60.12  $\pm$ 1.31 | 64.27 $\pm$ 1.96 |
| GPRGNN [13]          | 33.32 $\pm$ 1.71 | 46.20 $\pm$ 0.76  | 51.71 $\pm$ 2.49 |
| GGCN [12]            | 36.79 $\pm$ 1.97 | 52.19 $\pm$ 1.41  | 66.19 $\pm$ 2.73 |
| H2GCN [10]          | 35.21 $\pm$ 2.45 | 40.79 $\pm$ 1.42  | 60.13 $\pm$ 1.78 |
| InterpGPRGNN-GW | 36.55 $\pm$ 1.79 | 45.73 $\pm$ 1.15  | 52.19 $\pm$ 2.05 |
| InterpGGCN-GW   | 37.81 $\pm$ 2.17 | 55.49 $\pm$ 1.92  | 67.17 $\pm $1.54 |
| InterpH2GCN-GW  | 35.42 $\pm$ 1.79 | 41.46 $\pm$ 1.83  | 59.79$ \pm$ 1.97 |

Reviewer comment (Theory):
In the ‘Theorem 1 interpret the structure imbalance phenomenon.’ part, authors claim that stronger interplay with training nodes brings small generalization error, and nodes with smaller degrees engage in fewer interactions with training nodes thus having the trend in Fig2. However, it seems like a mistake according to the authors' theory. Nodes with smaller degrees engage in fewer interactions with training nodes should have bigger generalization error, while in fig2 nodes with fewer degrees have better performance.

Author response:
**R1**: We apologize for any ambiguity in Figure 2. The x-axis represents group indexes, not node degrees. Nodes are sorted by degree and divided into five groups, with Group 1 (lowest index) having the highest node degrees and Group 5 (highest index) the lowest.

Reviewer comment (Experiments):
The Experiment part mainly concentrates on node classification performance and ablation study. It would be better to design experiments to show how InterpGNN-GW increases the interplay between training and test nodes, which also strengthens the claims in theory and interprets the structure imbalance phenomenon.

Author response:
**R2**:  To demonstrate how InterpGCN-GW enhances the interplay between training and test nodes, we present a specific example with a 3-layer GCN.  We select a training set of $N_s$ nodes and $N_n$ test nodes isolated from the training set, where the initial L-hop interplay is 0. InterpGCN-GW, through attention mechanism, stores information about important nodes, essentially holding a mix of various node embeddings in  memory. For analytical purposes, we assume that the memory contains the full features of $N_m$ training nodes, which can be broadcasted to all test nodes. This effectively transforms all test nodes into 1-hop neighbors of training nodes of training nodes, connected to at least $N_m$ training nodes. Consequently, the interplay increases to at least $\frac{6}{N_m}$, leading in a smaller generalization error bound.

Reviewer comment (Experiments):
It would be more convincing if authors could provide full ablation study results on all datasets instead of separating ablated models to Tab 5 and Fig 6 and using different datasets.

Author response:
**R4**: We revise the paper to include a unified ablation study on 2 small graphs and all 3 large graph datasets.

Reviewer comment (Experiments):
Can InterpGNN-GW be used in a non-rigorous inductive setting? If yes, what about performance? For example, train InterpGNN-GW on the training nodes and broadcast the workspace to test nodes with test nodes still unseen to the training process. It should be similar to interplaying among out-of-batch nodes mentioned in the intuition part.

Author response:
**R5**: Thank you for your valuable insight, which has highlighted an advantage of InterpGCN-GW which we did not noticed. Unlike other methods that experience significant performance drops in inductive settings compared to transductive settings, InterpGCN-GW shows only a minor decrease in performance in non-rigorous inductive settings, closely aligning with its transductive results. In scenarios where test node features are unseen during training, the performance of InterpGCN-GW is as follows:

| Model         | Amazon2M         | Aminer-CS        | Reddit  |
|---------------|------------------|------------------|------------------|
| ClusterGCN    | 83.85 $\pm$ 0.27 | 61.59 $\pm$ 0.37 | 89.77 $\pm$ 0.35 |
| GRAND+        | 84.12 $\pm$ 0.18 | 62.48 $\pm$ 0.36 | 92.05 $\pm$ 0.23 |
| NodeFormer   | 84.25 $\pm$ 0.41 | 62.12 $\pm$ 0.24 | 90.75 $\pm$ 0.25 |
| InterpGCN-GW  | 88.13 $\pm$ 0.24 | 67.85 $\pm$ 0.19 | 92.79 $\pm$ 0.19 |

Reviewer comment (Evaluation):
Why does InterpGNN-GW fail to outperform some baselines on CoraFull, any in-depth or intuitive analysis?

Author response:
**R6**: InterpGCN-GW only fail to outperform GRAND+ with small difference on CoraFull dataset. This can be analyzed based on several dataset-specific factors:

**High Feature Dimensionality**: CoraFull has a feature dimensionality of 8710, far exceeding that of datasets like WikiCS (300), Computer (767), Reddit (602), and Amazon2M (100).

**Feature Sparsity**: The dataset's features are one-hot vectors and highly sparse. This, combined with the high dimensionality, predisposes models like InterpGCN-GW, which employ attention mechanisms, to potential overfitting.

**Method of Sparsification**: GRAND+, in contrast, uses Topk sparsification, effectively mitigating overfitting in high-dimensional, sparse scenarios.

**Heterophily of Corafull**: CoraFull tends more towards heterophily, with lower proportions of connected nodes sharing similar labels compared to other datasets.