Paper: Recurrent Distance-Encoding Neural Networks for Graph Representation Learning (ICLR-2024)

Reviewer comment (Theory):
RNN here is strange, the assumption here is that the information flow from K-hop to K-1 hop. but information can also go "outward", from k-2 hop to k-1 hop. RNN is not good to model the graph information.

Author response:
It’s true that each edge in the graph allows “bidirectional” information flow. However, if we focus on a particular node for which we want to generate a representation, it’s natural to categorize the other nodes into “shells” (Figure 1(b)). Nodes that belong to the same shell have the same shortest distance from the target node (considering running a breadth-first search from the target node). Then a recurrent neural network, which runs from the outermost shell to the innermost one, would be a reasonable choice to encode the neighborhood structure.

Besides, our model indeed allows “bidirectional” information flow. For two different target nodes (Node 1 and Node 2), their shells are different, the K-1 hop and K hop neighbors of Node 1 can become the K hop and K-1 hop respectively for Node 2, and the information propagates reversely when we generate the representation for Node 2. Therefore, considering the set of node representations as a whole, we won’t lose any structural information. This argument is supported by our rigorous theoretical analysis of expressiveness

Reviewer comment (Novelty):
I assume all the GNN drawback will also be here in this Model (over smoothing, etc). The model can be regarded as another interpretation of massage passing (even though the weight is different), for the target node, the model will aggregate all the K-hop node information.

Author response:
GRED doesn’t have the drawbacks of MPNN because the computational graphs of GRED and MPNN are different. We have added a straightforward example to Appendix E of the updated PDF to illustrate their difference. To aggregate all nodes in the K-hop neighborhood, MPNN needs to perform message passing for K rounds. As a result,  MPNN’s computational graph for the target node grows exponentially with K because the same neighbor will appear many times in the computational graph. The exponentially growing number of nodes are mixed together by MPNN to generate the representation for the target node, resulting in the known drawbacks (e.g., over-smoothing, over-squashing).  On the contrary, the computational graph of GRED includes each node in the K-hop neighborhood exactly once, and GRED further utilizes LRU to control how signals of distant nodes are propagated toward the target node. Therefore, GRED can more effectively leverage information beyond the local neighborhood than MPNN.

Reviewer comment (Experiments):
how the performance like if we change LRU to the vanilla RNN. seems LRU for the long range information pass is the key factor to win against MPNN.

Author response:
We have conducted experiments to compare LRU with vanilla RNN:

| ~   |  CIFAR10 $\uparrow$ | ZINC $\downarrow$ | Peptides-func $\uparrow$ |
| --- | --- | --- | --- |
|    Best MPNN      | 67.312$\pm$0.311 | 0.188$\pm$0.004 | 0.6069$\pm$0.0035   |
| GRED$_{\text{RNN}}$   | 69.215$\pm$0.080 | 0.160$\pm$0.005 | 0.4945$\pm$0.0024 |
| GRED$_{\text{LRU}}$   | **75.370$\pm$0.621** | **0.089$\pm$0.004** | **0.7041$\pm$0.0049** |

We replace the LRU component of GRED with a vanilla RNN:

\begin{equation}
    \mathbf{s}_{v, k}^{(l)} = \text{tanh} ( \mathbf{W}\_{\text{rec}} \mathbf{s}\_{v, k - 1}^{(l)} + \mathbf{W}\_{\text{in}}\mathbf{x}\_{v, K - k}^{(l)})
\end{equation}

where $\mathbf{W}\_{\text{rec}} \in \mathbb{R}^{d_s \times d_s}$ and $\mathbf{W}\_{\text{in}} \in \mathbb{R}^{d_s \times d}$ are two trainable real weight matrices. We use the same number of layers and the same $K$ for both models. We can observe that the performance of GRED with a vanilla RNN drops significantly. On CIFAR10 and ZINC where $K$ is not large, GRED$\_{\text{RNN}}$ still outperforms the best MPNN, but on Peptides-func where long-range information is needed (8 layers with $K=40$ per layer), the performance of GRED$\_{\text{RNN}}$ is worse than the best MPNN because vanilla RNNs are known to be difficult to train on long sequences, unlike the LRU.

Reviewer comment (Novelty):
Firstly, the novelty of the proposed architecture is somewhat limited, as there are already several similar approaches in the field that operate on K-hop neighborhoods in a similar manner. While the combination of permutation-invariant neural networks and linear recurrent networks is a sensible choice, it may not present a significant departure from existing methods. Specifically, the proposed method is very similar to [1], which proposed the following update rule: $h_u^{(t+1)} =COM(h_u^t, AGG_{u,1},..., AGG_{u,k} )$. The main difference is that the proposed approach uses an RNN for the $COM$ function. Moreover, there is no proper discussion of other k-hop approaches such as [2].

Author response:
We thank reviewer 8Mos for taking the time to review our work and are pleased she/he found: 1) our presentation to be very good | clear; 2) our theory to be interesting | rigorous; and 3) our empirical results to be well-organized and contributing to a comprehensive understanding of GRED's performance. 

We admit that we missed several references on K-hop GNNs, and we agree that a more detailed discussion of these methods would help improve the understanding of our contributions. We address these concerns as follows:

**Q1. The novelty of the proposed architecture is somewhat limited | Lack of a comparative analysis of existing K-hop GNNs**.

**Ans**: Compared with existing K-hop GNNs, the novelty of our proposed architecture is a new way to encode the K-hop neighborhood, i.e., with LRU. **The use of LRU ensures our model theoretically sound expressiveness as well as ability to address the over-squashing problem in long-range reasoning tasks**. To see why the novelty of our model is significant, we provide a comparative analysis of existing K-hop GNNs here:

Among K-hop GNNs, MixHop [3] uses powers of the adjacency matrix to access k-hop nodes. k-hop GNN [2] iteratively applies MLPs to combine two consecutive hops and propagates information towards the target node. **While they have shown that higher hop information can improve the expressiveness of MPNNs, they suffer from over-squashing**, i.e., they cannot effectively utilize information from distant nodes because the signals of distant nodes would decay rapidly due to iterative neighborhood mixing. As a result, the largest neighborhood these approaches operate on is 3-hop. The drawback of these approaches is also pointed out by the SPN paper [1]. The SPN paper [1] proposes a conceptual update formula which is similar to ours:

$$
h\_u^{(t+1)} = \text{COM} ( h\_u^{(t)}, \text{AGG}\_{u, 1}, …, \text{AGG}\_{u, K})
$$

**However, SPN simply uses weighted summation as COM to aggregate K hops. The weighted summation cannot guarantee the expressiveness of the model since it cannot uniquely represent sequences. On the contrary, the LRU component of GRED can achieve expressiveness and long-range modeling capacity simultaneously**: LRU can express an injective mapping of sequences, and at the same time prevent distant information from decaying by learning proper eigenvalues of the transition matrix.
We have updated the related work to include a comparison of K-hop GNNs (marked in red in the updated PDF)

To validate that GRED can encode the K-hop neighborhood more effectively than SPN,
We evaluate GRED on two additional datasets from TUDataset and compare against SPN.
As shown in Part III of “Additional Experimental Results”, GRED can outperform SPN with the same number of hops.

[1] Shortest Path Networks for Graph Property Prediction. Learning on Graphs Conference. 2022

[2] k-hop graph neural networks. Neural Networks. 2020

[3] MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing. ICML. 2019

Reviewer comment (Experiments):
Secondly, the experiments in the paper are conducted on a relatively limited set of datasets, focusing on long-rage benchmarks, and the absence of experiments on benchmark datasets like TUDatasets raises concerns about the generalizability of the proposed model. Expanding the experimental evaluation to a wider range of datasets would strengthen the paper's claims.

Author response:
**Q2. Considering that experiments are conducted on a limited set of datasets, could the authors discuss the generalizability of their proposed architecture to a wider range of datasets, including TUDatasets?**

**Ans**: We evaluated our model on 7 datasets in total. They come from different domains (images, social networks, biology), cover both long-range and non-long-range tasks, and are also widely used by graph transformer baselines we compare against. **Both Reviewer e7jX and Reviewer bs5m agree that our experiments are extensive**. To quote Reviewer e7jX: ‘the evaluation is very thorough’ and ‘the proposed method is benchmarked against a wide range of methods and on multiple benchmarks’, and to quote Reviewer bs5m: “The paper compares the proposed method on multiple benchmarks, and demonstrates its effectiveness against traditional GNN models”.

We further evaluate GRED on two additional datasets from TUDataset and compare against SPN. We follow the experimental setup of SPN, and report the average accuracy and standard deviation of $10$ splits:

|Model |  NCI1          | PROTEINS | 
| --- | --- | --- |
|DGCNN |  76.4$\pm$1.7  | 72.9$\pm$3.5 | 
|DiffPool | 76.9$\pm$1.9 | 73.7$\pm$3.5 |
|ECC      | 76.2$\pm$1.4 | 72.3$\pm$3.4 |
|GIN      | 80.0$\pm$1.4 | 73.3$\pm$4.0 |
|GraphSAGE | 76.0$\pm$1.8 | 73.0$\pm$4.5 |
|SPN [1] ($K=10$) | 78.2$\pm$1.2 | 74.5$\pm$3.2 |
|GRED ($K=10$) | **82.6$\pm$1.4** | **75.0$\pm$2.9** |

The baseline performance is directly copied from the SPN paper. GRED generalizes well to TUDataset and outperforms SPN with the same number of hops.

Reviewer comment (Experiments):
A sensitivity analysis of GRED with respect to the choice of K is missing.

Author response:
**Ans**: This is now in Appendix D of the updated PDF. Please refer to Part I of our “Additional Experimental Results” thread in our general response for a thorough analysis.

Reviewer comment (Experiments):
The training efficiency analysis is conducted against a transformer model only, but it would be useful to understand how this method compares to MPNNs. In particular, each node will have its own sequence of sets of nodes at distance k, so there might not be any shared computation that can be leveraged like in iterative 1-hop message passing methods.

Author response:
**Ans**: It’s true that each node has its own sequence of set representations, but the computations for all nodes in the graph can be done in batches. In the implementation, we use a matrix mask to indicate each node’s $k$-hop neighbors (just like the adjacency matrix indicates each node’s 1-hop neighbors). These masks are computed only once in preprocessing using the shortest path algorithm, and they are used to do aggregation at each hop. Then the input to LRU would be a batch of sequences of length K+1 (including the target node for each sequence). **We have added a code snippet of GRED in Appendix B of the updated PDF, to show how the computation is done**.

To compare the training efficiency of GRED with MPNN, we use the same number of layers but the MPNN counterpart only aggregates 1-hop neighbors at each layer. We measure the average training time per epoch:

| Model | ZINC | CIFAR10 | Peptides-func |
| --- | --- | --- | --- |
| GRED | 4.1s | 27.8s | 158.9s |
| MPNN | 2.8s | 21.7s | 91.6s |

We can observe that on ZINC and CIFAR10, the additional time required to use a larger neighborhood is marginal. On Peptides-func, the difference is greater since GRED uses $K=40$. However, this extra time, which is still less than the time required by GRIT, is acceptable considering the significant performance improvement

Reviewer comment (Experiments):
How would this method perform on non long-range benchmarks? Would it underperform compared to MPNNs who might only need a local receptive field to solve a task?

Author response:
**Ans**: The datasets in Table 1 and Table 2 (ZINC) are from [4], which is a widely used benchmark. These datasets are not particularly long-range. PATTERN and CLUSTER model communities in social networks and all nodes can be reached within 3 hops. MNIST and CIFAR10 are nearest-neighbor graphs of pixels, and just like their normal versions local information is important. Our model performs well on these datasets. We further evaluate our model on two additional datasets from TUDataset (please refer to Part III of our “Additional Experimental Results” thread) and our model also achieves good performance.

Our model can adapt to both long-range and non-long-range tasks by learning proper eigenvalues of the LRU transition matrix that control how fast the filters decay as the distance increases. For a long-range task, the learned eigenvalues are close to 1 which better keeps distant information, while for a non-long-range task, the magnitudes of the learned eigenvalues are smaller (please refer to Figure 3 for an illustration of the eigenvalues after training). Additionally, the choice of $K$ gives our model some flexibility to adjust the receptive field, and we found the optimal $K$ for a non-long-range task lies between 1 and the diameter (Part I of “Additional Experimental Results”)

[4] Benchmarking Graph Neural Networks. JMLR. 2022