Paper: Non-backtracking Graph Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the introduced techniques is limited. The authors do not sufficiently discuss closely related works:  
- Non-backtracking concepts in walk-based graph learning: The problem of backtracking has been investigated for random walk kernels, see:  
   * Pierre Mahé, Nobuhisa Ueda, Tatsuya Akutsu, Jean-Luc Perret, Jean-Philippe Vert: Extensions of marginalized graph kernels. ICML 2004  
   * Furqan Aziz, Richard C. Wilson, Edwin R. Hancock: Backtrackless Walks on a Graph. IEEE Trans. Neural Networks Learn. Syst. 24(6): 977-989 (2013)  
  The first paper by Mahé et al. proposes a transformation of an undirected input graph to a directed graph, where each undirected edge is represented by two nodes reflecting the two ways of traversing the edge. These nodes are connected such that walks with backtracking are not possible. The construction and the idea are highly similar to the method described in the paper under review.  
- The paper "Zhengdao Chen, Lisha Li, Joan Bruna: Supervised Community Detection with Line Graph Neural Networks. ICLR 2019" (cited but not sufficiently acknowledged) introduces a similar idea of avoiding backtracking to GNNs. A GNN implicitly performing message-passing on a directed line graph is proposed, conceptually highly similar to the technique proposed in the paper under review. Moreover, its strength for learning on graphs generated via the stochastic block model is investigated and explored.  
- The paper "Rongqin Chen, Shenghui Zhang, Leong Hou U, Ye Li: Redundancy-Free Message Passing for Graph Neural Networks. NeurIPS 2022" is closely related to the proposed method but only cited among others on page 4 and not sufficiently acknowledged. The approach uses simple paths (and cycles), allowing no repeated vertices at all instead of no repetition in subpaths of length two as the non-backtracking approach. Moreover, it investigates the link to oversquashing via the same techniques based on the Jacobian. A more detailed discussion of the differences compared to this work is necessary.

Author response:
We appreciate your notification and explanation about closely related works. We have discussed them in the general response 1, related works section, and added the detailed difference in Appendix A, as mentioned in general response 1. We clarify the distinction as follows:

**Mahé et al**.: We note that the construction of the graph in Mahé uses both node and edge representation, while **ours only uses edge-wise representation for non-backtracking**. Moreover, CINs [1, 2] could be considered an extent of Mahe et al., with message-passing between different dimension cell complexes.

**LGNN (Chen et al. `19)**: A notable distinction between NBA-GNN and LGNN is that LGNN employs the non-backtracking operator inside a layer. When layers are stacked, redundancy is not addressed at all, and consequently cannot alleviate over-squashing. However, **NBA-GNN keeps the non-backtracking characteristic across all layers to reduce redundant messages** and theoretically shows that the sensitivity upper bound is bigger.

**RFGNN (Chen et al. `22)**: We agree that the motivation is very similar. However, while RFGNN constructs a tree of non-backtracking paths from all nodes, NBA-GNN **considers non-backtracking by edge adjacency** through all layers. Thanks to this, while RFGNN suffers from a space and time complexity of $\mathcal{O} (\vert V \vert^{k+1})$, **our NBA-GNN only requires $d_{avg} \vert\mathcal{E}\vert$, irrelevant to the number of layers $k$**. This allows our NBA-GNN scalable to larger graph datasets, compared to RFGNN. Furthermore, in terms of theory, our work differs by **stating the sensitivity upper bound of non-backtracking update**, not comparing the relative inference of paths.

[1] Bodnar et al., Weisfeiler and Lehman Go Cellular: CW Networks

[2] Giusti et al., CIN++: Enhancing topological message passing

---

As we mentioned in global response 1, LGNN was the first to use non-backtracking and we have updated the manuscript accordingly. Although NBA-GNN and Mahé et al share the common idea of non-backtracking, it differs in many details. 

i) **Node set of transformed graph**: Figure 2 of Mahé et al. states that the transformed graph is defined as **$G=(V’, E’)$ where $V’=V\cup E$**, from an input graph $G=(V, E)$. However, our NBA-GNN computation graph can be written as **$G’=(V’, E’)$ where $V’=E$**.

ii) **Initial representation**: The random walk in Mahé et al must start from a node in the original graph, i.e., V, while the update process of **NBA-GNN only considers edge-wise representations**. Moreover, the edge $(i, j)$ only receives information from node $i$ in Mahé et al, while the edge representation of NBA-GNN is built by using both node $i, j$.

iii) **Node-wise aggregation**: NBA-GNN **aggregates the set of incoming and outgoing edges for a node representation**, for example, node-wise prediction in the VOC-SP dataset. In Mahé et al, I may have missed the content of node-wise aggregation. But based on IV) of Figure 2, it seems to **only consider random walks starting from a node**, not random walks ending at a node. We would be happy to provide more details if the reviewer could point out the section node-wise aggregation is described.

---

As we highlighted in global response 2, our contribution lies in being the first **to compare the degree of over-squashing between GNNs, aligning with different random walks**. While previous works have used metrics such as curvature$^{[1]}$ or the effective resistance$^{[2]}$ to assess the graph topology in the sensitivity upper bound, we have interpreted it using random walks. Furthermore, demonstrating the tightness of sensitivity bounds requires unrealistic strong assumptions, e.g., there always exist parameters that set the sensitivity term to zero.

[1] Topping et al., Understanding over-squashing and bottlenecks on graphs via curvature, ICLR 2022

[2] Black et al, Understanding over-squashing in GNNs through the Lens of Effective Resistance, ICML 2023

Reviewer comment (Theory):
Analysis of expressive power: The section argues that spectral analysis of GNNs overcomes the issues of Wesifeiler-Leman-based expressivity results. I cannot follow the reasoning. While focusing on spectral analysis allows to draw from existing results on non-backtracking walks, I have difficulties understanding Theorems 2 and 3. What exactly is the learning task in Theorem 2? How to interpret "can accurately map from graph $\mathcal{G}$ to node labels"? What is the influence of the learnable parameters of the GNN? Most importantly, it is unclear whether standard GNNs with WL expressivity cannot solve these learning tasks, at least theoretically.

Author response:
The learning task of theorem 2 is the node classification. In this theorem, we are discussing the ability of NBA-GNNs that classify each node in one graph, even in the very sparse case.

---

From the WL perspective, NBA-GNN is **as powerful as the WL test** (and other equivalently powerful GNN structures such as GIN) under the assumption that the aggregation and update functions in the layers of NBA-GNN, as specified in equation (3) in our paper, are injective. To illustrate this, consider the scenario where NBA-GNN maps graphs $G$ and $H$ to identical embeddings. In this case, an order of oriented edges $e_i\in E_{G}$, and $d_i\in E_{H}$ exists, satisfying $f(e_i)=f(d_i)$, where $f$ is the non-backtracking message passing update function. This implies that {$j|e_j\in N(e_i)$}={$j|d_j\in N(d_i)$} holds for any given $i$. By the definition of oriented edges, there also exists an order of vertices $u_i\in V_{G}$, and $v_i\in V_{H}$ such that {$j|u_j\in N(u_i)$}={$j|v_j\in N(v_i)$} for all $i$. Consequently, the 1-WL test determines that $G$ and $H$ are isomorphic, establishing that NBA-GNN is equally powerful as the WL test.

However, while we explore the previous works on the expressivity of GNNs, we found that the WL test has some limitations in that it cannot measure the expressive power concerning node classification tasks, and still, it is difficult to compare the proposed algorithms due to the wide performance gap between the 1-WL and 3-WL tests. Therefore, we focus on the spectral analysis in this work to analyze the theoretical performance of NBA-GNNs.

---

Yes, the aforementioned argument remains valid for arbitrary node labels, given that the function from end-node features to edge features is injective. This assertion holds true for our model as we define edge features as the concatenation of end-node features. (For the definition of a message initialization function, please refer to Appendix E.1 in the supplementary material.)

---

When comparing conventional MPNNs with NBA-GNNs in the context of the WL test, we can affirm their equivalence to the power of the WL test, but determining superiority is challenging under the assumption of injective aggregation and update functions. This challenge arises from the notable performance gap between the 1-WL and 3-WL tests, where many algorithms fall within this range.

Therefore, instead of comparison with the WL test, we analyze their performance through spectral analysis to demonstrate the superiority of NBA-GNNs. From a spectral perspective, **GNNs function as spectral filters** $^{[3]}$. However, in a very sparse regime, the spectrum of conventional MPNNs—specifically, the eigenvalues and eigenvectors of the adjacency matrix—lacks a spectral separation property, resulting in **the loss of hidden structure information**. In contrast, NBA-GNNs, with the non-backtracking matrix retaining a spectral separation property, **allow for the recovery of hidden structures**.

[3] Balcilar, Muhammet, et al. "Analyzing the expressive power of graph neural networks in a spectral perspective." Proceedings of the International Conference on Learning Representations (ICLR). 2021.

Reviewer comment (Evaluation):
The space and time complexity increases compared to standard GNNs.

Author response:
Yes, the space and time complexity increases compared to the backbone GNN. However, **NBA-GNN achieves superiority over complexity compared to previous work that alleviates over-squashing**. We provide the preprocessing time, test time per epoch, and memory usage below. 

Table 1 - Theoretical & actual time complexity for preprocessing($b$: branching factor, $k$: number of layers, and $d_{avg}$: average degree of nodes)

|                             | PathNN-SP             | DRew              | RFGNN                      | NBA-GNN                         |
|-----------------------------|-----------------------|-------------------|----------------------------|---------------------------------|
| Theoretical time complexity |$ \|V\| * b^k$           | $\|E\|\sim\|V\|*\|E\|$ | $\|V\|! / \|V-k-1\|!$        | $d_{avg}\|E\|$                          |
| Actual time                 | 666s                  | 123s              | N/A                        | 68s                             |
| Content                     | singles shortest path | $k$-hop indices     | $k$-depth non-redundant tree | Non-backtracking edge adjacency |

Table 2 - Average test time per epoch (Tested on a single RTX 3090, the batch size inside the parentheses)

| Model          | peptides-func | peptides-struct | VOC-SP     |
|----------------|---------------|-----------------|------------|
| PathNN-SP      | 3.349 (64)    | 2.120 (64)      | N/A        |
| PathNN-SP+     | 5.156 (64)    | 1.417 (64)      | N/A        |
| PathNN-AP      | 5.977 (64)    | 1.372 (64)      | N/A        |
| GraphGPS+LapPE | 0.639 (128)   | 0.926 (128)     | 1.307 (32) |
| DRew-GCN+LapPE | 0.933 (128)   | 1.053 (128)     | 6.468 (20) |
| NBA-GCN+LapPE  | 0.541 (200)   | 0.532 (200)     | 5.866 (30) |

Table 3: Memory usage (Single RTX 3090, the batch size inside the parentheses)

| Model          | peptides-func | peptides-struct | VOC-SP      |
|----------------|---------------|-----------------|-------------|
| GCN            | 8.5% (128)    | 13.6% (128)     | 20.12% (32) |
| PathNN-SP      | OOM (128)     | OOM (128)       | N/A         |
| PathNN-SP+     | OOM (128)     | OOM (128)       | N/A         |
| PathNN-AP      | OOM (128)     | OOM (128)       | N/A         |
| GraphGPS+LapPE | 89.7% (128)   | 88.08% (128)    | 45.62% (32) |
| DRew-GCN+LapPE | 46.82% (128)  | 46.82% (128)    | OOM (32)    |
| NBA-GCN+LapPE  | 18.86% (128)  | 16.98% (128)    | 47.30% (32) |

Reviewer comment (Evaluation):
Experimental evaluation:  
- The reported results in the tables are the maximum of two variants, which is slightly unfair. Please report the results separately.

Author response:
Yes, the NBA results presented in Table 1 represent the maximum of non-backtracking and begrudgingly walks. While we have reported this in accordance with the prior work DRew [1], it is worth noting that this approach may be perceived as unfair. We have provided **an ablation study in Figure 5(c) for a comparison between two variants** on the Peptides-func dataset. In response to the reviewer's request, we have also reported the variants in the peptides-struct dataset (given the absence of dangling nodes in the VOC-SP dataset). Despite these considerations, our results still demonstrate improvements compared to the baselines.

[1] Gutteridge et al., DRew: Dynamically Rewired Message Passing with Delay, ICML 2023

Reviewer comment (Presentation):
Minor remark — The caption of Figure 3 contains several repetitions.

Author response:
Thank you for your careful review. We have modified Figure 3 and its caption to make it clear.

Reviewer comment (Theory):
How does the expressivity of NBA-GNNs (e.g., NBA-GIN) compare to GIN?

Author response:
Building on a similar argument as discussed in the response to **W2-2**, if we assume that the aggregation and update functions in the layers of NBA-GNN, as specified in equation (3) in our paper, are injective, we can infer that NBA-GNN is at least as powerful as GIN or NBA-GIN.

Reviewer comment (Theory):
Can the approach be extended to avoid backtracking within $k$ hops?

Author response:
No, NBA-GNN considers redundancy by two consecutive steps in a random walk. Efficiently resolving redundancy occurred by cycles is an interesting future work, since the VOC-SP dataset contains a massive number of cycles.

Reviewer comment (Theory):
What is the intuition as to why begrudgingly backtracking should work better? Does this mean that redundancy is not inherently problematic?

Author response:
The underlying idea is to address the information loss caused by the excessive reduction of redundancy, particularly in proximity to dangling nodes—those with only one neighbor. When a message reaches a dangling node, it becomes unable to propagate information to other nodes, leading to information loss. To mitigate this, we employ begrudgingly backtracking.

Reviewer comment (Experiments):
LapPE enhances the performance for the long-range task. Are there results for GIN/GCN with LapPE?

Author response:
Yes, there are. As one can see in Table 1 in the paper, we have provided the results for GCN, GIN with and without LapPE, and LapPE. As the reviewer assumed, the LapPE indeed enhances the performance for long-range tasks.

Reviewer comment (Experiments):
Comparison to seemingly closely related previous work appears to be lacking (see Question 1). There appears to be previous work proposing the use of non-backtracking operators in GNNs, also investigating their model in the context of stochastic blockmodels [1]. I believe it to be pivotal for you to firstly, discuss the differences between their proposed Line Graph Neural Networks and your NBA-GNNs and to secondly, include their LGNN in your experimental baselines to demonstrate whether/which empirical differences exist.

Author response:
We appreciate your notification about the overlooked work and incorporated it in our general response 1, section 2, and appendix A. While LGNN is indeed a significant prior work first using the non-backtracking operator in the graph, it is essential to emphasize that it **does not address the over-squashing issue**. 

In contrast, our **NBA-GNN efficiently resolves redundant messages through non-backtracking, as theoretically demonstrated**. The experimental results outlined in the general response 1 underscore the suboptimal performance and high complexity of LGNN in real-world datasets. Therefore, in comparison, our primary contribution lies in **bridging the non-backtracking operator with the over-squashing problem, additionally offering superior space and time scalability compared to previous works alleviating over-squashing**.

Reviewer comment (Experiments):
NBA-GNNs are prohibitively expensive and the additional expense in terms of computation time is insufficiently explored (see Question 2). The NBA-GNNs you propose are rather expensive in the sense that you learn two embeddings per edge. While it is very good, that you have included a discussion of the additional memory cost in the "Limitations" paragraph, I also believe a discussion of the time complexity of your method to be necessary. Ideally you should evaluate both the time complexity in theory and also provide experimental evaluation of the computation time of your NBA-GNNs compared to the baseline methods.

Author response:
Yes, undoubtedly there is an increase in time complexity compared to GNNs. However, we emphasize our method is superior to other previous works alleviating over-squashing, on average test time per epoch and preprocessing times. From the two tables below, one can easily conclude that our method does not face bottlenecks in time compared to other baselines, **irrelevant to the number of layers**.

Table 1 - Theoretical & actual time complexity for preprocessing($b$: branching factor, $k$: number of layers, and $d_{avg}$: average degree of nodes)

|                             | PathNN-SP             | DRew              | RFGNN                      | NBA-GNN                         |
|-----------------------------|-----------------------|-------------------|----------------------------|---------------------------------|
| Theoretical time complexity |$ \|V\| * b^k$           | $\|E\|\sim\|V\|*\|E\|$ | $\|V\|! / \|V-k-1\|!$        | $d_{avg}\|E\|$                          |
| Actual time                 | 666s                  | 123s              | N/A                        | 68s                             |
| Content                     | singles shortest path | $k$-hop indices     | $k$-depth non-redundant tree | Non-backtracking edge adjacency |

Table 2 - Average test time per epoch (Tested on a single RTX 3090, the batch size inside the parentheses)

| Model          | peptides-func | peptides-struct | VOC-SP     |
|----------------|---------------|-----------------|------------|
| PathNN-SP      | 3.349 (64)    | 2.120 (64)      | N/A        |
| PathNN-SP+     | 5.156 (64)    | 1.417 (64)      | N/A        |
| PathNN-AP      | 5.977 (64)    | 1.372 (64)      | N/A        |
| GraphGPS+LapPE | 0.639 (128)   | 0.926 (128)     | 1.307 (32) |
| DRew-GCN+LapPE | 0.933 (128)   | 1.053 (128)     | 6.468 (20) |
| NBA-GCN+LapPE  | 0.541 (200)   | 0.532 (200)     | 5.866 (30) |

Reviewer comment (Theory):
The theoretical result in Theorem 1 is only a weak indication of alleviated over-smoothing (see Question 3). Your result in Theorem 1 appears to be of limited importance to me. The fact that your upper bound on NBA-GNNs is larger than the bound on standard GNNs could either mean that one of the bounded quantities is indeed larger than the other as you suggest, but it could equally well be the case that one of the two bounds is looser than the other in practice, i.e., given the bound on standard GNNs any larger upper bound is trivially also true and would surpass your larger bound, which might put the conclusions you drew from the magnitude of the two upper bounds in jeopardy. It would significantly strengthen your result if you could observe (even if just experimentally) that the considered derivatives are indeed larger for the non-backtracking operator than for the normalised adjacency matrix.

Author response:
We acknowledge that our upper bound may not be sufficiently tight as we discussed in the general response 2. However, our theorem is **the first to compare the quantity of over-squashing in GNNs, aligning with different random walks**.

We understand your reservations about the strength of our theoretical results despite our assertion of being the first to compare the degree of over-squashing between GNNs using different random walks. We would like to provide additional insights into the unique contributions of our work:

**1. Novelty in Comparison Methodology**: Our primary contribution lies in introducing a novel approach to compare the degree of over-squashing among GNNs. By aligning with different random walks, we believe we provide a fresh perspective that adds to the existing body of knowledge.

**2. Interpretation through Random Walks**: While previous works have utilized metrics like curvature or effective resistance to assess graph topology in the sensitivity upper bound, we've taken a distinctive path by interpreting it through the lens of random walks. This allows for a unique and complementary understanding of the graph structure.

**3. Realism in Sensitivity Bounds**: We acknowledge your point about the challenges of demonstrating the tightness of sensitivity bounds. Our intention is not to set unrealistic assumptions, but rather to provide a theoretical framework that is more aligned with real-world scenarios, where the sensitivity term may not be consistently set to zero.

We would greatly appreciate any specific feedback or suggestions you may have regarding aspects of the theoretical results that could be strengthened or clarified. Your insights are crucial to us, and we are dedicated to addressing any concerns to the best of our ability.

Once again, thank you for your thoughtful evaluation, and we look forward to the possibility of further discussion to enhance the clarity and robustness of our theoretical contributions.

Reviewer comment (Reproducibility):
The paper lacks a detailed description of the construction of the non-backtracking operator/walk/update and the related implementation in NBA-GNN.

Author response:
**W1** The paper lacks a detailed description of the non-backtracking operator

We discussed the non-backtracking operator in the sensitivity analysis, section 4.1. However, as the reviewer suggested, providing a detailed description of using the non-backtracking operator in NBA-GNNs can enhance comprehension. To address this, we added a detailed description in Appendix E regarding the application of the non-backtracking operator in NBA-GNNs.

Reviewer comment (Evaluation):
The time complexity of processing the non-backtracking seems high, and the preprocessing time is not reported. Additionally, the run time and memory usage of NBA-GNN compared with other GNNs is not reported, making it difficult to evaluate the proposed method comprehensively.

Author response:
**W2** Preprocessing time, run time, and memory usage

We thank reviewer $\textcolor{red}{WPCH}$ for suggesting a fair comparison with baselines, considering the complexity of our method. Our method achieves superior results in all three terms, preprocessing time, run time, and memory usage compared to previous works suggested to alleviate over-squashing.

- **Preprocessing time**

We compared the preprocessing time of the peptides-functional dataset. Our preprocessing time is approximately half of DRew, and 1/10 of PathNN-SP as shown in the table below. It is also worth noting that the preprocessing time of previous methods is proportional to the number of layers, **NBA-GNN is irrelevant to the number of layers**.

Table 1 - Theoretical & actual time complexity for preprocessing($b$: branching factor, $k$: number of layers, and $d_{avg}$: average degree of nodes)
|                             | PathNN-SP             | DRew              | RFGNN                      | NBA-GNN                         |
|-----------------------------|-----------------------|-------------------|----------------------------|---------------------------------|
| Theoretical time complexity |$ \|V\| * b^k$           | $\|E\|\sim\|V\|*\|E\|$ | $\|V\|! / \|V-k-1\|!$        | $d_{avg}\|E\|$                          |
| Actual time                 | 666s                  | 123s              | N/A                        | 68s                             |
| Content                     | singles shortest path | $k$-hop indices     | $k$-depth non-redundant tree | Non-backtracking edge adjacency |

- **Running time, memory usage**

The average test time per epoch and memory usage have been measured, by reducing the batch size of previous methods except GCN since the paper’s parameters did not fit on a single RTX 3090. We use 8 layers, a hidden dimension of 120 for all experiments, and have written the batch size inside the parentheses.

Table 2 - Average test time per epoch (Tested on a single RTX 3090)

| Model          | peptides-func | peptides-struct | VOC-SP     |
|----------------|---------------|-----------------|------------|
| PathNN-SP      | 3.349 (64)    | 2.120 (64)      | N/A        |
| PathNN-SP+     | 5.156 (64)    | 1.417 (64)      | N/A        |
| PathNN-AP      | 5.977 (64)    | 1.372 (64)      | N/A        |
| GraphGPS+LapPE | 0.639 (128)   | 0.926 (128)     | 1.307 (32) |
| DRew-GCN+LapPE | 0.933 (128)   | 1.053 (128)     | 6.468 (20) |
| NBA-GCN+LapPE  | 0.541 (200)   | 0.532 (200)     | 5.866 (30) |

Table 3 - Memory usage (Single RTX 3090)

| Model          | peptides-func | peptides-struct | VOC-SP      |
|----------------|---------------|-----------------|-------------|
| GCN            | 8.5% (128)    | 13.6% (128)     | 20.12% (32) |
| PathNN-SP      | OOM (128)     | OOM (128)       | N/A         |
| PathNN-SP+     | OOM (128)     | OOM (128)       | N/A         |
| PathNN-AP      | OOM (128)     | OOM (128)       | N/A         |
| GraphGPS+LapPE | 89.7% (128)   | 88.08% (128)    | 45.62% (32) |
| DRew-GCN+LapPE | 46.82% (128)  | 46.82% (128)    | OOM (32)    |
| NBA-GCN+LapPE  | 18.86% (128)  | 16.98% (128)    | 47.30% (32) |

Reviewer comment (Experiments):
What is the performance of NBA-GNN on the other two datasets in LRGB?

Author response:
**Q1** What is the performance of NBA-GNN on the other two datasets in LRGB?

We attempted additional experiments for PCQM and Pascal-SP. However, we note that these datasets are quite large compared to peptides and VOC-SP, with approximately 8 to 10 times more data. Regrettably, we only present the experiment results of NBA-GCN on PCQM, one can still see that the performance does not vary much even without tuning.

Table 4 - PCQM PCQM results without tuning
| PCQM  | Test Hits@1 | Test Hits@3 | Test Hits@10 | Test MRR |
|-------|:-----------:|:-----------:|:------------:|:--------:|
| GCN   |    0.1321   |    0.3791   |    0.8256    |  0.3234  |
| + NBA |    0.1282   |    0.3589   |    0.8186    |  0.3139  |
| GIN   |    0.1337   |    0.3642   |    0.8147    |  0.3180  |
| + NBA |    0.1349   |    0.3754   |    0.8347    |  0.3243  |

Reviewer comment (Novelty):
The paper misses important related work, specifically: Zhengdao Chen, Lisha Li, Joan Bruna. Supervised Community Detection with Line Graph Neural Networks. ICLR 2019. This paper was the first to propose the use of the non-backtracking operator in GNNs.

Author response:
Thank you for pointing out a significant related work. To incorporate your comment, we conceptually and empirically compare our work and the suggested work (in our General Response 1, Sections 2, and Appendix A). In comparison to the reference (Chen et al., ICLR 2019), our contribution is **the consideration of the non-backtracking operator specifically for the message redundancy problem**, rather than proposing a novel/first non-backtracking architecture in GNNs.

Reviewer comment (Novelty):
In light of the above, the proposed architecture is somewhat incremental.

Author response:
Thank you for pointing out a significant related work. To incorporate your comment, we conceptually and empirically compare our work and the suggested work (in our General Response 1, Sections 2, and Appendix A). In comparison to the reference (Chen et al., ICLR 2019), our contribution is **the consideration of the non-backtracking operator specifically for the message redundancy problem**, rather than proposing a novel/first non-backtracking architecture in GNNs.

Reviewer comment (Theory):
The theoretical results are not convincing.

Author response:
Yes, proposition 1 only holds for trees. However, we note that this proposition is only a motivating example to use non-backtracking, not a theoretical claim that NBA-GNN can mitigate over-squashing. We point out additional references [1,2] that investigate faster access time of non-backtracking walks compared to conventional random walks. 

[1] Yuan Lin and Zhongzhi Zhang. Non-backtracking centrality based random walk on networks. The Computer Journal, 62(1):63–80, 2019.

[2] Dario Fasino, Arianna Tonetto, and Francesco Tudisco. Hitting times for second-order random walks. European Journal of Applied Mathematics, 34(4):642–666, 2023.

---

Yes, theorem 1 only provides the upper bound for the sensitivity analysis. However, as mentioned in the general response 2, we would like to emphasize that **the contribution of Theorem 1 is being the first to compare the degree of over-squashing between GNNs, aligning with different random walks**. 

---

Thank you for the insightful comment. Indeed, our analysis of the regular graphs is more meaningful for graphs with a small degree. Nonetheless, we believe this result to be meaningful since many real-world graphs, e.g., community and molecular graphs, are sparse. Also, we believe that **one cannot compare the performance improvements in different datasets in an apple-to-apple way**, e.g., VOC-SP performance may be more susceptible to over-squashing in GNNs compared to the peptides-func dataset. 

---

While one might perceive it that way, our analysis is novel since **it integrates earlier works on the non-backtracking matrix into GNNs**. This integration has the potential to pave the way for advancements in the field, suggesting new perspectives that could lead to innovative insights and methodologies within the GNN framework.

Reviewer comment (Theory):
The claim that NBA-GNNs might help with oversquashing is supported by the assumption, backed only by empirical evidence, that NBA-GNNs have shorter access time than BA-GNNs. A proposition is provided stating that non-backtracking random walks have shorter access times, but it only holds for trees. The conditions of this proposition are too far away from the setup of NBA-GNNs to make a convincing claim.

Author response:
Yes, proposition 1 only holds for trees. However, we note that this proposition is only a motivating example to use non-backtracking, not a theoretical claim that NBA-GNN can mitigate over-squashing. We point out additional references [1,2] that investigate faster access time of non-backtracking walks compared to conventional random walks. 

[1] Yuan Lin and Zhongzhi Zhang. Non-backtracking centrality based random walk on networks. The Computer Journal, 62(1):63–80, 2019.

[2] Dario Fasino, Arianna Tonetto, and Francesco Tudisco. Hitting times for second-order random walks. European Journal of Applied Mathematics, 34(4):642–666, 2023.

Reviewer comment (Theory):
As the authors themselves note, Lemma 1 and Theorem 1 only provide upper bounds on the sensitivity of conventional and NBA-GNNs. It is a stretch to conclude that, because the upper bound for conventional GNNs is lower than the upper bound for NBA-GNNs, the sensitivities behave similarly. I understand that tighter results/lower bounds may not be possible, but considering that this is the main contribution of this paper, it falls somewhat short.

Author response:
Yes, theorem 1 only provides the upper bound for the sensitivity analysis. However, as mentioned in the general response 2, we would like to emphasize that **the contribution of Theorem 1 is being the first to compare the degree of over-squashing between GNNs, aligning with different random walks**.

Reviewer comment (Theory):
The authors do not comment on the fact that for moderate-to-large degree d, the decay rates of the sensitivity upper bounds for conventional and NBA-GNNs will become very close. Moreover, this finding is not in agreement with the empirical findings from Table 1, which show that NBA-GNNs lead to significant performance improvements on dense graphs. This could be regarded as evidence that the sensitivity upper bounds are not very tight.

Author response:
Thank you for the insightful comment. Indeed, our analysis of the regular graphs is more meaningful for graphs with a small degree. Nonetheless, we believe this result to be meaningful since many real-world graphs, e.g., community and molecular graphs, are sparse. Also, we believe that **one cannot compare the performance improvements in different datasets in an apple-to-apple way**, e.g., VOC-SP performance may be more susceptible to over-squashing in GNNs compared to the peptides-func dataset.

Reviewer comment (Novelty):
The second theoretical analysis, from Section 4.2, is not very novel, as it is essentially a restatement of theoretical results from the spectral clustering community.

Author response:
While one might perceive it that way, our analysis is novel since **it integrates earlier works on the non-backtracking matrix into GNNs**. This integration has the potential to pave the way for advancements in the field, suggesting new perspectives that could lead to innovative insights and methodologies within the GNN framework.

Reviewer comment (Novelty):
Important references on the expressivity of GNNs from spectral considerations are missing. See e.g. Kanatsoulis et al., and the work of Ribeiro, A.

Author response:
Thank you once again for bringing the overlooked work to our attention. We will certainly incorporate it into the related works section of our paper. The study by Kanatsoulis and Ribeiro introduces a novel approach to analyzing the expressivity of GNNs, utilizing linear algebraic tools and eigenvalue decomposition of graph operators. Their work showcases that GNNs have the capability to generate distinctive outputs for graphs with varying eigenvalues.

Reviewer comment (Writing):
The paper can be quite wordy, and repeat many of the same observations.

Author response:
We will review the manuscript to ensure that the repetitions are necessary for the overall coherence of the paper.

Reviewer comment (Theory):
While the limitations are briefly discussed, they perhaps deserve a more extensive treatment as the increase in computational complexity is quite high.

Author response:
To incorporate your comment, we provide extensive investigation on the increase in computational complexity incurred by our idea. 

Table 1 - Theoretical & actual time complexity for preprocessing ($b$: branching factor, $k$: number of layers, and $d_{avg}$: average degree of nodes)

|                             | PathNN-SP             | DRew              | RFGNN                      | NBA-GNN                         |
|-----------------------------|-----------------------|-------------------|----------------------------|---------------------------------|
| Theoretical time complexity |$ \|V\| * b^k$           | $\|E\|\sim\|V\|*\|E\|$ | $\|V\|! / \|V-k-1\|!$        | $d_{avg}\|E\|$                          |
| Actual time                 | 666s                  | 123s              | N/A                        | 68s                             |
| Content                     | singles shortest path | $k$-hop indices     | $k$-depth non-redundant tree | Non-backtracking edge adjacency |


Table 2 - Average test time per epoch (Tested on a single RTX 3090, the batch size inside the parentheses)

| Model          | peptides-func | peptides-struct | VOC-SP     |
|----------------|---------------|-----------------|------------|
| PathNN-SP      | 3.349 (64)    | 2.120 (64)      | N/A        |
| PathNN-SP+     | 5.156 (64)    | 1.417 (64)      | N/A        |
| PathNN-AP      | 5.977 (64)    | 1.372 (64)      | N/A        |
| GraphGPS+LapPE | 0.639 (128)   | 0.926 (128)     | 1.307 (32) |
| DRew-GCN+LapPE | 0.933 (128)   | 1.053 (128)     | 6.468 (20) |
| NBA-GCN+LapPE  | 0.541 (200)   | 0.532 (200)     | 5.866 (30) |

One can observe that our NBA-GNN indeed increases the computational cost, but our method still achieves superior time complexity over previous works that alleviate over-squashing.