Paper: Scalable and Effective Implicit Graph Neural Networks on Large Graphs (ICLR-2024)

Reviewer comment (Reproducibility):
I want to confirm my understanding. In subgraph sampling, the coarse nodes are excluded from being the target nodes but will be included in the sampled subgraph through the top-k Personalized PageRank (PPR) selection step, is that correct?

Author response:
Yes. Coarse nodes will be included in the sampled subgraph through top-k PPR selection.

Reviewer comment (Novelty):
The paper fails to position itself with recent related works. More SOTA papers in mini-batch training strategy should be included for discussion, such as [1-3]. The idea of introducing coarse nodes during the process of mini-batching is relatively straightforward and appears in existing literature such as [3]. Therefore, it hurts the novelty of this work regarding the first contribution without a thorough comparison with the current mini-batching GNN methods that also include coarse nodes. Meanwhile, there is a very relevant and similar work that should be added for comparison [4].

Author response:
First, we would like to emphasize again that our submission mainly focuses on scalable **implicit** GNNs instead of scalable traditional/explicit GNNs. Therefore, for scalable traditional/explicit GNNs, we only cover/discuss some well-known sampling-based works such as GraphSAGE, ClusterGCN, GraphSAINT as in USP [1]. 

For VQ-GNN [2], it uses quantization to scale up GNNs, which is a different direction for improving scalability compared with sampling-based works.

For SMGRL [3], in our humble opinion, it actually does not add coarse nodes for training. It directly trains on different coarse graphs rather than considering both original nodes and coarse graphs for training simultaneously. Additionally SMGRL is for explicit GNNs, which is not our main focus.

For VEQ [4] which is published in Big Data conference 2022, we agree that it is relevant and we apologize that we did not notice as it is not publicly available on Arxiv (conference proceedings are also not available without purchase). 
Comparing VEQ and our work SEIGNN, VEQ utilizes previous equilibrium states as an initialization for speeding up training while our work proposed a new solver to obtain unbiased approximation of equilibrium. For mini-batch training, VEQ has specific designs for update while our work SEIGNN is compatible with different sampling methods as shown in Table 6. Moreover, in terms of accuracy, we observe that SEIGNN is better than VEQ on some datasets such as Flickr, Reddit, Yelp, and ogbn-arxiv. We will include above discussions/comparisons into a later version of our manuscript.

[1] Unbiased Stochastic Proximal Solver for Graph Neural Networks with Equilibrium States, ICLR 2023.

[2] VQ-GNN: A Universal Framework to Scale-up Graph Neural Networks using Vector Quantization, NeurIPS 2021.

[3] SMGRL: Scalable Multi-resolution Graph Representation Learning, arXiv 2022.

[4] Efficient and Scalable Implicit Graph Neural Networks with Virtual Equilibrium, Big Data 2022.

Reviewer comment (Experiments):
For the second contribution of accelerated training with the Neumann solver, it is unclear how the authors constrain $n$ to be sufficiently large during experiments. In other words, does the performance of SEIGNN be affected even if the proposed stochastic solver is not an unbiased estimator? Could the authors elaborate more on this?

Author response:
The reviewer's question is how we set $n$ to be sufficiently large in experiments, to ensure it to be unbiased and what would happen if it is not unbiased with $n$ not sufficiently large.

We believe that it is a misunderstanding of our proof in Appendix A.1 and Proposition 1. $n$ in A.1 is a theoretical concept and not a real value usable in experiments. In experiment, we are always using an algorithm with $n = \infty$. Thus, the proposed stochastic are always ensured to be unbiased (either up to all terms when the assumption holds **OR** up to machine precision when the assumption does not hold).

We have revised the proof in the appendix (highlighted in blue) and clarified that `n` is a theorectical concept and in our empirical experiments $n = \infty$.

Reviewer comment (Reproducibility):
The authors do not provide the codes for reproducibility.

Author response:
For the code implementation, we agree that reproducibility is important for the community and we will definitely make the implementation publicly available upon acceptance of our submission.

Reviewer comment (Experiments):
How does modifying the sizes of the minibatch subgraphs affect the efficacy of SEIGNN's coarse nodes?

Author response:
We conduct the experiments by varying the batch size in a mini-batch subgraph (i.e., the number of target nodes). The results on Flickr are shown below.

| Batch Size | 8192 | 4096 | 2048 | 1024 | 512 | 256 |
| ---------- | -- | -- | -- | -- | -- | -- | 
| Accuracy   | 0.5564 | 0.5560 | 0.5509 | 0.5425 | 0.5078 | 0.4514 | 

We can see that using a very small batch size would lead to bad performance while using the batch size 8192 and 4096 have good accuracy. 
The reason of the bad performance of using a small batch size can be that a small batch size makes the information constrained in a small subgraph, which prohibits the information propagation from distant nodes.

Reviewer comment (Experiments):
How important is using PPR for the coarse node idea? Could SEIGNN generalize to other importance metrics?

Author response:
In our submission, we follow Shadow-GNN [1] to use PPR as an importance metrics to form a subgraph. As pointed out in [1], we can easily extend the approach by using other metrics such as katz index, SimRank [2], and feature similarity, which we leave as future work due to time imitations.

[1] Decoupling the Depth and Scope of Graph Neural Networks. NeurIPS 2021.

[2] Simrank: a measure of structural-context similarity. KDD 2002

Reviewer comment (Experiments):
What would be the effect of removing/keeping the coarse nodes post-training?

Author response:
We compare these two during the testing phase.

On Flickr, in testing, accuracy of removing coarse nodes: 0.5575, keep coarse nodes: 0.5551. On Reddit, in testing, removing coarse nodes: 0.9776, keeping coarse nodes: 0.9778.
We can see that either removing or keeping coarse nodes would have similar performance. Thus, we believe that either way can be fine for inference/test.

Reviewer comment (Experiments):
Why is the stochastic solver slightly better in performance than the Neumann solver in Table 4 of the ablation studies? Is the Neumann solver a truncated version here?

Author response:
Answer: 
Yes. Here, the Neumann solver is a truncated version of the infinite sum (as shown in Eq(7)). The better performance of the stochastic solver compared with the Neumann solver is caused by the unbiased approximation of the equilibrium obtained using the stochastic solver. We theoretically proved that the stochastic solver provides an unbiased approximation in Proposition 1 of our submission. In contrast, using the Neumann solver would incur some approximation errors of the equilibrium in expectation, which leads to slightly worse performance.