From: ICLR.cc/2023/Conference/Paper767/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper767/Area_Chair_C9NK
title: Notice
comment: Dear Reviewer qSsw,

If you do not want to update your score, please at least acknowledge that you have already read the rebuttal.

From: ICLR.cc/2023/Conference/Paper767/Reviewer_mB9y
title: Update after rebuttal.
comment: I thank the authors for considering my feedback on the new version of the paper (I have raised my score on the reviewed version)

From: ICLR.cc/2023/Conference/Paper767/Reviewer_svMx
title: Response to Authors' Feedback
comment: Thanks to the authors for the efforts to improve the manuscript. The revision has resolved part of my concerns, including the issues of I/O efficiency and some typos. However, the feedback has confirmed my worry that the framework is not applicable to GAT, which is a substantial limitation. As pointed out by other reviewers, there is also further space for improvement, like the clearness of figures, uniformity in presentation, and the convincement of experiments. 
I would maintain my original ratings.

From: ICLR.cc/2023/Conference/Paper767/Reviewer_AHA4
title: Response to author feedback
comment: Thanks the authors for the feedback and revision. The overall writing has been improved, but there are still some space for improvement like the other reviewers pointed out, e.g. usage of former/latter modules are not particularly clear especially in an architecture with forward and backward training. Figures are still not clearly described in caption for their purposes and far away from they first mentions. Tables are still not following the same format (some with variance and some without, even in same table). 

What I found more worrying is the experiment results in appendix thanks to the other reviewers mentioning. The proposed method came at a not negligible cost of performance while applying to larger dataset, and I am not convinced that the trade-off is worthwhile. Even further, SGNN seems to suffer more when increasing number of layers in base GNN, the study on it is limited to only one table. 

With that I would lower my score and still not leaning towards accepting the paper.

From: ICLR.cc/2023/Conference/Paper767/Authors
title: Response to Reviewer mB9y and thanks for the constructive suggestions
comment: We are glad that the reviewer could agree with the novelty of our paper. We apologize for the poor writing and careless typos. We have prepared this version and response to address your concerns. If the reviewer still has severe concerns, please let us know so that we can provide a further response to address the concerns.

**Q1**: (About presentation and readability)

**R1**: We have revised the paper carefully. We also try our best to clarify the intention of Eq. (5), which is the key of SGNN. We believe this revision could help readers to follow our idea easily. Sorry again for the previous version. 



**Q2**: The original motivation to the analysis in the paper is to increase efficiency with ...

**R2**: This is indeed an insightful question. The inefficiency of GNNs could not be neglected, especially when the depth grows. However, it should be pointed out that **even when GNNs are shallow, the inefficiency is still distinct**. 

Note that every BP iteration / step to learn parameters of GNNs requires time-consuming graph operations (*e.g.*, graph convolution). It usually needs hundreds or thousands of iterations to converge for BP. Let $t$ be the number of BP iterations. For an $L$-layer GNN, it requires $\mathcal{O}(eL)$ graph operations to train it. Therefore, it is still quite inefficient. 

In SGNN, let $e$ be the number of epochs (elaborated in Algorithm 1). For an SGNN with $L$ base models, it requires only $\mathcal(eL)$ graph operations during training. It benefits from the separability of base GNN models. Note that $t \gg e$ usually holds. In our experiments, $e\approx 10$. 



**Q3**: (The experimental analysis is not fully convincing)

**R3**: We first apologize for the inconsistent descriptions in the appendices. We fail to recheck the appendix in the previous version. We have polished the appendix carefully. 

For each module of SGNN, the settings are the same except for the output dimensions. 

For the model selection, we prefer to follow the schemes of some well-known works (*e.g.*, SGC, S2GC, GCNII) to ensure the fairness and improve reliability.  

To be more convincing, we also uploaded the source code, which will be publicly available after publication.

From: ICLR.cc/2023/Conference/Paper767/Authors
title: Response to Reviewer qSsw and thanks for the constructive suggestions (2/2) 
comment: **Q4**: If my understanding is correct, SGNN achieves good accuracy because it introduces more supervision because of it uses more supervision for each layer.

**R4**: It may be a reason why SGNN obtains better results on some datasets. But the aim of our experiments is to find whether the decoupling skill **seeking for the high efficiency** could approach the original GNN. The main advantage is the significant efficiency. 



**Q5.a**:  GAT seems a strong baseline in the node classification experiment. Why is it not used for node clustering?

**R5.a**: In fact, the unsupervised GAT, GATE [1], is usually not a competitive method in node clustering task. The experimental results are shown as follows. Due to the limitation of space, we omit the GATE. If the additional page is allowed, we will add the corresponding results into Table 1. 

|      | Cora  |       | Citeseer |       | PubMed |       |
| ---- | ----- | ----- | -------- | ----- | ------ | ----- |
|      | ACC   | NMI   | ACC      | NMI   | ACC    | NMI   |
| GATE | 0.658 | 0.527 | 0.616    | 0.401 | 0.673  | 0.322 |
| AGC  | 0.689 | 0.537 | 0.670    | 0.411 | 0.698  | 0.316 |
| SGNN | 0.743 | 0.555 | 0.673    | 0.416 | 0.695  | 0.334 |

[1 ] Amin Salehi and Hasan Davulcu, Graph Attention Auto-Encoders, arXiv: 1905.10715. 



**Q5.b**: The efficiency comparison in Figure 3 is unfair. The authors use the time to train an iteration but different could require different iterations to reach convergence. I suggest the authors directly use the time to reach convergence. 

**R5.b**: Sorry for the ambiguous description. In fact, the total time is the convergence time (for all models). The reason why the total time is divided by the number of iterations is to show that SGNN requires the cheapest cost for each step. 



**Q5.c**: Table 5 in the Appendix shows that SGNN-BP has lower accuracy than the baselines on the OBGN graphs, which makes the results in the main paper look like cherry picking. 

**R5.c**: The experiments on OGB datasets are not precisely fine-tuned. Since the aim of our paper is to find whether the decoupled model could approach the raw model, we report the results with simple settings of hyper-parameters. 



**Q5.d**:  It will be more interesting if the authors can show the accuracy degradation of other models (when increasing the number of layers).  I understand that GNN accuracy degrades because of over-smoothing but it is also possible that training strategy of SGNN does not work for more layers. It is important to validate whether SGNN suffers more penalties under more layers.

**R5.d**: We are working on the results of ClusterGAE with more layers. The results are coming soon. 

We do agree with the reviewer that it should validate whether the decoupling still works when GNNs go deeper. Note that the reported **SGNN is decoupled from the GAE** and we can conclude that SGNN is more robust to the depth than GAE, from Table 6.

From: ICLR.cc/2023/Conference/Paper767/Authors
title: Response to Reviewer qSsw and thanks for the constructive suggestions (1/2)
comment: We greatly thank the reviewer for the appreciation of our novelty and the constructive comments. We admit that there are some ambiguous statements (especially about the fully-separable property) and apologize for it. We now prepare a revised paper to clarify the roles of separability and full separability.  

**Q1.1**: Why is the separable (or fully separable) GNN assumption required? ......

**R1.1**: The inefficiency is caused by the graph operation instead of neural network operation. When training GNNs, each FP (forward propagation) iteration requires graph operations which are extremely expensive from the aspects of both time and memory. The separability, not the full separability, is used to **ensure that the graph operation can be executed only once**. Note that most of the popular GNNs are separable if they are single-layer or the intermediate activation functions are linear.  

The full separability is actually essential for the BT. In the previous version, the explanation of the role of full separability is too ambiguous and we now clarify it in this version. 

In Eq. (5), 
$$
H = \hat f(A, X, W) = \hat f_1(\hat f_0(A, X), W) = \hat f_1 (f^* ( A, X, U), W) = \hat f_1 (g_1^* ( A, g_0^*(X, U)), W), ~ \forall \hat f \in \mathcal{F}_k, f^* \in \mathcal{F}_d^* ,
$$
we actually convert a *separable* GNN into **a composite model composed of a *fully-separable GNN* and *an MLP*** by introducing the learnable parameter $U$. Formally, in Eq. (5), the fully-separable GNN is denoted by $f^*(A, X, U)$ and the MLP is denoted by $\hat f_1(\cdot, W)$. 

Note that during FT, $U$ is fixed as a constant. Let $\hat X = g_0^*(X, U)$ and then Eq. (5) is equivalent to $H = \hat f_1(g_1^*(A, \hat X), W)$. Clearly, **it is equivalent to the original separable single-layer GNN.** In other words, to modify any separable GNNs for BT, the required step is to replace the $X$ by $\hat X = g_0^*(X, U)$. 

In summary, **the decoupling skill proposed in this paper can be applied to any separable GNNs**. Note that most of single-layer GNNs are separable, including but not limited to GIN, S2GC, GNNs with skipping connections(*e.g*., APPNP, JKNet).



**Q1.2**: The second paragraph below Definition 3.1 gives me the feeling that SGNN only conducts graph operation once. However, Figure 1 and the fact SGNN uses SGC as the modules suggest that SGNN conducts graph operation in each layer. (So which is correct?) ...  How does that link to fully separable GNN?

**R1.2**: The statements, about "conduct graph operation only once" in our paper, mean that the graph operation is executed once for **each module** (*i.e.*, each base GNN). For instance, given an SGNN consisting of $L$ base modules, the graph operations are executed $L$ times in every FT. So, an SGNN conducts $\mathcal{O}(eL)$ graph operations where $e$ is the number of epochs. In $L$-layer GNNs, the graph operations are executed $\mathcal{O}(tL)$ times where $t$ is the number of BP iterations / steps. Note that $t \gg e$ usually holds since the classical BP usually requires lots of steps to converge. 

For the models with high-order graph operation (*e.g.*, SGC), we view a high-order operation as one graph operation, instead of multiple graph operations. 



**Q1.3**: Seems that fully separable GNN does not allow non-linearity. How does that affect the generality of SGNN? 

**R1.3**: The fully-separable GNN allows **one** non-linear activation function.  For example, a single-layer GCN, which is the base model in our experiments, is formulated as $H = \phi(AXW)$ where $\phi(\cdot)$ is the non-linear function. 

Besides, in Eq. (5), SGNN could introduce the additional non-linearity by $g_0^*(X, U)$ where $f^*(A, X, U) = g_1^*(A, g_0^*(X, U))$. 



**Q2**:  What is the physical meaning of U? How exactly can we get function Z=g(X,U), which is critical to solve the expected input for backward training.

**R2**: As mentioned in R1.2, the additional matrix $U$ is used to convert a separable GNN to *a composite model* consisting of **a fully-separable GNN** and **an MLP**. Formally, a separable GNN can be formulated as $f(A, X, W) = f_1(f_0(A, X), W)$. By introducing $U$, $f_0$ is replaced by a fully-separable GNN, $f_0(A, X) = g_1(A, g_0(X, U))$. Specifically, if $U$ is fixed as a constant and $g_0$ is linear function, then the composite model is equivalent to the original separable GNN. 

In our implementation, $g(X,U)$ is just a single-layer neural network. 



**Q3**: Why is SGNN efficient? 

**R3**: As mentioned in **R1.2**, when we want to compute the representation of node $v$ in the $L$-th module, we already have the trained representation, $H_{t-1}=X_t$, from the $(L-1)$-th module. When training the $L$-th GNN, we **do not need to execute the time-consuming graph operation in each BP step**, by virtue of the separable property. Instead, the separable base model firstly conducts the graph operation before the starting of BP. It even requires no neighbor sampling.

From: ICLR.cc/2023/Conference/Paper767/Authors
title:  Response to Reviewer svMx and thanks for the constructive comments (2/2)
comment: (Minors)

**Minors-Q1**: Missing important baselines in experiments. 

**Reply**: GIN and GraphSAINT are indeed important models. Although GIN is a well-known baseline, GCNII obtains better performance on most datasets. So we just choose a stronger method as the baseline. Similarly, GraphSAINT is also an important method that promotes the efficiency of GNNs, while the recently proposed S2GC is a stronger baseline. 



**Minors-Q2&Q3**: It will be better to mention in experimental settings which base model is chosen for SGNN-BT (GCN?), rather than in the analysis part.  It will be more clear if the best results in node classification tasks (e.g. Table 2, 3, 5) are highlighted.

**Reply**: Thanks for the kind reminding. The default base model is GCN. We now clarify the base model used in experiments. Meanwhile, we have highlighted the optimal results and sub-optimal results in Tables 2, 3, and 5. 



**Minors-Q4**: (Typos)

**Reply**: We apologize for these careless mistakes. We have polished the paper carefully and prepared this revision.

From: ICLR.cc/2023/Conference/Paper767/Authors
title: Response to Reviewer svMx and thanks for the constructive comments (1/2)
comment: We greatly thank the invaluable comments, which motivate us to clarify the core idea of SGNN. 

**Q1**:  Section 3.2 mentions that the fully-separable condition is essential for BT. Also, several powerful GNNs, such as GAT with the widely used attention mechanism, are not fully-separable or even not separable. Thus, it seems that they cannot be trained under this framework, which is a limitation. However, I notice that in Appendix E.2, it mentions that GAT can be a base model of SGNN, which has not been supported by experimental results. The two places seem contradictory and may need further elaboration.

**R1**: It is indeed an important question. At first, we apologize for the ambiguous statements about the *fully-separable* condition in P5. We fail to clarify the reason why we define the *separability* and why the additional parameters $U$ in Eq. (5) are required. 

The *fully-separable* condition is indeed essential for BT. However, in Eq. (5), 
$$
H = \hat f(A, X, W) = \hat f_1(\hat f_0(A, X), W) = \hat f_1 (f^* ( A, X, U), W) = \hat f_1 (g_1^* ( A, g_0^*(X, U)), W), ~ \forall \hat f \in \mathcal{F}_k, f^* \in \mathcal{F}_d^* ,
$$
we actually convert a *separable* GNN into **a composite model composed of a *fully-separable GNN* and *an MLP*** by introducing the learnable parameter $U$. Formally, in Eq. (5), the fully-separable GNN is denoted by $f^*(A, X, U)$ and the MLP is denoted by $\hat f_1(\cdot, W)$. 

Note that during FT, $U$ is fixed as a constant. Let $\hat X = g_0^*(X, U)$ and then Eq. (5) is equivalent to $H = \hat f_1(g_1^*(A, \hat X), W)$. Clearly, **it is equivalent to the original separable single-layer GNN.** In other words, to modify any separable GNNs for BT, the required step is to replace the $X$ by $\hat X = g_0^*(X, U)$. 

In summary, **the decoupling skill proposed in this paper can be applied to any separable GNNs**. Note that most of single-layer GNNs are separable, including but not limited to GIN, S2GC, GNNs with skipping connections(*e.g*., APPNP, JKNet). However, the separability is essential and GAT could not be applied. More specifically, it is caused by the fact that the graph attention relies on the learnable parameters $W$.  

We also apologize for the inconsistent statements in Appendix E and we have refined the appendices. 

We now prepare a revision to refine the confusing statements and wrong descriptions. Thanks again for your comments. 



**Q2**: It seems that after a module is trained in each epoch, new representations of all nodes will be computed and stored, and then input to the next module. My question is, when talking about efficiency, should the consuming time of intermediate inference and I/O be taken into account? 

**R2**: It is also a fair consideration of efficiency. We try to address this concern from the following two aspects: 

- At first, we want to re-emphasize that the consuming time is recorded from the beginning of loading data from disk to GPU in our experiments. Therefore, the consuming time of I/O has been included. 
- The passing of intermediate representation also exists (**implicitly**) in the conventional multi-layer GNNs. In SGNN, the passing is **explicit**. More importantly, a multi-layer GNN will retain all intermediate representations in GPU memory while SGNN only requires the input and output of the current module $\mathcal{M}_t$ in GPU memory. In SGNN, the intermediate representations can be dropped if the module has been trained. As a result, SGNN consumes smaller space. 



**Q3**: I'm not very clear about whether the projection matrix $R$ in the supervision setting mentioned in Page 5 is shared across different decoupled modules. I'm also curious about whether the additional projection will lead to error accumulation in such a setting.

**R3**: The projection matrix is not shared by all decoupled modules, since all modules are independent of each other. The middle modules can be the unsupervised ones as well. 

The projection matrix is not additional. It is the essential projection for the softmax layer. It is usually encapsulated in the functions/methods of diverse frameworks (e.g., pytorch).  

The error accumulation of supervised cases is indeed an attractive problem and it would be a core of our future works.

From: ICLR.cc/2023/Conference/Paper767/Authors
title: Response to Reviewer AHA4 and thanks for the constructive suggestions
comment: We thank the reviewer for the constructive comments. We are glad that the reviewer thought the decoupling and FT/BT are interesting. We prepare a revision to polish the paper carefully and clarify the preconditions of base models of SGNN. We apologize for the poor writing. 

We conclude the following concerns of the reviewer and try our best to address them. 

**Q1**: My main concerns are that most of the GNNs are not fully-separable due to the non-linearity, not sure if the assumption can be hold.

**R1**: We apologize for the ambiguous (even misleading and wrong) statements of fully-separability. In fact, **the assumption / precondition of base models for SGNN is just the separability.**  We intend to modify the separable GNNs by introducing the fully-separable property, which is essential for BT. Note that the separability is actually common for GNNs, *e.g.*, GIN, APPNP (with linear activation for intermediate layer), JKNet, S2GC. Here are the details: 

In Eq. (5), 

$$
H = \hat f(A, X, W) = \hat f_1(\hat f_0(A, X), W) = \hat f_1 (f^* ( A, X, U), W) = \hat f_1 (g_1^* ( A, g_0^*(X, U)), W), ~ \forall \hat f \in \mathcal{F}_k, f^* \in \mathcal{F}_d^* ,
$$

we actually convert a *separable* GNN into **a composite model composed of a *fully-separable GNN* and *an MLP*** by introducing the learnable parameter $U$. Formally, in Eq. (5), the fully-separable GNN is denoted by $f^*(A, X, U)$ and the MLP is denoted by $\hat f_1(\cdot, W)$. 

Note that during FT, $U$ is fixed as a constant. Let $\hat X = g_0^*(X, U)$ and then Eq. (5) is equivalent to $H = \hat f_1(g_1^*(A, \hat X), W)$. Clearly, **it is equivalent to the original separable single-layer GNN.** In other words, to modify any separable GNNs for BT, the required step is to replace the $X$ by $\hat X = g_0^*(X, U)$. 

In summary, **the decoupling skill proposed in this paper can be applied to any separable GNNs**. Note that most of single-layer GNNs are separable, including but not limited to GIN, S2GC, GNNs with skipping connections(*e.g*., APPNP, JKNet).



**Q2**: The sampling process in each layer of the FT and BT seems to consume a lot of storage spaces to maintain previous/future matrices. How does the memory efficiency look like?

**Reply**: It should be pointed out that a conventional $L$-layer GNN also requires storage spaces to store the intermediate features (output by middle layers). More importantly, **SGNN is also more efficient from this aspect**, since SGNN only needs to keep the output of the $(L-1)$-th model and *it can discard the output of shallower models*, *i.e.*, 
$\mathcal{M}_1, \mathcal{M}_2, \ldots, \mathcal{M}_\{L-2\} $.   



**Q3**: The conclusion from figure 2 is hard to interpret, why are they non-trivial and how does it compared to other non-GNN 2D visualization/embedding?

**R3**: Figure 2 is used to visualize the latent representations produced by classical GCN and its decoupled one. We can find that the last module of SGNN outputs more cohesive features, regarding different categories. Figure 2 is also an experimental validation of the theoretical analysis. 



**Q4**:Results about larger dataset in appendixes doesn't show SGNN has a strong use case when compared to traditional methods.

**R4**: The aim of our experiments is to investigate whether the decoupling skill **seeking the high efficiency** could approach the original GNN. The main advantage is the significant efficiency. From Table 5, we find that SGNN approaches the raw model GCN, which SGNN is decoupled from.

From: ICLR.cc/2023/Conference/Paper767/Reviewer_AHA4
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper the authors proposed a GNN architecture named SGNN that decouple a deep GNN into smaller modules for efficient training. The framework contains a traditional forward pass with greedy optimization called Forward Training (FT) and a redesigned backward propagation named Backward Training (BT). The backward training resembles the auto encoder framework, and the authors provide error bounds when using SGNN under certain condition of the network structure. Experimental results shows efficacy of SGNN in most datasets.
strength_and_weaknesses: Pros: 
- Ablation study on hyper parameters in appendix.
- Training efficiency plots are more realistic and easy to understand the tradeoff between complexity and accuracy. 
- Comparison to existing methods explain the main difference and contribution of the works. 

Cons: 
- I found the paper is not very well polished with a couple of typos and mismatch font style. Some that I found includes: 
'While retaining' instead of 'with retaining', T-th instead of t-the in section 3.1, missing m in the complexity, No bold/underline fonts in table 2/3 and tables in appendixes showing the winning model. Missing variance in table 3
- Lack of comparison to other efficient GNNs that decouple the propagation and neural network, like PPNP, GPR-GNN, etc. 
- The sampling process in each layer of the FT and BT seems to consume a lot of storage spaces to maintain previous/future matrices. How does the memory efficiency look like? 
- The error bound only apply for linear fully-separable GNNs, which is rare to find in real-world cases. 
- The conclusion from figure 2 is hard to interpret, why are they non-trivial and how does it compared to other non-GNN 2D visualization/embedding? 
clarity,_quality,_novelty_and_reproducibility: The paper can be understood clearly with a bit of effort. The architectural novelty is limited compared to stacked auto encoders, but the cascaded forward/backward training trick is interesting and does show improvements in real-world dataset. Most of the baseline results are computed with open source codes and datasets. 
summary_of_the_review: Although the authors proposed a way to train GNN with stochastic optimization under FT and BT, my main concerns are that most of the GNNs are not fully-separable due to the non-linearity, not sure if the assumption can be hold; The decoupling still haven’t solve the computational issues with large graph where even one single layer propagation is difficult and application seems limited. Results about larger dataset in appendixes doesn't show SGNN has a strong use case when compared to traditional methods. Hence, I am leaning towards not recommending an acceptation of the paper. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper767/Reviewer_mB9y
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper investigates an approach to compute and train a layered GNN architecture by decoupling it into individual shallow modules, leading to a potentially more efficient approach.
The proposal is assessed experimentally against literature models, showing the potentialities of the introduced approach.

Update after rebuttal: I thank the authors for considering my feedback on the paper (I have raised my score on the reviewed version)
strength_and_weaknesses: Strengths:
* The approached problem seems rather interesting
* The theoretical analysis is interesting and seems rather sound (though I could not follow all the steps)

Weaknesses:
* Presentation and readability of the paper can be improved. In particular, some tables and figures appear quite far from the point of interest in the manuscript (e.g., fig 2 is at page 6, but is commented at page 8). Moreover, I found a bit confusing and potentially unclear the reference to the terms “former modules” and “latter modules”.
* The original motivation to the analysis in the paper is to increase efficiency with the need of increasing depth of the GNN architecture. However, the experimental analysis is mostly limited to 2-layers architectures, hence one could argue that perhaps this limitation is not so impactful, as the number of layers does not need to be very large. In the appendix, some further results are presented for deeper architectures (i.e., larger number of modules in the proposed model), which show a generalized decreasing performance with the increasing depth. I therefore suggest that to fully appreciate this work, the proposal should be better demonstrated on problems that require actually a deeper architectural setup.
* The experimental analysis is not fully convincing. In particular, the hyperparameter tuning is not clear in the description of the experiments. In the current version, it is not clear how the hyperparameters of the individual models are tuned, and if any fair model selection procedure is applied or not. For fairness of comparison, a model selection should be performed for each model in the comparisons individually, and separately for each task. Besides, some aspects of the settings are not clear. For example, in the paper it is said that eta is set to 1 by default, while in the appendix it is said that its value is set to 1000. I think that this quantity should be treated as a proper hyperparameter and optimized among the others as mentioned above.
clarity,_quality,_novelty_and_reproducibility: The clarity and readability of the paper can be improved. Some figures/tables are relatively far from the point of interest in the text, and the manuscript presents some typos (E.g., “if a L-layer GNN achieves”,  “improvements on Reddit with high efficiency, Specifically speaking”).
summary_of_the_review: The paper is routed in interesting ideas. I think the overall relevance of this submission is hampered by issues in terms of presentation, scope of the analysis in relation to the motivations, and experimental settings.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper767/Reviewer_svMx
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper focuses on the training efficiency of GNN. In the mini-batch training setting of GNN, the aggregation-like operations will lead to exponential increases in batch size with the growth of layers due to neighborhood dependency. The authors propose to decouple a multi-layer GNN as multiple simple modules, which perform classical forward training greedily. To compromise the information loss caused by unidirectional information delivery, the authors design a backward training mechanism. They theoretically prove that the error produced by linear modules will not accumulate on unsupervised tasks in most cases. The theoretical and experimental results show that the proposed framework is of both efficiency and effectiveness.
strength_and_weaknesses: - Strength
    1. The paper is well-written in general. The content and structure are easy to follow.
    2. The idea of decomposing multi-layer GNN into modules that can be trained simultaneously is interesting and somewhat novel. The implementation is simple, yet proves to work both theoretically and empirically. It is possible to be used in ultra-scale industrial scenarios.
    3. The theoretical analyses seem solid. The authors focus on unsupervised tasks and give the error bound of the model in general cases.
    4. The authors conduct abundant experiments to prove their claims, including quantitative results in node clustering and node classification, training time, convergence curve, embedding visualization and hyper-parameter analysis. The corresponding analyses are sufficient. Results in large graphs strongly show the usefulness of the method. Also, sufficient experiment details are provided for reproducibility.
- Weaknesses

    **Majors**

    1. Section 3.2 mentions that the fully-separable condition is essential for BT. Also, several powerful GNNs, such as GAT with the widely used attention mechanism, are not fully-separable or even not separable. Thus, it seems that they cannot be trained under this framework, which is a limitation. However, I notice that in Appendix E.2, it mentions that GAT can be a base model of SGNN, which has not been supported by experimental results. The two places seem contradictory and may need further elaboration.
    2. It seems that after a module is trained in each epoch, new representations of all nodes will be computed and stored, and then input to the next module. My question is, when talking about efficiency, should the consuming time of intermediate inference and I/O be taken into account?
    3. I'm not very clear about whether the projection matrix $R$ in the supervision setting mentioned in Page 5 is shared across different decoupled modules. I'm also curious about whether the additional projection will lead to error accumulation in such a setting.
    
    **Minors**

    1. Missing important baselines in experiments. GIN [1] is one of the most expressive GNN models. GraphSAINT [2] is the representative of subgraph sampling methods under graph sampling. Both are mentioned in related works but not compared in experiments.
    2. It will be better to mention in experimental settings which base model is chosen for SGNN-BT (GCN?), rather than in the analysis part.
    3. It will be more clear if the best results in node classification tasks (e.g. Table 2, 3, 5) are highlighted.
    4. Typos in definitions and theorems may cause confusion. For example, should it be $f:\mathbb{R}^{n\times n}\times \mathbb{R}^{n\times d}\times \mathbb{R}^{d\times k}\rightarrow \mathbb{R}^{n\times k}$ in the paragraph after Definition 3.1? Also, it seems to be $||P-XX^t||=\varepsilon$ in Line 2 of Theorem 4.1. In Theorem 4.2, it may be $||P-HH^t||\leq \varepsilon+O(\sigma_*^2)$.
    5. The illustration of the model (Figure 1) can be further improved by highlighting the relation between $H_t$ (the output of a module) and $Z_{t+1}$ (the expected input of a module), which may give a better picture of the back training process.

[1] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.

[2] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
clarity,_quality,_novelty_and_reproducibility: - Clarity

    The paper is well-written in general. The content and structure are easy to follow. However, it will be better if further improved.
    
- Quality

    This work proposes a simple yet effective framework for training GNN with high efficiency. The sufficient theoretical analysis and experimental results well support the major claim of the paper, and prove the effectiveness of different components in the model.

- Novelty
    
    The idea of decomposing multi-layer GNN into modules that can be trained simultaneously is somewhat novel, which is rare in previous works related to GNN training.
    
- Reproducibility
    
    The authors provide enough experiment details for reproducibility. Source code is not available before publication.
summary_of_the_review: The idea of the paper is interesting, novel and of practical value. The sufficient theoretical and experimental results show that the proposed framework is of both efficiency and effectiveness, outperforming existing baselines. However, the framework seems not generalizable to some powerful models like GAT. Also, there is some space for improvement, like typos, presentation and experiments.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper767/Reviewer_qSsw
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a framework named stacked graph neural network (SGNN), which allows to train different layers of GNN separately to avoid the neighbor explosion problem of current GNN models. Theoretical analysis shows that SGNN provides good approximation of the Laplacian matrix and experiment results show that SGNN yields good accuracy and efficiency. 
strength_and_weaknesses: Strong points

1. Improving the training efficiency of GNN models is important. 

2. The framework of SGNN is formalized, which makes it possible to combine with different base models.

3. Theorical analysis is conducted to show that SGNN provides good approximation. 

Weakness

1. Presentation of the SGNN framework leaves several questions.

2. Experiment needs to be significantly improved.


clarity,_quality,_novelty_and_reproducibility: Good novelty, OK quality.
Clarity needs to be improved. 
Codes are not provided (only with the promise that codes will be provided), thus reproducibility can be improved.
summary_of_the_review: Thanks for submitting to ICLR. I have some questions (maybe naive) and suggestions for the authors.

1. Why is the separable (or fully separable) GNN assumption required?
My understanding is that fully separable GNN means that the graph operation and neural network operation satisfies the law of associativity in a layer of the GNN model. The second paragraph below Definition 3.1 gives me the feeling that SGNN only conducts graph operation once. However, Figure 1 and the fact SGNN uses SGC as the modules suggest that SGNN conducts graph operation in each layer. (So which is correct?) On a high level, in the forward training, SGNN requires the output of each module to match final target output; in the backward training, SGNN requires the output of a former module to match the expected input of its downstream module. Seems that the requirement is that we should be able to solve the expected input of its downstream module efficiently. How does that link to fully separable GNN?
Seems that fully separable GNN does not allow non-linearity. How does that affect the generality of SGNN?

2. Backward training unclear.
I am confused by Eq(5)-(7). What is the physical meaning of U? How exactly can we get function Z=g(X,U), which is critical to solve the expected input for backward training.

3. Why is SGNN efficient?
According to my understanding, SGNN also consists of layers of stacked modules, and each module is essentially a 1-layer GNN. Say, we sample one node v for training, which requires v’s L-layer embedding for forward training; to compute v’s L-layer embedding, the (L-1)-layer embedding for v’s neighbors are required; and the dependency goes on. To compute the L-layer embedding of any sampled node efficiently, SGNN needs to precompute the 1 to (L-1) layer embedding of all nodes in the graph. However, this makes SGNN very similar to full-graph training, which computes output embedding for all nodes in the graph. 

4. If my understanding is correct, SGNN achieves good accuracy because it introduces more supervision because of it uses more supervision for each layer. 

5. Experiments need to be significantly improved.
(a) GAT seems a strong baseline in the node classification experiment. Why is it not used for node clustering? 
(b) The efficiency comparison in Figure 3 is unfair. The authors use the time to train an iteration but different could require different iterations to reach convergence. I suggest the authors directly use the time to reach convergence.
(c) Table 5 in the Appendix shows that SGNN-BP has lower accuracy than the baselines on the OBGN graphs, which makes the results in the main paper look like cherry picking. 
(d) Table 6 in the Appendix is interesting as it shows how SGNN works with more layers. It will be more interesting if the authors can show the accuracy degradation of other models (when increasing the number of layers). I understand that GNN accuracy degrades because of over-smoothing but it is also possible that training strategy of SGNN does not work for more layers. It is important to validate whether SGNN suffers more penalties under more layers.

There are several unclear places in the paper for me. I will adjust the ratings after rebuttal.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold