From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This papers considers the oversmoothing problem in GNNs and proposes a novel GNN architecture based on a gradient gating module. A multi-rate update schema implemented as graph gating operation is designed as a wrapper for GNN layers. The module is proved to alleviate oversmoothing through theoretical analysis and simulations, and shows good experimental results on the benchmarks.

Strengths: 
The proposed operator is backed by theory to alleviate the oversmoothing issue, and demonstrates promising results in practice. The idea of introducing gating mechanism to GNNs is novel and intriguing. 

The paper is well written and easy to follow. 

The experiments are on large-scale datasets and results are convincing. 

Weaknesses: 
There are concerns about hyperparameter tuning, but the authors addressed the issue during the rebuttal period.  Since there is a plethora of methods claiming to solve the oversmoothing problem, the reviewers suggest that the authors elaborate why their method should be better than existing ones in the manuscript. 

justification_for_why_not_higher_score: There are concerns about hyperparameter tuning, but the authors addressed the issue during the rebuttal period.  Since there is a plethora of methods claiming to solve the oversmoothing problem, the reviewers suggest that the authors elaborate why their method should be better than existing ones in the manuscript. 
justification_for_why_not_lower_score: The proposed operator is backed by theory to alleviate the oversmoothing issue, and demonstrates promising results in practice. The idea of introducing gating mechanism to GNNs is novel and intriguing. 

The paper is well written and easy to follow. 

The experiments are on large-scale datasets and results are convincing. 
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Thanking the Reviewer
comment: We thank the reviewer for your positive assessment of our revised version and for raising our score.

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_8MQJ
title: Thanks for the response
comment: I have checked comments from other reviewers as well as responses from the authors. I think the authors have properly addressed most problems raised before, so I increase my score towards acceptance.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Thanking the Reviewer
comment: We thank the reviewer for appreciating our rebuttal and revised version and for raising their score.

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_xxu8
title: Thanks for the response
comment: I have read the response and raised my score. I like the method proposed in the paper and hope to see more explorations in this direction. It seems that there still isn't a single $p$ that can achieve results no worse than $G^2$-GraphSAGE reported in Table 2, but this also indicates that $p$ captures distinct properties of the datasets, and practically users can do some additional tuning for each dataset. Regarding the parameter-matching settings, it's better to compare in a regime where changing #parameters actually has an effect on the performance (and I think this regime exists).

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Thanking the Reviewer
comment: We thank the reviewer for appreciating the merits of our revised version and rebuttal and for increasing our score.

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_uYz6
title: Response
comment: Thank you for your efforts. The authors did a great job addressing my concerns and I have adjusted my score accordingly. In my view, the work is a valuable addition to the field and worth publishing at the conference.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer uYz6 Part 2/2
comment: 2.
   * **c)** The reviewer is justified in observing that the confidence intervals for some datasets in Table 2 are relatively large and make it difficult to claim that one method is superior to another. In this context, we would like to say that we follow a large corpus of literature in considering these datasets and in labelling one method as superior to another, for instance, Li et al 2022, Rusch et al 2022a, Di Giovanni et al 2022 etc. Moreover, we have also identified the three best-performing models, rather than a single best-performing model, in order to make a fairer comparison. The corresponding text in the main article is now modified to reflect these observations. Nevertheless, we would like to emphasize that the confidence intervals are considerably lower for Squirrel and Chameleon datasets allowing one to better compare different models. On these datasets, our proposed model is either on par or outperforms the state of the art, even when confidence intervals are taken into consideration. This observation also holds for the large graphs presented in Table 3. The corresponding confidence intervals are much smaller in amplitude  allowing for a fair comparison between different models and showcasing the superior performance of ${\bf G}^2$-GraphSAGE on the snap-patents and arxiv-year datasets, over available baselines. 

3. **Using  ${\bf G}^2$ for homophilic graphs:** we would like to point out that in addition to Genius (homophily level of 0.61), we have also considered the standard Cora benchmark (homophily level of 0.81). The corresponding results are presented in Figure 3, from which we observe that  ${\bf G}^2$-GCN is on par with (or even slightly superior to ) baselines on this homophilic graph dataset. Together with the results on the synthetic Cora dataset (Figure 5), it is reasonable for us to conclude that  ${\bf G}^2$ is comparable to other GNNs on homophilic datasets while providing state-of-the-art performance on (large) heterophilic graphs. 

4. The reviewer is correct in asserting that there are a plethora of methods that claim to solve the oversmoothing problem. We differentiate our framework from many such methods by i) rigorously proving that gradient gating  alleviates the oversmoothing problem (Proposition 3.4) and backing it up with empirical evidence (Figure 2); ii) empirically showing that  ${\bf G}^2$ allows using a very large number of hidden layers without any deterioration of accuracy (Figure 3) as compared to many other methods that claim to solve the oversmoothing problem and yet fail to retain accuracy with increasing the number of hidden layer; and iii) by extensive experimentation that allows us to show that  ${\bf G}^2$ is comparable to other GNNs on homophilic datasets while providing state-of-the-art-performance on (large) heterophilic graphs.

We sincerely hope that we have addressed the concerns of the reviewer, particularly on the empirical evaluation of the proposed method, satisfactorily in the revised version and would kindly request the reviewer to update their score accordingly.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer uYz6 Part 1/2
comment: We thank the reviewer for appreciating the merits of our paper and their suggestions for improvement. Below, we address the concerns raised by the reviewer. 

1. We agree with the reviewer's observation that a possible explanation for the superior performance of our proposed models on the multi-scale node level regression task (presented in Table 1) could lie in the fact that these models also perform significantly better for the classification tasks on the underlying heterophilic graphs (shown in Table 2). Following the reviewer's suggestion, we test this hypothesis by also evaluating architectures such as PairNorm and GCNII, which are either very close in accuracy or outperform ${\bf G}^2$-GCN and ${\bf G}^2$-GAT on classification for Squirrel and Chameleon datasets, also for node-level regression on the same underlying graphs. The results are now reported in Table 1 and show that despite these methods being comparable for classification on heterophilic graphs, they are significantly outperformed by ${\bf G}^2$-GCN and ${\bf G}^2$-GAT on multi-scale regression. Thus, following the reviewer's hypothesis, this experiment demonstrates that it is not just the superior performance on the underlying heterophilic graphs, but rather the multi-rate nature of ${\bf G}^2$ that explains its improved performance on this task.  This aspect is further reinforced in **SM** Figure 6 where we investigate (ablate on) the sensitivity of the results of our ${\bf G}^2$ procedure with respect to its multi-rate nature. In particular, a parameter is introduced to modulate the rate matrix in Eqn (4). We observe from Figure 6 that the best performance is observed when the full multi-rate aspect of our architecture is utilized. This provides further evidence of the crucial role played by the multi-rate nature of our model in learning this task with multiple scales. 

2. **Further ablation studies:**

   * **a)** **Necessity of the GNN $\hat{\bf F}$ in Eqn (4):** we would like to emphasize that we had already mentioned in the previous version of the article that indeed one can set $\hat{\bf F} = {\bf F}$ in Eqn (4). Indeed, we had done so in quite a few of the best performing models in Table 2. To further clarify this point, we have now added a new table (**SM** Table 4) where we state the learning tasks for which we set $\hat{\bf F} = {\bf F}$ and those where this is not the case. Clearly, $\hat{\bf F} \neq {\bf F}$ provides additional flexibility to ${\bf G}^2$ and might improve the expressivity of the resulting GNN in some cases. To test this hypothesis, we have now added a new table (**SM** Table 5) where we ablate this element in our architecture on three datasets to observe that on at least two of them, there is a significant gain in performance when $\hat{\bf F} \neq {\bf F}$. Hence, this design choice should be available to the user when they deploy ${\bf G}^2$ on graph learning tasks.


   * **b)** Further ablating the multi-rate component, vis a vis channels, by averaging the $p$-norms of differences per node and obtaining a single gating score per node instead of our proposed architecture where the gating score is generated per node and per channel feature: We thank the reviewer for this suggestion and test their hypothesis in the revised version. To this end, we have added a short section in the **SM** on the **ablation of multi-rate channels** in ${\bf G}^2$. Following the reviewer's suggestion, we consider Eqn 16 where a single gating score for a node is generated (by averaging over all feature channels) and the resulting model is tested on three different datasets. The results are presented in **SM** Table 6. We observe from this table that there is a considerable gain in performance with the multi-rate channel architecture (Eqn (4)) over single rate per channel architecture (Eqn (16)) on the Squirrel and Chameleon datasets. Hence, following the reviewer's line of thought, we conclude that this experiment provides some empirical justification in support of our claim that *in realistic graph learning tasks one can encounter multiple rates for the flow of information*.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer f9D8 Part 2/2
comment: 4. **Method is not convincingly evaluated:** we also kindly disagree and believe to have provided considerable experimental evaluation of our method on diverse tasks and datasets. We are glad that other Reviewers share our position. A total of 10 datasets are considered in classification tasks and further 2 datasets are considered for node-level regression. The underlying graphs span a wide range of homophily levels, ranging from $0.07$ (snap-patents) to $0.81$ (Cora). Similarly, the underlying graphs that we consider span a range of sizes, starting from small graphs (Texas with only $183$ nodes and $295$ edges) to very large graphs (snap-patents with $2.92$ million nodes and $13.97$ million edges). Finally, we have also compared with a very large number of baselines i.e., 16 baselines in Table 2 and 11 baselines in Table 3. On all these datasets, our models perform either on par or significantly better than the state of the art. We believe that the provided experimental evidence, together with the theoretical analysis, convincingly shows that the proposed framework advances the state of the art for GNNs. We further included additional experiments in response to other comments, that hopefully will make the Reviewer change their position. 

We sincerely hope that we have addressed the concerns of the Reviewer, particularly on the novelty of the methods, their reproducibility and on their experimental evaluation, satisfactorily in the revised version and would kindly request the reviewer to update their score accordingly.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer f9D8 Part 1/2
comment: We thank the reviewer for reading our paper and for your comments. We address the concerns you raised below:

1. **Example where multi-rate learning might be necessary:** We would like to point out that we provide exactly such an example in our experiments where we consider gradient gating for a multi-scale node level regression task (see Section 4, p. 6). In this experiment, the Chameleon and Squirrel graphs are considered and the underlying task is to predict continuous node targets (average monthly web-page traffic for every node). The first observation, presented in Figure 4, is that there is a wide range of scales (spread over almost five orders of magnitude)  in the node values. Thus, this task clearly is of a multi-scale nature. We investigate whether the proposed multi-rate GNNs would be superior to baselines on this task and present the results in Table 1. As seen in this table, adding the multi-rate mechanism to the baseline GNN significantly reduces the test errors.  In particular, for Chameleon, adding multi-rate (ours) reduces the error with respect to the baseline GCN by 33% and with respect to the baseline GAT by 34%. Similarly for Squirrel, adding multi-rate (ours) reduces the error with respect to the baseline GCN by 51% and with respect to the baseline GAT also by 51%. These very significant improvements in performance are a testament to the role played by our multi-rate model. This aspect is further reinforced in an ablation study in **SM** Figure 6 where we investigate  the sensitivity of the results of our gradient gating procedure with respect to its multi-rate nature. In particular, a parameter is introduced to modulate the rate matrix in Eqn (4). Figure 6 shows that the best performance is observed when the full multi-rate version of our mechanism is utilized. This provides further evidence of the crucial role played by the multi-rate nature of our model in learning this graph-based task with multiple scales.

2. **Connections of our proposed framework to GRU and its novelty:** we would like to emphasize that we have devoted a paragraph in the Related Work (Section 5) to the connections between our proposed framework and existing gating procedures. In particular, we have clearly stated that the idea of gating originated in the RNN community, with LSTM and GRU being prominent examples. This idea of gating was also used for GNNs in papers such as Bresson and Laurent 2017 where gating based on residual connections were proposed for GNNs. However, to the best of our knowledge, *our article is the first to present an architecture where the gating is modulated by the graph gradients*. This is the key mechanism allowing to provably alleviate oversmoothing in GNNs (see Proposition 3.4 and the contrast with Proposition 3.3) and allows to take advantage of deep multi-layer GNN architectures. Moreover, the explicit use of multi-rate mechanism within ${\bf G}^2$ also differentiates it from GRUs as well as other gated GNN frameworks. 

3. **Lack of reproducibility:** We politely disagree with this assertion and are happy that all the other Reviewers consider our results reproducible (uYz6: "Good, as the descriptions are sufficiently clear and the authors provide their implementation"; 8MQJ: "should not be difficult to reproduce the experimental results"; xxu8: "Code is already provided to prove reproducibility"; and WoCb: "presentation of the proposed method is easy to follow"). Following the standards of the current GNN literature, we have clearly described the benchmarks and the methods. Moreover, we have provided the code for generating the results with all our proposed models. Moreover, we have now added a new table (**SM** Table 7) where we describe the precise range of hyperparameters that we use in our random search procedure, described in **SM** Section B, for obtaining the best performing hyperparameters. In Table 8, we provide the optimal values of the hyperparameters $p$ that we have introduced in the proposed ${\bf G}^2$-architecture. Since the code is provided, we did not further clutter the **SM** with detailed values of the number of layers, parameters etc. However, if the reviewer wishes, we will be happy to add these values in the **SM**.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer WoCb Part 2/2
comment: **3.** In response to the Reviewer's suggestions about the experimental results, we would like to make the following points:

**a)** We had already provided results regarding the model performance w.r.t. number of hidden layers. This is presented in Figure 3 where the classification accuracy of different models (Y-axis) are plotted vs the number of hidden layers (X-axis) for the standard Cora benchmark dataset. As seen from this figure, the accuracy of the baseline GCN drops drastically as the number of hidden layers is increased. Moreover, the accuracy also drops significantly, albeit the onset of the accuracy drop is  delayed, when one adds various remedies for oversmoothing such as residual connections, DropEdge, and even the very recently proposed GraphCON framework. Only for the gradient gating framework, we observe a small but perceptible *gain in accuracy* when the number of hidden layers is increased. We believe that this experiment clearly shows how gradient gating can alleviate oversmoothing to enable the use of very deep GNNs and obtain improvements in model performance.  


**b)** We thank the reviewer for suggesting a different measure of oversmoothing alled *mean average distance* (MAD) [1], which we have now added to the list of references. We, on the other hand, follow Rusch et al 2022  that used the decay of Dirichlet energy as the measure of oversmoothing. The current article is clearly not the venue for discussing different extant notions of oversmoothing. Nevertheless, we would like to point out that one can readily calculate that the decay of Dirichlet energy (to zero) automatically implies the convergence of MAD to zero. On the other hand, MAD can decay to zero, while the Dirichlet energy may remain bounded away from zero. Given these considerations, we have followed the reviewer's advice and plotted a MAD analogue of Figure 2 in **SM** Figure 9. As shown in this figure, MAD decays very rapidly in this experiment for the baseline GCN and GAT architectures while remaining constant for the gradient-gated versions of both GCN and GAT. Thus, even with this metric, our conclusions about the alleviation of oversmoothing by gradient gating hold true. 

**c)** We follow the reviewer's suggestion to compare our gradient gating method with other gated GNNs. To this end, we consider the ResGatedGCN architecture of Bresson and Laurent 2017 and add the results to Table 2. As seen from this table, our gradient-gated GCN significantly outperforms ResGatedGCN in every single benchmark. These results motivate us to conclude that gradient gating can be superior in performance to other gated GNNs. 

We sincerely hope that we have addressed the concerns of the reviewer, particularly on the connection between the theoretical part and methodology, satisfactorily in the revised version and would kindly request the reviewer to update their score accordingly.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer WoCb Part 1/2
comment: We thank the reviewer for appreciating the merits of our paper and their suggestions for improvement. Below, we address the concerns raised by the reviewer. 

1. **Novelty of gradient gating in GNNs:** We would like to emphasize that gating in GNNs has been used before and the relevant references were already provided in the related work section (Section 5) of our article. The real novelty of our approach, as also evident from the name "Gradient Gating", lies in *the use of graph gradients to modulate the multi-rate flow of information in GNNs*. To the best of our knowledge, this is the first time that such an approach has been presented. We would like to clarify that, unlike asserted by the Reviewer that the idea "seems to be directly from previous work" (Lim 2015), the notion of a graph gradient, by analogy to classical differential operators in calculus, is classical and has been around for at least half a century. The reference we cite (Lim, Hodge Laplacians on Graphs, 2015) is merely a survey paper providing a definition of the graph gradient and relation notions. This paper does not even consider graph neural networks, let alone the use of gradient gating in this context, and does not overlap with the contents of our article. 

2. **Strong connection between the oversmoothing problem and the gradient gating method:** We would like to point out that the theoretical analysis presented in our paper rigorously establishes such a connection. In particular, in Proposition 3.4, we prove that the gradient-gated GNN will neccessarily alleviate the oversmoothing problem. On the other hand, in Proposition 3.3, we prove that the lack of gradient gating can lead to oversmoothing in GNNs. These two propositions establish a clear connection between oversmoothing and gradient gating. This is further illustrated with the numerical example presented in Figure 2 where we show that the Dirichlet energy decays exponentially fast with respect to increasing the number of hidden layers for baselines such as GCN and GAT where no gradient gating is used, a quantitative evidence of oversmoothing in these models. In contrast, adding Gradient Gating alleviates oversmoothing as the Dirichlet energy remains constant even when the number of layers increases up to $1000$ !. We thus kindly ask the Reviewer to check these results again and revisit their point.
**The reviewer's point about moving some conclusions regarding oversmoothing from section 3 to section 2 is very welcome.** In this regard, we would like to point out that we had already provided a paragraph on the intuition behind the connection between gradient gating and oversmoothing at the end of Section 2 (where we identify early stopping of the message passing procedure due to gradient gating as the key mechanism in preventing oversmoothing). Moreover, this intuition is reconnected with the theoretical analysis in the very last sentence of Section 3 where we justify this intuition from the rigorous statement about oversmoothing and gradient gating in Proposition 3.4. We hope that these comments will allow the reader to understand our rationale behind the design of the gradient gating procedure and its role in preventing oversmoothing.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer xxu8
comment: Thank you for appreciating the merits of our paper and suggestions for improvement. Below, we address the concerns raised by the reviewer. 

1. **Selection of the hyperparameter $p$ in Eqn 4 (the exponent of the gradient gating term):** we followed the suggestion about checking the sensitivity of the performance of ${\bf G}^2$-GNNs with respect to the hyperparameter $p$. This is now provided in **SM** Figure 7 where we plot the accuracy of ${\bf G}^2$-GraphSAGE while varying $p \in [1,5]$ for the heterophilic datasets considered in Table 2. We see from this figure that the accuracy of ${\bf G}^2$-GraphSAGE is (approximately) independent of the value of $p$, with very minor variations in accuracy for varying $p$. This indicates that even fixing at a set value, for instance with $p=2$ or $p=3$, will transfer very well to different tasks. If users want to improve the performance further, then a simple hyperparameter grid (or random) search in the interval $[1,5]$ can be readily performed to yield marginally better results. This experiment, together with the theoretical analysis in Proposition 3.4, indicates to us that the results of ${\bf G}^2$ are not sensitive to the choice of $p$. 
This point was also raised by Reviewer 8MQJ, and we hope to have addressed the concern of both.  

2. **Comparison in the parameter/compute match settings:** Unfortunately, it is not possible to access the number of parameters for every baseline architecture on every dataset reported in Tables 2 and 3 as the baselines reported in these tables are taken from the underlying references, where these parameter values have either not been provided or are not readily available. Given this limitation, we decided to investigate the Reviewer's question by fixing a dataset (Texas from Table 2) and comparing the performance of two architectures -- the classical GCN and its gradient-gated version ${\bf G}^2$-GCN. To this end, we varied the number of parameters for both architectures and reported the classification accuracy for Texas for a variety of parameter values. The results are presented in **SM**, Figure 8 where the number of parameters (X-axis) for both models is varied from $16$ to $256$ and Classification accuracy is reported (Y-axis). From this figure, we conclude that for the same number of parameters, ${\bf G}^2$-GCN significantly outperforms baseline GCN, by increasing accuracy by almost 40%. This experiment shows that it is clearly not the number of parameters that is responsible for the superior performance of ${\bf G}^2$ over the baselines, but rather other features such as gradient-gating and multi-rate information propagation could explain its superior performance.

We sincerely hope to have address the concerns raised by the Reviewer to their satisfaction and would kindly request them to update their score accordingly.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to Reviewer 8MQJ
comment: Thank you for appreciating the merits of our paper and suggestions for improvement. Below, we address the concerns raised by the reviewer. 

1. The reviewer is correct in asserting that setting ${\boldsymbol{\tau}}^n = \frac{1}{2} {\bf ID}$ in Eqn (2) leads to the residual message-passing GNN (Eqn 1), but with a scale factor $\frac{1}{2}$ on the right-hand side. However, we would like to point out that we do not aim to directly recover the residual message-passing GNN (Eqn 1) from Eqn (2). Rather our form Eqn (2) has the following two limit cases: if ${\boldsymbol{\tau}}^n = {\bf ID}$ in Eqn (2), we recover the standard (non-residual) GNN (Eqn 5). On the other hand, if ${\boldsymbol{\tau}}^n = {\bf 0}$, we recover that ${\bf X}^n = {\bf X}^{n-1}$ and there is no update in the node features. The latter property is necessary as we want to ensure that there are no further updates at a node, when the local graph gradients are close to zero and this is intuitively the mechanism via which the proposed architecture ${\bf G}^2$ alleviates oversmoothing. Moreover, the formula in Eqn (2) can be thought of as a *convex combination* of the afore-mentioned limit cases. This convexity property is instrumental in proving Proposition 3.1 on the lower and upper bounds on hidden states and provides stability to ${\bf G}^2$ for *any* choice of coupling function. Given these considerations, we posit Eqn (2) as the basic form of ${\bf G}^2$.  If one wants to recover the output of a residual GNN, it is enough to realize that there is usually a linear output layer (decoder). Since Eqn (2) with above choice of ${\boldsymbol{\tau}}^n$ only differs from a residual GNN with a scale factor $\frac{1}{2}$, its output can be readily recovered by adjusting the weights of the linear output layer. 


2. **Role of the hyperparameter $p$ (exponent in the gradient gating term in Eqn 4):**  we would like to point that the theory (see Proposition 3.4) does not require large values of $p$. Rather, oversmoothing is alleviated for any $p>0$. Moreover, we have followed the reviewers' suggestion and have now provided additional experimental results on the sensitivity of the performance of ${\bf G}^2$-GNNs with respect to the hyperparameter $p$. This is provided in **SM** Figure 7 where we plot the accuracy of ${\bf G}^2$-GraphSAGE while varying $p \in [1,5]$ for the heterophilic datasets considered in Table 2. We see from this figure that the accuracy of ${\bf G}^2$-GraphSAGE is approximately independent of the value of $p$, with very minor variations in accuracy for varying $p$. This indicates that even a fixed value of $p$, such as $p=3$ leads to a good performance. Further improvement can be achieved by a simple hyperparameter grid (or random) search in the interval $[1,5]$. 
This point was also raised by Reviewer xxu8, and we hope to have addressed the concern of both. 


3. **tanh activation:** As can be clearly observed from Eqn (4) (2nd line), the argument to the activation function is a non-negative quantity. Since  $\tanh:[0,\infty) \mapsto [0,1]$, we can use $\tanh$ as the activation function and maintain the interpretation of ${\boldsymbol{\tau}}^n$ as rates. 

4. We term our architecture as *Gradient Gating* to emphasize that the use of graph gradient information is a key mechanism to alleviate oversmoothing (Proposition 3.4). Conversely, we show that the lack of gradient gating can lead to oversmoothing (Proposition 3.3). 

5. We have included paper [1], suggested by the reviewer, in our references and added the results for one of the models F$^2$GAT from [1] in Table 2. We chose F$^2$GAT as it has the highest average rank among all the F$^2$ models. We note that adding the results for all the F$^2$ models would make Table 2 very difficult to interpret as it already had 18 rows. 


6. We thank the reviewer for pointing out that GloGNN and LINKX were missing from Table 2. It was an oversight on our part as the results on small- and medium-scale heterophilic graphs were only presented in the Appendices of the papers proposing LINKX. We have now included these results in Table 2 and are happy to report that our G2-based GNNs either outperform GloGNN and LINKX on these datasets or are on par with them. 

7. The minor issues with duplicated references was fixed. 

We sincerely hope to have improved both the theoretical justifications as well as empirical results to the reviewer's satisfaction and would kindly request them to update their score accordingly.

From: ICLR.cc/2023/Conference/Paper3998/Authors
title: Reply to all the reviewers
comment: We would like to thank all five reviewers for their thorough and patient reading of our article and for praising our paper as "good work" (WoCb) proposing a "novel mechanism alleviate the over-smoothing problem" and a method that is "interesting and promising" (uYz6), "novel and substantially different from existing methods" (8MQJ) and "backed by theory" (xxu8) and "theoreticaly justifies" (uYz6). We are also glad the reviewers appreciated our "strong experimental results" (uYz6) that "demonstrate the superiority of the proposed method" (WoCb), "state-of-the-art performance"  (8MQJ), and "good results in practice" (xxu8), and found that the paper "is easy to follow" (WoCb), "well-motivated and clearly explained" (uYz6), and that the clarity is "very good" and "the presentation is clear and easy to follow" (uYz6).  

The constructive suggestions allowed us improve the paper, which we hope the reviewers will find to satisfactorily address the raised concerns. 

We uploaded a revised version of the paper incorporating the reviewers' suggestions. The following main changes were made:

- Figure 3: added residual GCN baseline
- Table 1: added GCNII and PairNorm baselines
- Table 2: added LINKX, GloGNN, ResGatedGCN and F$^2$GAT baselines
- SM Figure 7: added more datasets to test the influence of parameter $p$
- Added SM Figure 8 (parameter-constrained comparison)
- Added SM Figure 9 (different oversmoothing criterion, MAD)
- Added SM Table 7 (hyperparameters)
- Added SM section on ablation on $\hat{F}$
- Added SM section on ablation of multi-rate channels 

We answer the points raised by each of the reviewers individually in our comments below. Please note that all the references to page numbers, sections, figures, tables, equation numbers and references, refer to those in the revised version. We hope that based on the answers to their questions and the improvements made to the revised version, the reviewers will increase their scores.

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_8MQJ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes gradient gating ($G^2$), which assigns different trainable weights for different nodes and representation channels in each layer of GNN. The proposed method can be easily combined with different GNN, e.g., GCN or GAT. Theoretical results prove that $G^2$ can help alleviate the over-smoothing problem in GNN, and empirical results demonstrate that the proposed framework achieves state-of-the-art performance on node classification with different heterophilic graphs, both standard and large-scale. 
strength_and_weaknesses: Strength:
- This paper proposes a novel mechanism for GNN to alleviate the over-smoothing problem
- Empirical results against several existing baseline methods show state-of-the-art performance of the proposed method

Weakness:
- Some parts on the proposed method may need more explanations:
  - I am a bit curious on how $G^2$ can reduce back to standard GNN with residual connection.  Equation (5) in section 3 shows its connection with standard GNN when $\tau^n=0$. And from (2), I suppose setting $\tau^n=1/2$ will only differ in a constant with (1). Nevertheless, it is a bit confusing why there is a difference in constant. The authors may need to add some explanations on that. 
  - It is a bit unclear why there is an additional hyper-parameter $p$ on computing $\tau$, considering the expressive power of GNN. The sensitivity analysis in appendix A seems to show that we need a large enough $p$ to ensure good performance. Are there some explanations on that behavior?
  - Also, since $\tau \in [0,1]$, why tanh activation function is used for $\tau$? I suppose the output of tanh function lies in $[-1,1]$? 
  - Moreover, it is a bit confusing how the proposed method is connected to gradient, as the method name "gradient gating" indicates. I suppose it is more appropriate to call it "channel weighting" as $\tau$ is directly multiplied on each channel of node representations. 
- Empirical results:
  - I think a very relevant paper [1] is missing in current version. The authors may consider adding some discussions or empirical comparisons with it. 
  - Also, why are GloGNN and LINKX missing in Table 2? Are there any special reasons not to compare with them?
  - It may be also good to show some visualization results on $\tau^n$, and see how it changes with layer $n$. I wonder if there will be some interesting patterns, e.g., only few channels receive large weights while other channels have near-zero weight. 
- Minor issue: some references seem to be duplicated. Please double-check. 

[1] Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective. WWW 2022

clarity,_quality,_novelty_and_reproducibility: Clarity: the main idea of this paper is clear from my perspective, though some parts need further clarifications. 

Quality: overall I think this paper has a fair quality, while both theoretical justifications and empirical results can be further improved. 

Novelty: though I am not really an expert in this field, I think the proposed method is novel and substantially different from existing methods.

Reproducibility: the authors have uploaded their code as supplementary material, and hyper-parameters for each method and data set are also provided in appendix. Though I did not really run their code, I think it should not be difficult to reproduce the experimental results. 

summary_of_the_review: While this paper proposes a novel method and empirical results show some improvements, more explanations can be added to make the proposed method more clear, and some baseline methods are also missing in comparison. As such, I think current version is still marginally below the acceptance bar. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_xxu8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a novel GNN architecture based on a gradient gating module. The module is proved to alleviate oversmoothing through theoretical analysis and simulations, and shows good experimental results on the benchmarks. 
strength_and_weaknesses: Strength: The proposed operator is backed by theory to alleviate the oversmoothing issue, and demonstrates good results in practice.

Weakness: It introduces another hyperparameter $p$ which requires additional tuning efforts. It is not discussed whether the performances still match when number of trainable parameters or computations are the same. 

Could the authors provide some additional results to show:

1. Is there a good $p$ that transfers well across different tasks? Or is there some principle to make tuning $p$ easier?

2. How do the results compare in the parameter or compute match settings?
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow, while providing theoretical insights. The method is novel to my knowledge. Code is already provided to prove reproducibility.
summary_of_the_review: Overall I like the method proposed in the paper and the experimental results shows good improvements, therefore I lean towards accepting the paper. It would be even better if my concerns are addressed.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_WoCb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work proposes a graph gating method for advancing the capacity of overcoming over-smoothing problem of GNN models. Theoretical proof and analysis are provided to demonstrate its superiority on dealing with over-smoothing issue with the help of ordinal differential equation (ODE). Experimental results look impressive on graph heterogeneity problem. The presentation and writing are overall good for catching up with the main idea.
strength_and_weaknesses: Strength:
1. A multi-rate update schema implemented as graph gating operation is designed as a wrapper for GNN layers.
2. Theoretical analysis on the properties of  the proposed method is provided.
3. Extensive experimental results on classification and regression tasks demonstrate the superiority of the proposed method.

Weakness:
1. Employing gating method to control the message passing flow is not a new idea. The difference of this work is the employment of graph gradient method. The idea of graph gradient seems to be directly from previous work (Hodge laplacians on graphs, 2015). However, it's lack of critical details about how to get the gradient of node features as the difference of node features of end points in single edge, and how to adapt to the gradient gating module for the graph gating method. 

2. From the main content, it's still difficult to conclude a strong connection between over-smoothing problem and the gradient gating method. Before dividing into the theoretical analysis,  the gradient gating can adaptively control the message passing flow with information gathering from the difference of node features between central node and its neighbors. While, it's not easy to draw the conclusion because dynamic changes of the aggregation weight over the neighbors is a common property of many GNN methods. Until the theoretical part, some clues induced from the theoretical upper bound of Dirichlet energy state give an alternative way to help dig more insights about the proposed gating method. Maybe authors should consider to move some conclusions from the theoretical analysis ahead to the main content (Section 2).

3. The experimental analysis mainly focuses on the classification performance. It misses important experiments to show the capacity of overcoming over-smoothing problem. I strongly suggest authors to show the performance of the proposed method along with the increasing number of GNN layers, and the employ the metrics from pioneering work [1] to validate the proposed method. Besides, it should consider more baseline which also applies gating mechanism to the GNN layers, such as but limited to GaGN [2] etc. 

References:
1. Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View, AAAI 2020.
2. GaAN: gated attention networks for learning on large and spatiotemporal graphs, UAI 2018.
clarity,_quality,_novelty_and_reproducibility: The presentation of the proposed method is easy to follow, but the connection between theoretical part and methodology is the weak point. 
summary_of_the_review: Overall it's a good work along with both methodology and theoretical analysis. But it still has certain space for further improvement.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_f9D8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a general module that alleviates the over-smoothing problem and allows the design of deep GNNs. The theoretical results are useful.
strength_and_weaknesses: Strength:
- The paper proposes reasonably theoretical results.

Weakness:
- The proposed method is not well motivated. The paper shows that "However, it is reasonable to expect that in realistic graph learning tasks one can encounter multiple rates for the flow of information (node updates) on the graph". I hope the paper can include any example showing that multi-rate learning is necessary.
- The proposed idea, shown in Figure 1, reminds me of the GRU layer. I hope the authors can clearly point out the similarity of ideas in the paper, and discuss why the proposed idea is different/novel.
- The paper does not provide a sufficient description of the methods. There is no description of the number of layers, hidden dim size, number of trainable parameters, etc. of the proposed and baseline methods, making the experimental results hardly convincing.
- I hope the paper can at least include experiments on some 
clarity,_quality,_novelty_and_reproducibility: The paper is relatively clearly written. 
The paper is hardly reproducible, given the lack of descriptions of full architecture/hyperparameters for the proposed method and the baselines.
summary_of_the_review: Overall, the paper proposes a GRU-like module for alleviating the over-smoothing problem for GNN. The proposed idea lacks sufficient motivation. Although the paper provides reasonable theoretical results, the proposed method is not convincingly evaluated. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3998/Reviewer_uYz6
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose a method for gating the updates in message passing graph neural networks. The key idea is to measure the similarity of a node's representation with its neighbors, and to reduce the update magnitude when they become very similar (i.e., oversmoothing happens). The authors present theoretical and experimental evidence of the effectiveness of their approach to tackle the oversmoothing problem in GNNs. 
strength_and_weaknesses: Strengths:
* The method achieves strong experimental results on a variety of tasks and datasets.
* The method is generally well-motivated and clearly explained.
* The authors present theoretical justification of their method.

Weaknesses (more details in the next section):
* Some of the experiments do not convincingly evaluate the described properties.
* There is relatively little ablation analysis of the proposed model.
* There is a plethora of methods claiming to solve the oversmoothing problem. The authors do not explain why their method should be better than existing ones.

clarity,_quality,_novelty_and_reproducibility: **Clarity**: very good; the presentation is clear and easy to follow.

**Quality**: the model is theoretically justified and obtains strong experimental results. 

**Novelty**: while the idea of gating is not new and many methods to solve the oversmoothing problem have been proposed, the model presented by the authors appears sufficiently novel.

**Reproducibility**: Good, as the descriptions are sufficiently clear and the authors provide their implementation.

**Detailed comments**: 
Regarding the first weakness: Table 1 evaluates the model's ability to perform multi-scale node-level regression. However, the datasets studied are heterophilic datasets that are also studied in the detailed experiments in Table 2. So one explanation of the results could be simply that the model performs well on these particular (heterophilic) datasets, and not necessarily the fact that it does well on multi-scale prediction. To properly evaluate this, the authors could compare to a method that performs well on Squirrel/ Chameleon but does not as good on the multi-scale prediction.

Regarding the second weakness: Some more ablation analysis is needed in my view. For instance:
* Is the second GNN in Figure 1 required? Why not compute the local graph gradients on the representations coming from the GNN F instead of F_hat?
* How does the gradient gating perform without the multi-rate component? A simpler architecture could simply average the P-norms of the differences per node and thus get a single gating score per node instead of one per node and feature dimension.

The last point is related to the claim in the paper:
> We observe that in (1), at each hidden layer, every node and every feature channel gets updated with exactly the same rate. However, it is reasonable to expect that in realistic graph learning tasks one can encounter multiple rates for the flow of information (node updates) on the graph. 

First, the claim that "in realistic graph learning tasks one can encounter multiple rates for the flow of information" is not clear to me and not justified. Clearly showing that having multiple rates per feature channel yields benefits over just one rate per node could bring some empirical justification for this.

Another concern I have is the large error bars in Table 2. In many cases, the difference between the leading models is much smaller than the error intervals. How can we claim that one method outperforms the other in such a scenario?

In addition, the authors only compare their method on a single homophilic dataset (genius) according to their own definition of homophily levels of <0.3 are considered heterophilic. Given that most GNNs are made for homophilic datasets, there should be a better comparison in this setting given the claim ghat G^2 improves performance on both homophilic and heterophilic datasets.

summary_of_the_review: Overall, I think the method is interesting and promising enough to warrant publication if my concerns from the detailed comments are adequately addressed. Currently, the evaluation is not rigorous enough to support all of the authors' claims about their method.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper