Paper: On the Stochasticity in Graph Neural Networks (ICLR-2024)

Reviewer comment (Theory):
There are rooms for improvement on the statements of Theorems 3 & 4: The statements do not directly tell us that oversmoothing and oversquashing can be alleviated. Both contain inequalities between the one with stochasticity and the original one. If the difference is marginal, as the number of layers $K$ increases, this would not affect the improvement. For example, if the values of the two are $1/2$ and $1/3$, then $(1/2)^K \approxeq (1/3)^K$ even for the moderate $K$.

Author response:
To address your concern, we have derived the Jensen's gap analytically for Theorem 3, which is related to the variance of the edge distribution---the higher variance correspond to slower decay in Dirichlet energy, which we also illustrate in Figure 2. With a sufficient structural noise, the difference is considerable and amplified as the number of layer increases. On the other hand, the Jensen's gap for Theorem 4 can also be related to the moments of the edge distribution, albeit the derivation is less intuitive.

Reviewer comment (Theory):
Technical contributions seem to be rather limited: The proof that the authors relied on is the convexity of the activation functions (or that of the first derivative of the activations), which can be limited in practice. For example, sigmoid and tangent hyperbolic functions are not convex.

Author response:
We have rewritten Theorem 3 and 4 to cleanly separate the roles of activation functions---they are not necessary conditions, as no activation or identity activation also suffices. On the other hand, concave activation functions like `sigmoid` and `tanh` might indeed break these theorems, so we included "convex activation functions" as one of the design principles in Section 4 (Architecture).

Reviewer comment (Theory):
The stochasticity assumption on the case where edge strength can be perturbed, but no edge is added in the original adjacency matrix, is bit limited.

Author response:
As stated in the Limitation section, the scope of this paper only entails perturbation of edge weights, with no added edge, as that would require more assumptions on the nature of the graph which is not always available, as Zhang et al. 2018 has shown.

Reviewer comment (Writing):
It would have been nice if the assumptions that the authors made is presented in a clearer manner (e.g., by explicitly put "Assumption" separately from the normal texts or Theorem/Lemma statement).

Author response:
We have reworked the presentation of the theorems so that the assumptions are more clearly stated at the beginning of the theorems.

Reviewer comment (Writing):
p-2 $A[:,v, v] \sim q(Z)$: Does this mean that for each feature (i.e., for each column) $A[v, v]$ only becomes zero? or the entire row $A[v, :]$ or column $A[: v]$ become zero?

Author response:
The later. We have rewritten the confusing paragraph.

Reviewer comment (Writing):
p-5: Does $K$ indicate the number of layers?

Author response:
Yes! We have clarified.

Reviewer comment (Writing):
p-5 in the proof of Theorem 2: Does $\mathcal{I}(\cdot)$ indicate the indicator function?

Author response:
Yes! We have made this clearer in the text now.

Reviewer comment (Presentation):
the paper is 'slow,' the contributions start from page 4

Author response:
We have overhauled the structure of the paper, with compressed introduction and preliminary sections and a new theoretical presentation framework, where we interweave the problems statements with theoretical evidence. Please let us know if you believe that this has improved the flow and readability of the paper.

Reviewer comment (Theory):
the theoretical result while being nice are not enough to address GNN issues

Author response:
We have enriched our theoretical framework with new insights, including the derivation of Jensen's gap for Theorem 3 and the separation of convexity requirement for activation functions. We believe that these new insights justifies the empirical performance of stochasticity-incorporating GNNs and provides new guidelines for stochastic GNN design. We would be grateful if you would elaborate on why you think the previous results are not enough so we can further improve the manuscript to address your concerns.

Reviewer comment (Experiments):
The paper lacks a comprehensive comparison with existing stochastic GNN models, limiting the understanding of how BRONX performs in relation to other approaches.

Author response:
Thanks for pointing this out. We have now emphasized in the paper that, BRONX treats the edges *individually* and assigns different distributions to different edges, which, to the best of our knowledge, is entirely novel and allows transductive learning.  
All previous methods samples the prior distribution for all edges, making transductive learning impossible.

Reviewer comment (Experiments):
Could you provide a more comprehensive comparison of BRONX with existing stochastic GNN models to better understand its performance and advantages?

Author response:
Thanks for pointing this out. We have now emphasized in the paper that, BRONX treats the edges *individually* and assigns different distributions to different edges, which, to the best of our knowledge, is entirely novel and allows transductive learning.  
All previous methods samples the prior distribution for all edges, making transductive learning impossible.

Reviewer comment (Writing):
Could you explain what "$\mathcal {I}$" means in equation (20)?

Author response:
It is the indicator function. We have now clarified this. Thanks again for raising this.

Reviewer comment (Evaluation):
Could you discuss any potential limitations or drawbacks of the BRONX framework? Addressing these challenges would enhance the understanding and applicability of the approach.

Author response:
We have reworked the limitations sections to reflect the limitations and future directions.

Reviewer comment (Experiments):
It would be beneficial if you could discuss the scalability of the BRONX framework and its performance on larger datasets. This would provide a better understanding of its practical utility.

Author response:
We have now included a complexity analysis---the runtime and storage complexity scales linearly with regard to the number of edges, making it feasible for large systems.

Reviewer comment (Reproducibility):
It would be helpful if you could provide more detailed explanations and examples of the practical implementation of the BRONX framework to assist readers in applying it in real-world scenarios.

Author response:
We have now expanded the Details section in the appendix. The python package is also distributed open-source.

Reviewer comment (Theory):
There may be some minor errors in the Equation (29), which can be corrected.

Author response:
Thank you! We have now corrected the errors!