Paper: Flood and Echo: Algorithmic Alignment of GNNs with Distributed Computing (ICLR-2024)

Reviewer comment (Reproducibility):
The flood and echo mainly focused on the forward propagation part. But how are its gradients computed? To compute the gradient for weight parameters, we have to use the “gradient with respect to hidden representations output” and “the input node representation that multiplied to the weight parameters”. In flood and echo, it seems like we have to save the hidden embeddings for each node to compute gradient?

Author response:
Note that the main difference lies in the node activations order, but the graph convolutions and computations still stay the same in principle. The main difference is that they happen more sparsely and in a fashion that follows the flooding and echo wave pattern. In practice, the mechanism is implemented in pyg which then makes use of automatic differentiation to get the gradients. The embeddings which need to be stored to compute the gradients are the same as for the regular MPNNs.

Reviewer comment (Theory):
Compared to ordinary message passing GNN, the major difference is the selection of neighbors and the number of propagation steps (aka the number of layers in GNN). In order to propagate information to the full graph, the proposed “flood and echo” requires twice the propagation steps of ordinary message passing GNN due to echo back. I am not sure I understand this correctly and how this can benefit in terms of efficiency.

Author response:
So if information should be propagated throughout the entire graph and back, then also a regular GNN would need the same number of propagation steps. The Flood and Echo Net only activates a subset of nodes at each propagation step, so even though the number of propagation steps (in terms of layers) is the same, the number of node updates is significantly less (each node is activated a constant number of times instead of O(D) times) and so is the number of messages required to achieve this.

Reviewer comment (Theory):
The "flood" part seems to me is the forward propagation of ordinary GNN, where this GNN's depth is the diameter of the graph?

Author response:
Both the flood and echo are always part of the computation - we have added a new figure in the Appendix A which shows how nodes are activated in the forward pass for both the flood and echo part. A GNN which executes diameter many layers would have a similar reach but with a completely different computation tree (with more updates to every node). We hope this clarification was already helpful, we are not entirely certain what causes the confusion.

Reviewer comment (Writing):
What does phases stand for on line 5 in Algorithm 1?

Author response:
One execution of a flood followed by an echo is seen as one phase. The Flood and Echo Net could execute multiple such phases.

Reviewer comment (Reproducibility):
The final node representation might be different if the flood start node is chosen randomly each time?

Author response:
The flood and echo mechanism is always the same, but there are different modes, one of which we present here is the random start. Depending on the different starting nodes the individual node representations might indeed differ.

Reviewer comment (Reproducibility):
In the paragraph above Section 4, the authors said “in every run, we only keep the node embedding for the chosen start node”, but why? If we need to compute the embeddings for all nodes in the graph and compute its gradient, shouldn’t we save this for all nodes?

Author response:
Again, the mechanism stays the same but in the “all” mode our goal is to only derive the embedding of the node that is marked as the starting node. To save computational effort, we could instead apply the mode with a fixed start where we keep all node embeddings after an execution. One could even think of a mix between the two strategies or come up with more modes in future work.

Reviewer comment (Experiments):
The authors evaluate on highly specific benchmarks which seem to be extremely aligned to the Flood and Echo net. It would be more valuable if the authors evaluated on existing algorithmic benchmarks from the field of Neural Algorithmic Reasoning [3].

Author response:
> Regarding W3. Q3) Would the authors be able to provide a reason for why they did not evaluate on existing algorithmic benchmarks such as CLRS-30?

Thank you for pointing out the CLRS-30 benchmark. We would like to test and compare our architecture on the NAR problems that they study in future work. However, currently we see some issues on why we can not apply the Flood and Echo Net. The main obstacle we see is the graph representation of the problems in CLRS. They utilise fully connected graphs as base for their computation, which one side defeats the purpose of Flood and Echo which tries to scale the number of messages sub quadratically. On the other hand, in a fully connected graph, the flood and echo would perform a single activation of all nodes and then echo back. The strengths of the Flood and Echo Net are to exchange information to far away nodes and the mechanism still efficiently generalises to larger graphs - this just does not really make sense if the graph is fully connected as nodes are directly connected and if the graph grows quadratically for larger sizes it cannot be efficient anyway. Lastly, we do hope that we can incorporate some comparisons in the future, as certainly the graph problems which are part of CLRS could be adjusted to make appropriate comparisons. However, as of now there are both conceptual and technical hurdles to overcome in order to perform comparisons in a straightforward way.

Reviewer comment (Novelty):
The underlying idea is not new in many perspectives. First, it is studied in rooted subgraph based GNNs, along with distance-to-root feature usage. One can refer to Bohang Zhang's ICML 23 paper. Second, it shares certain similarity to "Agent-based Graph Neural Network". In principle, I feel the author mainly provide another angle of viewing it as kind of execution mechanism of GNNs.

Author response:
We had a closer look at the mentioned paper, which defines Subgraph GNNs the following way: Subgraph GNNs unifiy the view of GNNs which not only rely on their immediate neighbourhood but across different elements, i.e. which one can capture with subgraphs (multi hop neighbourhood or incorporating distance information). 
However, I fail to see how exactly one could express the Flood and Echo Net in this framework. Note that in our work, at each given timestep, only a subset of nodes is currently active and updates their node embeddings. Because these activations happen one after another based on the distance to the starting point, this is similar to a wave pattern. Moreover, because these activations (and node updates) happen one after another, information can propagate through the graph following the pattern. The mechanism of a node A which is closer to the starting point updating itself before a node B that is farther away (the updated A embedding might even have an influence on B) is central to the idea of the Flood and Echo Net.  We have added an additional Figure in the Appendix A to highlight the execution mechanism more clearly.

Currently we do not see any way of expressing the same dynamics in the framework of subgraph GNNs - and therefore, I believe they are not that closely related. However, we have added an extended paragraph about the connection in the Appendix which discusses subgraph GNNs.

We agree that the paper shares some similarity to Agent-based Graph Neural Networks (AgentNet) in the sense that we propose a different computation mechanism compared to more standard GNNs. Which is also a reason why we mention the paper in our related work section. However, there are important key differences between the two proposed methods. In AgentNet there are a fixed number of agents which traverse the graph structure similar to a learned random walk - whereas in Flood and Echo Net, all nodes (not only where an agent is currently active) perform computations determined by the wave activation pattern.

We think thatFlood and Echo Net goes beyond “another angle of viewing” as it defines a very specific computation flow which is inherently different from standard MPNNs and other execution paradigms such as the aforementioned AgentNet.

Reviewer comment (Theory):
The expressivity study/proof is technique-wise simple. Also the author need to discuss the expressivity comparison inside permutation equivalent setting. We don't discuss expressivity for permutation sensitive model as it can achieve universality easily. Hence here the author should focus on Flood And Echo All when discussing expressivity.

Author response:
We understand the concerns of this particular point. Note, that the execution mechanism for the Flood and Echo Net always stays the same, regardless of the chosen operation mode and our analysis still holds. Moreover, the proposed scheme does not break permutation equivariance, but introduces an additional tie-break in distinguishing the starting node, which is a lot weaker than universality (i.e. by including random features). Finally, note that we provide the expressivity experiments for the Flood and Echo All mode in Table 1. Even in the “all” mode it is more expressive - you can consider the example in Figure 2 where every node (no matter the starting node) can distinguish the two graphs.

Reviewer comment (Experiments):
The way of "backward and forward" "wave" technique is also used in another paper "A Practical, Progressively-Expressive GNN", where they proved that doing this kind of wave back and forth multiple times towards convergence is the most powerful one. I'm wandering whether the author tested doing the flood and echo multiple times.

Author response:
Thank you for pointing out this work, we were not aware of it and have included a reference in our own work. Indeed it is possible to execute multiple phases of the Flood and Echo Net. In fact, we execute 2 phases in the experiments presented in the extrapolation section which we also mention in the main text.

Reviewer comment (Experiments):
All this method shares similarity to subgraph GNNs, I'm wandering whether the author can provide some comparison of subgraph GNNs for these extrapolation tasks.

Author response:
Note that Flood and Echo Net operate on the original graph topology, which has the advantage that its mechanism is relatively simple and computationally inexpensive (compared to higher order message passing). Comparing subgraph GNNs which would consider a higher order neighbourhood might not be a really fair comparison. Moreover, the main reason why we did not incorporate this comparison is that the extrapolation experiments provably require the exchange of information over a distance of O(D). So if a subgraph GNN only considers its O(1) neighbourhood, I don’t think it will be able to learn anything sensible in these tasks. As a comparison we have included other baselines such as RecGNN which is aware of this issue and scales the number of rounds to overcome this limitation. If you have any concrete suggestion on a subgraph GNN which would still make sense in this context I am happy to have another look  to see if we can include them in our comparisons.

Reviewer comment (Reproducibility):
The implementation of the proposed framework might be complex as it is not supported by the existing graph learning frameworks such as DGL and PyG.

Author response:
The implementation of the proposed Flood and Echo Net might seem more complex at first than it actually is. The non-trivial part is to ensure the correct message flow, which can be done using proper masking so that the entire model can still run efficiently on the GPU. We have implemented the architecture in PyG and will release the entire code base upon publication.

Reviewer comment (Evaluation):
The computation is more expensive than GNN. Each round of full-graph traverse can only compute one node's embedding.

Author response:
Note that for all presented modes the execution of a Flood and Echo Net always computes embeddings for all nodes. The difference lies in the chosen starting point and what embeddings you want to keep for the final prediction. For some mode, such as the “all” mode, it might make sense to run an execution for every node as the start once and only keep its own embedding after the run. However, we want to stress that just from a computational point of view an entire phase of the Flood and Echo net is as expensive as a single round of a standard MPNN (using O(m) messages).

Reviewer comment (Experiments):
The idea of Flood and Echo is borrowed from distributed computing. Can we apply it to distributed GNN training/inference? What are the benefits we can expect?

Author response:
The Flood and Echo execution pattern only ever activates a few nodes at a time, maybe this could be leveraged during inference time for more efficiency. If you want to focus on distributed training/inference in the sense of scaling to multiple machines we do not see an immediate connection. In this work, we are mainly inspired by the algorithm design done in distributed computing. For distributed execution on multiple machines we think this concerns a different area of distributed computing.