Paper: Graph Neural Networks Gone Hogwild (ICLR-2024)

Reviewer comment (Experiments):
I think the paper only considers distributed inference, but does not deal with distributed training. I am not sure I understand why one would want only the inference to be distributed if the training is not. More precisely, if the graphs are so large that inference needs to be distributed, how was the model even trained?

Author response:
Thank you for your time in reviewing our work. We believe there may have been a misunderstanding about the focus of the paper. While inference over large graphs is one application where asynchronous, distributed inference may be desired, there are a number of other applications on small graphs (i.e. multi-agent setting). In this work we point out large graphs as a possible application, but do not focus on this per se. Our synthetic experiments are particularly motivated by tasks that would be relevant to multi-agent systems (e.g. counting of number of agents, summing agent features, performing localization). In these applications, asynchronous and distributed inference is absolutely desirable, as it has the benefits of not requiring central control, which introduces algorithmic and communication overhead as well as a central point of failure for the system. 

Furthermore, it is entirely possible that distributed inference might be desirable even if the GNN was trained in a non-distributed manner. Training and inference can have vastly different resource requirements.  Models are only trained once and so that can be done in a centralized way if desired, but inference will be done indefinitely and is constrained by whatever device(s) the network will be deployed to. To use a modern example of this discrepancy: although it cost tens of millions of dollars for OpenAI to train ChatGPT, that cost is exceeded by a single week of providing inference as a service with the model. Finally, distributed inference and distributed training are essentially independent problems. For instance, for extremely large graphs we could have trained using an existing distributed training method. 

We also point out that while the maximum graph size in our synthetic experiments is 100 nodes, we also run on the PPI dataset, where the average graph contains over 2000 nodes.

Reviewer comment (Experiments):
The experimental section is limited and tends to be not very clear. I think it should explore much larger graphs having millions of nodes, and certainly not those having 100 nodes that can be easily ran by standard non-distributed GNNs. For example, what about the ogb MAG240M dataset?

Author response:
Thank for reading through our reply, and for the response. 

We've uploaded a revised version of the paper where we have de-emphasized the large graph application and instead focused on the synthetic tasks motivated by multi-agent systems, to be more in line with the experiments we present. While inference over large graphs remains a glaring example of a relevant application for asynchronous GNNs, we believe this exploration should be left to future work (particularly since this would require engineering a distributed training pipeline that is amenable to energy GNNs, where a linear system depending on all node embeddings has to be solved in the backward pass to obtain gradients).

In our revised paper, when we introduce the experiments we explain our choice for comparisons against other models, so it is more clear why comparisons are limited. We emphasize again that our non-synthetic experiments are limited because we focus on tasks which require distributed, asynchronous inference (since this is the main focus of the paper); the benchmark tasks don't really belong to this group. This is also why we report performance on benchmarks in the appendix rather than in the main paper.

Regarding the evaluation process for MUTAG and PROTEINS; we are aware of the evaluation strategy of [1], and point out that the values reported for IGNN do not follow this strategy. Rather than taking the mean performance across all folds for each epoch, and reporting the max mean performance, the authors of IGNN appear to first take the max performance for each fold, then report the mean of the max performance.

Reviewer comment (Experiments):
Please evaluate on large scale datasets, e.g., MAG240M, where distributed executions are needed.

Author response:
Thank for reading through our reply, and for the response. 

We've uploaded a revised version of the paper where we have de-emphasized the large graph application and instead focused on the synthetic tasks motivated by multi-agent systems, to be more in line with the experiments we present. While inference over large graphs remains a glaring example of a relevant application for asynchronous GNNs, we believe this exploration should be left to future work (particularly since this would require engineering a distributed training pipeline that is amenable to energy GNNs, where a linear system depending on all node embeddings has to be solved in the backward pass to obtain gradients).

In our revised paper, when we introduce the experiments we explain our choice for comparisons against other models, so it is more clear why comparisons are limited. We emphasize again that our non-synthetic experiments are limited because we focus on tasks which require distributed, asynchronous inference (since this is the main focus of the paper); the benchmark tasks don't really belong to this group. This is also why we report performance on benchmarks in the appendix rather than in the main paper.

Regarding the evaluation process for MUTAG and PROTEINS; we are aware of the evaluation strategy of [1], and point out that the values reported for IGNN do not follow this strategy. Rather than taking the mean performance across all folds for each epoch, and reporting the max mean performance, the authors of IGNN appear to first take the max performance for each fold, then report the mean of the max performance.

Reviewer comment (Presentation):
Why some entries of the tables have the std while some others have not?

Author response:
- Why some entries of the tables have the std while some others have not?

A standard deviation is always included for running GCN asynchronously, since under asynchronous execution the predictions vary across random runs. We perform 10 random runs to obtain these standard deviations. For IGNN and energy GNN, performance does not depend on the run since these architectures guarantee convergence of embeddings under asynchronous execution, so no standard deviation is provided. For the 10 node counting experiment, the dataset is extremely small and consists of only 10 graphs, so we report average performance across 10 folds of the data. Finally, for the MUTAG and PROTEINS datasets, we report mean accuracy and standard deviation across 10 splits of the dataset. For all other table entries where standard deviation was not included, it is either because we report performance on a single test set from the dataset, or because the architecture gives the same predictions under asynchronous execution.

That being said, we have decided for all experiments to run across 10 folds, as well as with random network parameter initializations. After this, standard deviations will be included in all table entries. We will have updated results uploaded to the paper by end of day Friday.

Reviewer comment (Writing):
Some GNN related terminology is difficult to follow, due to not having explicit definitions.

Author response:
We agree that several of the terms we used could have been clearer, and may have lead to confusion. Regarding each of the terms you mention:

- Finite depth GNNs are GNNs that have a fixed number of layers of message passing. Infinite depth GNNs are defined in the background section.
- We use the term “contractive” short for “contraction map” throughout the paper as an attribute of functions (e.g., “update functions” in the abstract, “message passing function” in 2.3, etc.). A function $f$ is a ***************contraction map*************** with respect to its argument, which is to say $\exists \mu \in [0, 1)$ such that $||f(x) - f(y)|| \leq \mu ||x - y||$ for any $x, y$ in the domain of $f$. Informally this means that the distance between any two points *********decreases********* after we apply the function. In the context of GNNs, the argument in question is the node embedding (which we denote with $h$ in this work).
- We use the term “energy function” to refer to the objective function associated with the optimization problem. In the first two paragraphs of section 5 we explain the optimization-based view of GNN node updates and proceed to call the objective function being optimized an ‘energy function’
- We do use implicit GNN and fixed point GNN somewhat interchangeably, which is not technically accurate. Implicit GNNs (as a concept) refer to GNNs in which the mapping from data (features) to node embeddings is given implicitly rather than explicitly. Optimization based GNNs and fixed point GNNs are both implicit GNNs in this broad view. In our evaluation section, we are overloading “IGNN” to mean the specific architecture presented in the paper “Implicit GNNs” Gu et al. 2020.

We have made several edits to the main paper to attempt to clarify this terminology.

Reviewer comment (Writing):
Section 4 last paragraph is difficult to understand.

Author response:
Regarding the final paragraph of section 4, it may be clarifying to provide an example. Suppose the input graph has dynamic/time-varying node or edge features; for instance, a sensor network where where the node features correspond with sensor measurements (which change over time). Consider the case of using a finite-depth GNN to generate predictions. At some time $t$, a forward pass is executed, in which each node performs a fixed number of rounds $L$ of message passing to generate predictions. If at some time $t’ > t$ the features of the nodes or edges change, the node predictions will now be outdated. To update predictions, another forward pass of the network needs to be coordinated. Coordinating with a global controller is non-trivial, fragile, and adds communication overhead and bottlenecks. Distributed coordination amongst the nodes is similarly non-trivial. The implicit GNN operates continuously, repeatedly updating its embedding, and changes in feature inputs are handled ‘for free’ in the sense that the network does not need to re-start a forward pass. This feature is not necessarily important for GNNs in general, but it certainly is for applications where we want distributed GNN inference and the graph features are dynamic inputs. We do not consider any applications where this is relevant but hope to pursue this in future work.

Reviewer comment (Writing):
In Section 4, the last paragraph explains the advantages of a fixed-point GNN over finite depth GNNs, but it is unclear what the benefit is due to insufficient explanation. For example, what is meant by “change its output in response to changes in the inputs” and “coordinating another forward pass of the network,” and why is this important for GNNs?

Author response:
Regarding the final paragraph of section 4, it may be clarifying to provide an example. Suppose the input graph has dynamic/time-varying node or edge features; for instance, a sensor network where where the node features correspond with sensor measurements (which change over time). Consider the case of using a finite-depth GNN to generate predictions. At some time $t$, a forward pass is executed, in which each node performs a fixed number of rounds $L$ of message passing to generate predictions. If at some time $t’ > t$ the features of the nodes or edges change, the node predictions will now be outdated. To update predictions, another forward pass of the network needs to be coordinated. Coordinating with a global controller is non-trivial, fragile, and adds communication overhead and bottlenecks. Distributed coordination amongst the nodes is similarly non-trivial. The implicit GNN operates continuously, repeatedly updating its embedding, and changes in feature inputs are handled ‘for free’ in the sense that the network does not need to re-start a forward pass. This feature is not necessarily important for GNNs in general, but it certainly is for applications where we want distributed GNN inference and the graph features are dynamic inputs. We do not consider any applications where this is relevant but hope to pursue this in future work.

Reviewer comment (Experiments):
Only a total of 3 GNN architectures are tested in the experiments. Having more architectures or explaining why the chosen architectures are sufficient would be beneficial.

Author response:
We understand that the motivation behind our choice of architectures to compare to in experiments may not have been clear in the paper. 

- For the synthetic tasks we are concerned with performance during asynchronous, distributed inference. We only consider one ‘finite-depth’ or synchronous GNN (a 2 layer GCN) because all GNNs of this flavor will exhibit the same malignancies under asynchronous inference. We compare against IGNN, because to our knowledge this is one of the only existing implicit GNN architectures which converges under asynchronous inference. EIGNN is another instance that we mention (which we note is extremely similar to IGNN), but we found in our experiments that performance of their architecture depends on solving for the fixed point exactly rather than using an iterative method (see appendix 10); the forward pass thus requires global information, which is not easily amenable to distributed inference. Due to global communication requirements of EIGNN’s forward pass and its similarity to IGNN, we exclude comparisons against EIGNN. Another architecture we mention is the implicit GNN architecture proposed by Scarcelli; however, as we note in the paper, their node update function is not a contraction map over the whole domain and thus does not share the same convergence guarantees as EGNN and IGNN.
- For the real world experiments we only compare against our implementation of GCN and IGNN; however, we do provide values reported in other papers for other architectures. We asterisk these values in our tables because their evaluation metrics are biased to overstate the typical-case performance. Inspection of the code corresponding to these papers shows that test performance is computed by creating 10 train/test splits, and averaging the *best* test performance achieved during training. For our results (not asterisked) we measure mean test performance across 10 splits after a consistent number of epochs that is not determined by test performance for that split, so the value we report are strictly lower than that in the other papers. We could have added a more comprehensive comparison, but as our goal was not to show SOTA performance, we believe the context we provide for performance of other methods was sufficient.

Reviewer comment (Evaluation):
It is unclear if the proposed energy GNN always performs best, or whether there could be cases where it fails to perform well.

Author response:
Furthermore, in all experiments we see the proposed energy GNN performs best, but is this always the case and could there be cases where energy GNN fails to perform well?
    - Certainly there are cases where energy GNNs will fail to perform ‘well’, in the sense that other GNN architectures may perform better (this is certainly the case for our real-world dataset experiments; while we achieve competitive performance with existing architectures, we do not achieve SOTA performance). However, most architectures are not robust to asynchronous inference; energy GNNs are. What we attempt to show in our experiments is that energy GNNs both have this property and are reasonably performant.

Reviewer comment (Experiments):
The convergence curves of Energy GNN are not reported.

Author response:
Regarding your concerns about providing convergence curves for energy GNN: do you mean at inference time, or at training time? We provide curves demonstrating the convergence of the log dataset loss during asynchronous energy GNN inference for the chains experiment (in Appendix A.7, Figure 2). While we can certainly provide these curves for other experiments, the purpose of Figure 2 is to illustrate empirically that energy GNNs (and IGNNs) are robust to asynchronous inference, supplementing the theoretical guarantees for convergence that we show hold in appendix A.4. If the reviewer instead means that training curves are not provided, we can certainly provide those.

Reviewer comment (Experiments):
What's the performance of GCN (async) on real-world datasets?

Author response:
GCN (and other architectures assuming synchrony across layers) does not provide reliable predictions under asynchronous execution. If reliable predictions are desired, these GNNs are automatically disqualified. When we perform experiments with real-world datasets, we momentarily ignore asynchronous inference, and instead provide an initial evaluation of how expressive/powerful energy GNNs are relative to other GNNs on a level (synchronous) playing field.

Reviewer comment (Presentation):
It would be great if the authors can provide a similar histogram in Figure 1(d) for Energy GNN.

Author response:
Energy GNNs, given enough iterations and small enough step size, always converge to the same value (within some numerical tolerance). The histogram in this case would just be a single peak (again, within some numerical tolerance).

Reviewer comment (Reproducibility):
Some reference links in the supplementary material are broken.

Author response:
Fixed in new upload, thank you for pointing this out.

Reviewer comment (Presentation):
The citation format is not consistent. Some first names are abbreviated (e.g., A Bojanczyk) but some are not.

Author response:
Fixed in new upload, thank you for pointing this out.