Paper: Graph Learning with Distributional Edge Layouts (ICLR-2024)

Reviewer comment (Experiments):
The evaluation is limited to smaller datasets (max 4k nodes). However, larger datasets such as QM9 and ZINC contain over 100k graphs. Showing performance increases on these datasets can further validate the method.

Author response:
In response to your concern regarding the evaluation's limitations, we appreciate your feedback and have made enhancements. The evaluation now incorporates results from the large-scale ogbg-molhiv dataset [1], presented in Table 10 on Page 20. We observe consistent performance enhancement on this dataset, aligning with existing results. This additional evaluation would provide more comprehensive and accurate support.
| ogbg-molhiv               | Valid AUROC           | Test AUROC            |
|---------------------------|-----------------------|-----------------------|
| GAT                       | 77.67 $\pm$ 2.78      | 74.45 $\pm$ 1.53      |
| GAT (DEL-F)                | **78.99 $\pm$ 1.69** | **75.27 $\pm$ 1.32** |
| Graph transformer         | 78.20 $\pm$ 1.18      | 76.51 $\pm$ 0.93      |
| Graph transformer (DEL-F)  | **80.12 $\pm$ 1.39** | **76.86 $\pm$ 0.95** |
| GPS                       | 80.08 $\pm$ 1.39      | 76.71 $\pm$ 1.12      |
| GPS (DEL-F)                | **82.57 $\pm$ 0.95** | **76.93 $\pm$ 1.00** |

Reviewer comment (Evaluation):
I noticed that the computational efficiency is sub-optimal, requiring ~0.1 to 2 seconds per graph. Note that while preprocessing can save training time, it does not save inference time, which is more critical practically.

Author response:
In real-world applications, computing the graph layout for real-time test data does incur additional overhead. However, this overhead can often be mitigated through distributed computing. It's noteworthy that we report a minimal amount of preprocessing time using CPU threads (one or eight). For datasets other than D\&D, our processing speed for a single graph is typically around 0.1 seconds, while for D\&D, it is approximately 1 second. Given that the average number of nodes in existing graph classification datasets generally does not exceed that of D\&D, our method remains feasible in the majority of scenarios.

Reviewer comment (Theory):
Lack of Theoretical Analysis: The paper could benefit from a more in-depth theoretical analysis of DEL. While the empirical results are promising, a theoretical framework that explains why DEL works and under what conditions would provide stronger support for the proposed approach.

Author response:
* Theoretical Analysis: Layout algorithms are seldom explored in graph representation learning, and proposing a comprehensive theoretical framework falls somewhat beyond the scope of this article. Nevertheless, DEL holds the potential to connect with certain theoretical frameworks. For instance, in Figure 7 on page 18, it is observed that DEL readily recognizes edges connecting different connected components. This observation could potentially be linked to spectral theory, as such edges often exert a significant influence on spectral distribution [1]. We will investigate more on the theoretical aspect in our future work.

Reviewer comment (Evaluation):
Sampling Layout Numbers: While the study addresses the impact of the number of sampled layouts on the performance, the explanations and discussions in this section are somewhat brief. More detailed insights into how the number of layouts affects the performance and potential trade-offs would enhance the paper's depth.

Author response:
* Sampling Layout Numbers: The determination of the number of sampling layouts is contingent upon the characteristics of specific downstream tasks and datasets. In practical terms, ensuring an adequate number of sampling layouts is essential for the model to capture the potential energy distribution of the graph accurately. However, an excessively high number of layout samples can result in information redundancy and low efficiency, thereby impacting the performance of its practical use.

Reviewer comment (Theory):
Computational Complexity: The paper discusses the computational complexity of DEL, but there is room for a more comprehensive discussion, particularly concerning scalability. Discussing the implications of computational complexity in real-world, large-scale applications is crucial.

Author response:
* Computational Complexity: In real-world applications, graph sizes for graph classification tasks typically do not exceed those of the D\&D dataset. If the graph size is stable, the preprocessing overhead of DEL is linearly correlated with the number of graphs. The reported preprocessing times in the paper are based on the utilization of either 1 or 8 CPU threads; employing additional threads would further reduce preprocessing time.  Furthermore, for more efficient handling of node classification tasks of large-scale graphs, we can also combine DEL with some more efficient large-scale graph layout algorithms which capable of accommodating millions of nodes [3].

Reviewer comment (Theory):
Why DEL works and under what conditions would provide stronger support for the proposed approach?

Author response:
* DEL acquires global information, not attainable by MPNN through the sampling of the layouts based on potential energy distribution. This unique capability positions DEL as a potent enhancer of graph representation learning. In Appendix D, we present some initial analysis, reserving the construction of a more solid and comprehensive theoretical framework for future research.

Reviewer comment (Evaluation):
Why does DEL perform better in lower dimensions, and what does this mean for practical applications?

Author response:
* Though there is not a ready theory interpreting why 2D and 3D layouts are sufficient to deliver good performance, we can regard graph spectral theory as an analogous counterpart. In graph spectral theory, the majority of Laplacian eigenvalues for a graph are very close to 0, while minor eigenvalues far from 0 are sufficient to depict the overall structure of the graph. This surprisingly aligns with the observation that 2D/3D layouts are adequate for capturing the main structure of the entire graph. Therefore, lacking specific prior knowledge, we prioritize performing DEL in a low-dimensional space. Simultaneously, DEL sampling multiple layouts in a low-dimensional space can further assist DEL in learning a more accurate energy distribution of the graph, leading to performance improvement. However, these are initial analyses, and more solid and comprehensive theoretical analyses will be left to future work.