Paper: Subgraph-To-Node Translation for Efficient Representation Learning of Subgraphs (ICLR-2024)

Reviewer comment (Experiments):
There are not many real-world datasets for the supervised subgraph classification. The authors should definitely consider synthetic datasets, e.g. those considered by Alsenter et. al. (2020). It is not clear why the authors have not considered such synthetic datasets.

Author response:
Thank you for your suggestion. We agree with your comment about synthetic datasets and have conducted experiments on them. The table below summarizes the performance of GCNII+S2N on synthetic datasets.

|     | Density    | Cut-Ratio  | Coreness   | Component   |
|----------------------|------------|------------|------------|-------------|
| GLASS        | 93.0 ± 0.9 | 93.5 ± 0.6 | 84.0 ± 0.9 | 100.0 ± 0.0 |
| SubGNN               | 91.9 ± 1.6 | 62.9 ± 3.9 | 65.9 ± 9.2 | 95.8 ± 9.8  |
| Sub2Vec              | 45.9 ± 1.2 | 35.4 ± 1.4 | 36.0 ± 1.9 | 65.7 ± 1.7  |
| GCNII / S2N+0        | 67.2 ± 2.4 | 56.0 ± 0.0 | 57.0 ± 4.9 | 100.0 ± 0.0 |
| GCNII / S2N+A        | 93.2 ± 2.6 | 56.0 ± 0.0 | 85.7 ± 5.8 | 100.0 ± 0.0 |
| GCNII / S2N+0 + RWPE | 74.8 ± 3.6 | 85.2 ± 5.1 | 56.1 ± 3.0 | 100.0 ± 0.0 |
| GCNII / S2N+0 + RWPE | 93.6 ± 2.0 | 89.2 ± 2.6 | 77.4 ± 9.1 | 100.0 ± 0.0 |

S2N+A outperforms the state-of-the-art (GLASS) on Density, Coreness, and Component datasets. S2N+0 shows the same performance as GLASS only in Component. For each dataset, the attributes that affect the subgraph properties (i.e., labels of synthetic datasets) are known (as illustrated in the table below) [1]. Because S2N compresses the global graph structure, it is challenging to learn Cut-Ratio, which requires exact information about the border structure (i.e., global structure). Learning Density and Coreness of subgraphs require their internal structures. Therefore, S2N+0, which does not maintain internal structure, relatively underperforms baselines.

| Density | Cut-Ratio  | Coreness  | Component     |
|--------------|---------|--------------------|-----------------|
| Internal structure | Border structure | Internal structure, border structure and position | Internal and external position |

To address this issue, we can add structural encoding to the input features, particularly Random Walk Positional Encoding (RWPE) [2]. The efficiency of S2N is maintained since the RWPE is computed once before training and only requires the memory complexity of $O(N)$. RWPE allows S2N to significantly improve the performance of Density and CutRatio, but not of Coreness. We interpret that RWPE for subgraphs can encode internal and border structures well but cannot encode border positions. We leave the development of structural encoding for S2N as future work.

Above all, tasks that depend only on specific attributes barely exist in the real world. We can find evidence from the high performance of S2N on real-world datasets. Our proposed methods provide GNN practitioners with an efficient solution for real-world subgraph-level applications. It is analogous to lossy compression of images (e.g., JPEG) to serve them efficiency-sensitive applications.

We will add all results and discussion to the revised paper.

[1] Alsentzer, Emily, et al. "Subgraph neural networks." Advances in Neural Information Processing Systems 33 (2020): 8017-8029.

[2] Dwivedi, Vijay Prakash, et al. "Graph Neural Networks with Learnable Structural and Positional Representations." International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
The authors do not clarify if the considered datasets are adequately large so that the GPU speed/memory really forms a bottleneck for learning.

Author response:
For the datasets we used, the number of nodes is 17K, 15K, 57K, and the number of edges is 317K, 3.2M, 4.6M (Table 4 in the Appendix). In the table below, we introduce some datasets referred to as large-scale in our GNN research community. The number of edges compared to these datasets is at a similar level; thus, similar scalability is required to model the datasets in this paper using GNNs. Particularly, when using attention models (e.g., GATv2), we cannot load the largest dataset using baselines into a single GPU of 11G VRAM.

| Name          | # Nodes | # Edges | Source |
|---------------|---------|---------|--------|
| Penn94        | 42K     | 1.4M    | [1]    |
| arXiv-year    | 169K    | 1.2M    | [1]    |
| twitch-gamers | 168K    | 6.8M    | [1]    |
| ogbl-ddi      | 4K      | 1.3M    | [2]    |
| ogbl-biokg    | 93K     | 5.4M    | [2]    |
| ogbn-arxiv    | 169K    | 1.2M    | [2]    |

[1] Lim, Derek, et al. "Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods." Advances in Neural Information Processing Systems 34 (2021): 20887-20902.

[2] Hu, Weihua, et al. "Open graph benchmark: Datasets for machine learning on graphs." Advances in neural information processing systems 33 (2020): 22118-22133.

Reviewer comment (Theory):
I am not sure about handpicking the Configuration Model (CM) as a justification for low computational complexity of S2N. Why has this model been picked: One could definitely study some other random graph models and ask the same questions?

Author response:
The complexity of S2N strongly depends on the distribution of translated edge weights. Thus, we need a random graph model that can analytically calculate the distribution of edge weights (i.e., the number of shared edges in two subgraphs). When using the configuration model (CM), the distribution of S2N's edge weights can be derived from the degree distribution of the global graph. This is possible because CM calculates the probability of edge existence through the degrees of a pair of nodes. Note that CM is frequently used in analytically calculating numerous network measures [1].

We also emphasize that CM only requires a degree sequence or a distribution. That means CM can also generate graphs generated by other random graph models. For example, when the degree distribution is Poisson distribution, CM generates graphs close to the Erdős–Rényi model. CM can also generate degree distributions with other distributions, for example, power-law distributions. See [2] for more details.

We will add this justification to the revised paper.

[1] Barabási, Albert-László. "Network science." Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371.1987 (2013): 20120375.

[2] Newman, Mark. Networks. Oxford university press, 2018.

Reviewer comment (Theory):
When you coarsen a graph, how is the structure of the original subgraphs preserved? It is not clear how to use the coarsened graph to say something about the subgraphs in the original graph.

Author response:
It seems that graph coarsening in CoS2N has not been written clearly and in detail. The details of this are as follows, and we will include them in the revised paper.

1. Apply the graph coarsening method to the global graph. The output is a partition of nodes in the global graph. In other words, graph coarsening assigns one super-node to each node in the global graph.
2. Construct induced subgraphs of the global graph per a set of nodes in the same super-node. We call them virtual subgraphs.
3. Merge a set of virtual subgraphs and an existing set of real subgraphs. Until this step, the structures of the original subgraph are preserved as is.
4. Perform S2N translation of this union set of subgraphs.

Reviewer comment (Novelty):
The graph coarsening process lacks novelty. It is quite straightforward and well-known to treat the subgraphs into a single node and link the nodes that share the nodes in the original graphs.

Author response:
We argue that the value of simplicity and the improved performance and efficiency over existing methods reinforce the genuine novelty of our contributions. We would like to offer the following points to counter the critique.
- **Value of Simplicity**: The simplicity of our proposed approach is one of its key strengths and contributions. By distilling the problem to its core and focusing on a simple yet effective idea, we can provide an efficient and effective solution for subgraph-level tasks.
- **Novel Perspective and Fundamental Question**: We emphasize that it is not well-known to treat the subgraphs into a single node and link the nodes that share the nodes for subgraph representation learning. How to design an efficient data structure for subgraphs is a novel perspective that not only complements but advances the field's understanding and approaches to the problem.
- **Outperforming State-of-the-Art**: We note that our proposed S2N translation is on par with or better than the state-of-the-art. Our approach is fast and lightweight while maintaining performance levels, and should not be dismissed as straightforward.

Last but not least, we would like to quote the other reviewer `1NPi`'s evaluation of the novelty: *"The idea of using graph coarsening for subgraph classification is novel (to the best of my knowledge) and deserves further study."*

We think our strengths have not clearly revealed in the introduction, and based on the above bullets, we will revise them so that they are clearly visible. We would appreciate you reconsidering the novelty of our method.

First, thank you for understanding the novelty of our research from the perspective of graph coarsening for subgraph representation learning.

We agree with the reviewer's comment regarding other existing tasks that map subgraphs to nodes (such as graph generative models). However, as the reviewer pointed out, this approach has not been used in the subgraph-level task, and rather, research for this task has been conducted by designing complex models. We emphasize that our primary contribution is not developing a technical and specialized method but exploring the fundamental question of subgraph representation learning and proposing a novel perspective.

Nevertheless, we will survey existing studies of mapping subgraphs to nodes for various downstream tasks like JT-VAE. In the next revision, we will include this line of work and compare it with our proposed S2N.

Reviewer comment (Theory):
Can this approach distinguish the two subgraphs that share the same number of nodes, i.e., is the proposed structure reconstructable? For instance, what if the red subgraph is connected to the right side of the blue subgraph in Figure 1? It may generate the same subgraphs and same number of shared nodes.

Author response:
Generally, S2N can distinguish between two subgraphs with the same number of nodes. This is because S2N's node will have different neighborhoods in the S2N graph depending on neighbor subgraphs in the global graph. The case where S2N cannot distinguish between two subgraphs of the same size is when (1) the global structure around subgraphs is completely identical and (2) features of subgraphs are identical. However, it is difficult to exist due to real-world graphs' rich features and complex dynamics.

Reviewer comment (Theory):
Lack of details for the selection step of the subgraphs to be mapped into new nodes. How do you select the subgraphs and how do you prove that the selected subgraph is the optimal choice?

Author response:
In this study, all given subgraphs are mapped to new nodes. In detail, all training subgraphs are translated into nodes in the training stage, and all training and evaluation subgraphs are translated into nodes in the evaluation stage (See Section 5). How to select subgraphs that make up the optimal S2N graph is an interesting research topic but is not the scope of this paper. We leave this as a future study and will mention it in the revised paper.

Reviewer comment (Experiments):
Lack of backbone architectures, which are limited to GCN-based. What about other GNN backbone architectures such as GIN? Is the proposed method restricted only to GCN as proved in Section 4.2?

Author response:
This is a valid critique and thank you for pointing it out. We are now conducting experiments using GIN/GAT plus separated/connected forms. We will upload the results as they come out. Please understand that getting full results before the discussion period might not be possible since the connected form requires more computational resources.

We uploaded [the revised paper](https://openreview.net/pdf?id=BeuTCoe3bf) that addresses remaining issues:
1. In the related work section (Section 2), we added additional papers about mapping subgraphs to nodes to represent meaningful clusters. In particular, we compared our work with JT-VAE in Appendix A.1.3.
2. We updated results using GIN and GAT with separated and connected forms on PPI-BP and EM-User datasets (Table 6 and Appendix A.8). For GIN and GAT, We confirm that S2N outperforms classic data structures: separated and connected forms.

Reviewer comment (Novelty):
What is the difference between the existing works and the proposed works on super-nodes? I cannot clearly understand what the node boundaries in super-nodes are unknown is in Section 2.

Author response:
A node boundary is the boundary between a subgraph (or super-nodes) and the global graph. When we say node boundary is unknown, the super-node is not given to existing graph coarsening methods. These methods aim to find a set of super-nodes that partitions a given graph according to specified constraints. Unlike these, we view given subgraphs as super-nodes and explore whether performance and efficiency differ in subgraph-level downstream tasks when the super-nodes are mapped to nodes. That is, the purpose and primary conditions of the research are different. We will revise this part clearly by choosing straightforward terminologies.

Reviewer comment (Experiments):
There lack of ablation study of certain hyperaprameters and design choices. For example, the authors "use two well-known GNNs" but it is unclear why alternative choices are not discussed or used. Given the abundance of GNN models nowadays and the fact that GCN and GCN2 are relatively earlier (before 2021), it is unclear if the adoption of more recent GNN models could yield better results. The authors mention one baseline, SubGNN, uses pre-trained embeddings by GIN, yet there is no explanation of why GIN can or cannot be used for the proposed methods.

Author response:
Dear Reviewer AUEj.

We kindly remind you that there is one day left until the discussion period ends. We uploaded [the revised paper](https://openreview.net/pdf?id=BeuTCoe3bf) addressing your concerns. If you have any additional questions after reading our revised paper, we are happy to discuss them.

In summary, our revision includes:
1. We added experiments using GATv2 (a more recent GNN) and GIN (Table 6 and Appendix A.8).
2. We cited the GIN paper.
3. We added ablation studies regarding the number of layers (including no message-passing) and readout methods (Table 7, 8 and Appendix A.9).
4. We provided more descriptions of existing methods, SubGNN and GLASS (Appendix A.1.1).
5. We added the model-architecture-level comparison with DiffPool (Appendix A.1.2).

Thank you again for your time and effort in reviewing our paper. We are looking forward to your response.

Reviewer comment (Presentation):
I suggest the authors provide more descriptions of existing methods, esp. SubGNN and GLASS. For example, if and what GNN models are used. There is some detail in Appendix A, but an additional section that focuses on the architectural comparison of all the methods would further enhance the clarity of the paper.

Author response:
We agree with the reviewer's comment that a detailed architecture description will help with clarity. We describe the architectural differences between SubGNN, GLASS, and S2N below. This will be included in the Appendix.

SubGNN, GLASS, and S2N improve different parts of the machine learning pipeline to solve subgraph-level tasks. SubGNN designs a whole model, GLASS augments input data through a labeling trick, and S2N uses a new data structure.

SubGNN performs message-passing between subgraphs (or patches). Through this, the properties of internal and border structures for three channels (position, neighborhood, and structure) are learned independently. To learn a total of 6 (2 x 3) properties, SubGNN designs patch samplers, patch representation, and similarity (weights of messages) for each property in an ad hoc manner. To learn internal positions, for example, SubGNN patches nodes inside the subgraph uses its representation as a message, and uses distance-based similarity as weights. By the complex model design, SubGNN requires a lot of computational resources for data pre-processing, model training, and inference.

GLASS uses plain GNNs but labels input nodes as to whether they belong to the subgraph (the label of one) or not (the label of zero). Separate node-level message-passing is performed for each label to distinguish the internal and border structures of the subgraph. GLASS's labeling trick is effective, but hard to handle multiple labels from multiple subgraphs in a batch. Although the authors of GLASS propose a max-zero-one trick to address this issue, small batches are still recommended. In addition, using a large global graph requires significant computational and memory resources.

In comparison, our proposed S2N uses the new data structure that stores and processes subgraphs efficiently. By compressing the global graph, computational and memory resource requirements are reduced. There are no restrictions on batch learning so we can train S2N graphs in the full batch.

Reviewer comment (Writing):
Writing issues, e.g. lack of citation of GIN.

Author response:
Dear Reviewer AUEj.

We kindly remind you that there is one day left until the discussion period ends. We uploaded [the revised paper](https://openreview.net/pdf?id=BeuTCoe3bf) addressing your concerns. If you have any additional questions after reading our revised paper, we are happy to discuss them.

In summary, our revision includes:
1. We added experiments using GATv2 (a more recent GNN) and GIN (Table 6 and Appendix A.8).
2. We cited the GIN paper.
3. We added ablation studies regarding the number of layers (including no message-passing) and readout methods (Table 7, 8 and Appendix A.9).
4. We provided more descriptions of existing methods, SubGNN and GLASS (Appendix A.1.1).
5. We added the model-architecture-level comparison with DiffPool (Appendix A.1.2).

Thank you again for your time and effort in reviewing our paper. We are looking forward to your response.