Paper: GraphMaker: Can Diffusion Models Generate Large Attributed Graphs? (ICLR-2024)

Reviewer comment (Evaluation):
The main weakness of the paper is that the empirical results show that the proposed method only marginally outperforms the SBM in the evaluation aspects (graph property, discriminative, benchmarking). While the GraphMaker graphs seem to match graph statistics slightly better (in aggregate), and also better align ranks in benchmarking, the discriminative aspect (Table 2) shows that GNN models trained on synthetic graphs from GraphMaker vs those from SBM do about the same when trained on the source data.

Author response:
This is indeed a critical insight. Various homophily measures have been developed to measure the similarity between nodes and their neighbors in a graph [1-3], in particular the proportion of nodes that have the same label as their neighbors. These papers have pointed out that Cora and Citeseer are highly homophilous, and it’s likely that Amazon Photo and Amazon Computer also have this issue. When we apply SBM to such graphs, with high probability the generated nodes will be connected to neighbors that have the same label. In addition, the node attributes can also be approximately conditionally independent of each other given the node label, as shown in Table 8 in the Appendix. These two factors may explain the competitive performance achieved by SBM for MPNNs, which perform localized smoothing.

We agree that more heterophilous datasets with more complicated attribute, label, and structure correlations can be better testbeds. We plan to include such datasets in the next version of the paper.

Reviewer comment (Experiments):
The graphs used in the empirical study are too homophilous, and thus the label-conditioning aspect of GraphMaker could essentially be copying what the SBM is already doing.

Author response:
This is indeed a critical insight. Various homophily measures have been developed to measure the similarity between nodes and their neighbors in a graph [1-3], in particular the proportion of nodes that have the same label as their neighbors. These papers have pointed out that Cora and Citeseer are highly homophilous, and it’s likely that Amazon Photo and Amazon Computer also have this issue. When we apply SBM to such graphs, with high probability the generated nodes will be connected to neighbors that have the same label. In addition, the node attributes can also be approximately conditionally independent of each other given the node label, as shown in Table 8 in the Appendix. These two factors may explain the competitive performance achieved by SBM for MPNNs, which perform localized smoothing.

We agree that more heterophilous datasets with more complicated attribute, label, and structure correlations can be better testbeds. We plan to include such datasets in the next version of the paper.

Reviewer comment (Experiments):
The authors did not benchmark against the EDGE graph diffusion model (Chen et al. 2023), which would be an interesting baseline alongside the SBM, as it similarly does not generate the attributes.

Author response:
As mentioned in the above unified response, we agree that a direct comparison will help us better understand the strengths and limitations of the proposed model. We plan to include some previous diffusion-based graph generative models as additional baselines in the next version of the manuscript.

Reviewer comment (Writing):
The task that predicts the node labels and edge existence given the node attributes is more like a link prediction task, not a graph generation task. Is there any specific reason or reference that defines the task as a graph generation task?

Author response:
> 1. What’s the difference between link prediction and graph generation?

Link prediction considers the scenario of inferring additional edges from an existing incomplete graph [1, 2]. In contrast, graph generation requires creating a graph from scratch without leveraging an existing incomplete graph [3, 4].

Reviewer comment (Reproducibility):
What is the time complexity of GraphMaker? As the model is diffusion-based, it seems that the inference and training time may be long. How does GraphMaker benefit from large attributed graph generation? Hard to understand what a minibatch strategy is which is introduced in the introduction part.

Author response:
> 2. What is the time complexity of GraphMaker for edge generation? What is the minibatch strategy introduced in the introduction part?

GraphMaker first samples a noisy graph from the prior distribution, and then repeatedly refines it. At each timestep, it computes node representations with an MPNN, and then performs binary classification for all node pairs. This yields a time complexity of $O(TN^2)$, where $T$ is the number of diffusion steps and $N$ is the number of nodes. While this appears to be costly, we only need to perform node representation computation once per diffusion step, which can be used for all node pairs. Besides, binary classification for multiple node pairs given node representations can be performed in parallel. In addition, the optimal $T$ is less than 10 based on empirical studies. In terms of wall clock time, generating the largest graph (13K nodes) considered in the paper with the most costly GraphMaker model takes less than 10 minutes on an NVIDIA RTX A6000 GPU during inference.

Another issue is the space complexity of $O(N^2)$, which prevents us from performing binary classification for all node pairs at once on a single GPU. Hence we adopt the minibatch strategy. During training, we use a minibatch of node pairs for a gradient update. During inference, we generate the whole graph structure by iterating over minibatches of node pairs.

For larger graphs, we agree that $O(N^2)$ indeed will be problematic, and we may leverage methods for approximate nearest neighbor. For example, locality sensitive hashing [5] hashes similar data points into the same bucket with high probability, and therefore it avoids all pair comparisons.

Reviewer comment (Writing):
Some technical details are not clearly introduced, especially in 3.5. It was only mentioned that "which generates node attributes through an external approach conditioned on node labels." How does this external approach/node label conditioning work exactly? What kind of label is suitable for this purpose? How are these node labels related to Y in 3.2? These are not clearly explained.

Author response:
> 3. How are node attributes obtained for GraphMaker-E with an external approach conditioned on node labels?

As explained in the unified response, we leverage the empirical distribution $P(Y)\prod_v\prod_f P(X_{v, f} | Y_v)$ computed from the original data for the experiments in this paper. We agree it’s better to mention it explicitly in 3.5 rather than 4.1.

Reviewer comment (Writing):
Motivation of the decoupled approach is not well articulated, and the choice of the word "decoupled" is misleading. First, the explanation/motivation of why GraphMaker-Asyn is better is not laid out convincingly. From the experiments, the results are also quite mixed. Essentially, together with GraphMaker-E, there are three alternative versions proposed, but there are no clear insight on why each version plays to its strength in certain scenarios. Secondly, GraphMaker-Asyn is not really decoupled, as both edge and attribute generation are still trained together.

Author response:
> 1. Motivation of the decoupled approach GraphMaker-Async is not well articulated. Besides, the choice of the word ``decoupled'' is misleading.

We provided a motivation for choosing GraphMaker-Async over GraphMaker-Sync from the perspective of edge dependency modeling in the above unified response.

We used the word ``decoupled'' as GraphMaker-Async consists of two subnetworks trained independently and once trained it first generates attributes and then edges conditioned on fixed attributes. We are open to other options if the reviewer has suggestions for better alternatives.

Reviewer comment (Writing):
What is the difference between node attributes and labels? In Section 2, both are defined as categorical.

Author response:
> 2. What is the difference between node attributes and labels as both of them are categorical in section 2?

The confusion comes from the fact that one important application of our generative model is to generate and publish synthetic graphs without sharing the original data. ML models (such as Graph Neural Networks) can be trained over these synthetic graphs and are expected to achieve similar performance as those trained directly over the original graphs. To evaluate our model for this application, we adopted node classification and link prediction tasks as a proof of concept and considered the large-attributed graphs that are widely used for node classification tasks [1, 2]. The node labels defined by the node classification tasks are used as the conditional information in our generation process. It is true that distinguishing labels and attributes is often just an artifact, defined by an ML task, which is irrelevant to the data generation procedure. However, if a downstream ML task is given, using the labels as conditions may indeed help. In our case, we find that directly viewing the node label as a node attribute yields worse performance for more than 80% cases for the downstream node classification tasks as shown in section 4.4.

Reviewer comment (Novelty):
The application scenario of large attributed graph generation is unclear. Currently, generation of small graphs such as molecules are popular as they may potentially drive killer applications such as drug discovery.

Author response:
> 4. What is the application scenario of large attributed graph generation?

As elaborated in the first paragraph and unified response, firstly, large attributed graph generation can enhance the understanding of complex networks [3, 4]. Secondly, the generated graphs can be shared with the public in replacement of the original sensitive data while preserving data utility.