Paper: On the Role of Edge Dependency in Graph Generative Models (ICLR-2024)

Reviewer comment (Evaluation):
Overlap seems not to be a good meausre for diversity of small graphs since it does not take isomorphism of graphs into account.

Author response:
Thanks – this is a great point. We have considered the same issue before – one can avoid it by defining a notion of overlap that looks at the maximum edge overlap between two graphs taken over any permutation of the node ids. Unfortunately, this is a very hard metric to compute in both theory and practice, due to its connection to the graph isomorphism problem. Developing a useful metric inspired by this approach that can be computed efficiently, and studying its theoretical properties, would be a very nice direction for future work.

Reviewer comment (Reproducibility):
How do you select maximal clique one by one in Algorithm 1? Is there overlap between two max cliques?

Author response:
We use a fast state of the art implementation for maximal clique listing, more specifically [MACE](https://research.nii.ac.jp/~uno/code/mace.html). Indeed, two maximal cliques can overlap. E.g., two 4-cliques can overlap on the 3 common nodes, but each one has a separate node not connected to each other. Hence, edges/nodes common among 2 (or more) maximal cliques have higher probability of being sampled, helping the benchmarks achieve high triangle counts in low overlap.

Reviewer comment (Experiments):
Why the statistics results of GraphVAE are missing in Figures 3-6 and 9?

Author response:
GraphVAE is a method designed for generating small networks (GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders), hence it faces scalability issues for moderate to large sized graphs. For this reason, we only evaluate this method on a subset of small-sized datasets. Notice that this method employs a fully connected MLP to generate all $n \choose 2$ possible edges, hence for graphs >1K we face both runtime and memory issues.

Reviewer comment (Experiments):
How do you generate the small subgraphs of benchmarks in Table 2? Is there a unified approach for this?

Author response:
Yes, we use a unified approach for all datasets. The approach is based on using a maximal clique enumeration algorithm (see answer to 1), followed by Algorithm 1 that samples either (i) each edge of a maximal clique independently for the EI benchmark, (ii) each node of a maximal clique independently for the NI benchmark, or (iii) each maximal clique independently for the FD benchmark.

Reviewer comment (Experiments):
How large a graph can this method apply to? Do the experiments not use large datasets just because it is difficult to verify the results?

Author response:
The scalability of this approach depends on the maximal clique listing algorithm used. We could easily scale to graphs of ~4-5k nodes, which is comparable (or larger) to the size of graphs that DL methods can scale to (e.g., NetGAN/CELL, or GraphRNN). For applications to larger networks, we note that our models only need some groupings of the input graph’s edges, not necessarily its maximal cliques. Therefore, edge groupings like dense subgraph discovery, decomposition methods (Elfarouk et al. 2022), and near-clique discovery (Tsourakakis et al. 2013), which are tractable to compute, could be used instead of max cliques.

Reviewer comment (Novelty):
Three categorizations for graph generative models cannot include the most recent generative models. Can you give more examples of recent generative models (e.g., GDSS, GraphARM, DiGress, etc.) that fit the proposed categorization?

Author response:
Similarly to ERGMs, auto-regressive approaches (like GraphARM, and including diffusion models like DiGress) fall into the FD category of our hierarchy. Depending on the mapping function, flow-based models (like GraphAF) are likely to be FD as well.

Reviewer comment (Writing):
The definitions for EI, FD, NI models are vague, and a more formal definition for categorization is needed. For instance, is the definition of FD model formal? It’s hard to understand what “the generative models allow for any possible distribution A” means.

Author response:
We believe our definitions of the EI and NI model classes are formal and unambiguous, though we are open to any specific criticisms. Regarding the FD model definition, indeed, it may be more clear to rephrase as “Any categorical distribution $\mathcal{A}$ over adjacency matrices $\boldsymbol{A} \in \{0,1\}^{n \times n}$ of undirected graphs without self-loops is an FD graph generative model.” We hope this addresses any concern about ambiguity. The FD definition will necessarily be sparser than the others, since NI and EI are carefully-constructed subsets of FD.

Reviewer comment (Evaluation):
A detailed explanation of the strength of using overlaps for graph generative models is missing. The authors emphasize that accuracy and diversity are key characteristics to evaluate the graph generative models but it’s hard to understand why overlap can be a good evaluation framework.

Author response:
Overlap is perhaps the simplest characterization of the (lack of) diversity of graphs generated by a graph generative model - roughly, it is the fraction of edges shared between two sampled graphs, in expectation. A graph generative model having high overlap means that sampling from it tends to generate similar sets of edges, rather than a diverse pool of edges. Consequently, requiring low overlap is a simple way of preventing a graph generative model from achieving high accuracy (high fidelity to the input graph, e.g., in terms of a high triangle count) by simply outputting the same or a very similar set of edges as the input graph.

Reviewer comment (Evaluation):
Lack of comparison to commonly used metrics such as MMD or V.U.N. MMD and V.U.N is the most popular evaluation metric for graph generative models. Still, I cannot find any comparison of overlap to current evaluation metrics for graph generative models.

Author response:
These metrics are not appropriate for the setting we test, which involves generation of a graph that is similar to a single input graph. Note that we test the same setting considered in the Variational Graph Auto-Encoder (Kipf and Welling, 2016) and NetGAN (Bojchevski et al., 2018; Rendsburg et al., 2020) papers, among others. By contrast, MMD is suitable for comparison of distributions of graphs, that is, a setting with multiple input graphs and multiple out graphs. VUN is also intended for such a setting, specifically for molecule generation.

Reviewer comment (Writing):
Hard to understand the key contribution of this paper. Is it right that the proposed evaluation framework is to compare the overlap between generated graphs and test graphs?

Author response:
The x-axis in Figure 2 is the overlap of a model that was fit to an input graph; each curve represents a different method with which to fit a model, such as GraphVAE. The y-axis represents accuracy, since, as you note, it shows how closely these models match some statistic, such as triangle count, of the input graph, which is shown by the “True” dotted line. Therefore, these are essentially several plots of accuracy vs diversity. What we find is that methods overall struggle to attain high accuracy while maintaining high diversity, and as the diversity requirement is decreased even simple methods like the ones we propose can be competitively accurate as some prior deep ones.

Reviewer comment (Evaluation):
Is triangle-based evaluation limited to the models that contain many triangles? For instance, for grid graph generation, can the proposed method capture the graph generation quality well?

Author response:
The primary advantage of our proposed methods is that they are simple and allow for directly controlling overlap. They can capture any kind of graph well in the sense that if there is no restriction on overlap/diversity, by setting $p=1$, they simply output the input graph and hence capture all of its statistics. When required to generate a diverse set of graphs, however, our clique-based methods may not be suitable for certain kinds of graphs like grid graphs. Clique-based graph generation in general is more suitable for graphs like social networks, which have strong community structure.

Reviewer comment (Presentation):
What does the x-axis mean in Figure 2? I can briefly understand that the lines following the dotted line (True) are desirable but cannot understand what Figure 2 means exactly.

Author response:
The x-axis in Figure 2 is the overlap of a model that was fit to an input graph; each curve represents a different method with which to fit a model, such as GraphVAE. The y-axis represents accuracy, since, as you note, it shows how closely these models match some statistic, such as triangle count, of the input graph, which is shown by the “True” dotted line. Therefore, these are essentially several plots of accuracy vs diversity. What we find is that methods overall struggle to attain high accuracy while maintaining high diversity, and as the diversity requirement is decreased even simple methods like the ones we propose can be competitively accurate as some prior deep ones.

Reviewer comment (Theory):
I am concerned about the cubic nature of the bounds. These seem to be very loose and it is hard to see how these can be used in practice, or even to provide insight about how to structure the generative process. This issue is especially important for the case of edge-independent models (cubic in the product $n \cdot Ov(\mathcal A$)).

Author response:
Indeed, the bounds for the triangle count in terms of node count and overlap are cubic in the node count. This is not entirely surprising: the total number of possible triangles is also cubic in the node count (specifically, it is $\binom{n}{3}$), and we are showing that a model’s overlap bounds the fraction of these triangles that can appear in expectation. Further, we provide examples demonstrating the tightness of our triangle count bounds. These examples show that there must necessarily be a term that is cubic in the node count, at least in the absence of further constraints. Under specific models, we may expect overlap to scale inversely with the number of nodes: for example, with Erdős–Rényi random graphs or related models such as stochastic block model graphs, assuming we fix the average number of friends (i.e., expected node degree), the overlap scales as $O(n^{-1})$. Thus, our bounds scale subcubicly in $n$.

Importantly, we note that the triangle bounds in terms of overlap and node count are not the tightest bounds that we provide. We also provide tighter bounds in terms of overlap and volume (i.e., expected number of edges) for EI and NI models - see the proof of Theorem 1 on page 6, prior to the application of Lemma 1. These overlap/volume bounds can be significantly tighter, but we apply Lemma 1 and convert them to the simpler overlap/node count bounds to highlight the conceptual differences between different levels of our proposed hierarchy, which is our focus.

Reviewer comment (Experiments):
Given the subtlety of the bounds, something that could facilitate seeing their applicability would be to have some synthetic verifications, but no experiments show the application of the bounds to verify their veracity.

Author response:
Thank you for the useful suggestion. As you point out, a simulation can corroborate the tightness of our bounds. In Appendix E, we have added a plot with a simulation that shows that the example models indeed achieve the tight bounds.

Reviewer comment (Reproducibility):
It is important to highlight the methodological complexity of using the bounds to achieve the target structure in the sampled graphs. Currently, Algorithm 1 has one line that describes MCDF and a couple that describe MCEI and MCNI, while most of the details are not fully presented in the main body of the paper.

Author response:
The sampling of a graph with any of our proposed methods (MCEI, MCNI, and MCFD) involves sampling a primary graph $G_p$ and a residual graph $G_r$. Essentially all of the novel content of our methods — specifically 1) the use of max cliques, 2) the addition of edge dependency in the MCNI and MCFD models, and 3) the ability to easily control overlap — occurs in the sampling of $G_p$. We are able to compactly describe exactly how $G_p$ is sampled in Algorithm 1 due to the simplicity of this procedure. However, due to space constraints, we unfortunately have to appendicize Algorithm 2, which explains the sampling of $G_r$. In brief, Algorithm 2 samples a $G_r$ from a simple distribution such that the union of $G_r$ and $G_p$ (i.e., the union of their edge sets) matches the node degrees of the input graph in expectation.

Reviewer comment (Experiments):
Identifying cycles can be computationally expensive and a scalability analysis of the method could be useful.

Author response:
Our proposed generative models are based on maximal clique enumeration, which determines their scalability. This is theoretically intractable in the worst case, but often practical for small-to-mid-sized real-world networks, such as the ones in this paper. For applications to larger networks, we note that our models only need some (possibly overlapping) groupings of the input graph’s edges, not necessarily its maximal cliques. Therefore, edge groupings like dense subgraph discovery, decomposition methods (Elfarouk et al., 2022), and near-clique discovery (Tsourakakis et al., 2013), which are tractable to compute, could be used instead of max cliques.

Reviewer comment (Theory):
What is the cost of a growing number of nodes to the technique proposed for bounding the number of triangles?

Author response:
Indeed, the bounds for the triangle count in terms of node count and overlap are cubic in the node count. This is not entirely surprising: the total number of possible triangles is also cubic in the node count (specifically, it is $\binom{n}{3}$), and we are showing that a model’s overlap bounds the fraction of these triangles that can appear in expectation. Further, we provide examples demonstrating the tightness of our triangle count bounds. These examples show that there must necessarily be a term that is cubic in the node count, at least in the absence of further constraints. Under specific models, we may expect overlap to scale inversely with the number of nodes: for example, with Erdős–Rényi random graphs or related models such as stochastic block model graphs, assuming we fix the average number of friends (i.e., expected node degree), the overlap scales as $O(n^{-1})$. Thus, our bounds scale subcubicly in $n$.

Importantly, we note that the triangle bounds in terms of overlap and node count are not the tightest bounds that we provide. We also provide tighter bounds in terms of overlap and volume (i.e., expected number of edges) for EI and NI models - see the proof of Theorem 1 on page 6, prior to the application of Lemma 1. These overlap/volume bounds can be significantly tighter, but we apply Lemma 1 and convert them to the simpler overlap/node count bounds to highlight the conceptual differences between different levels of our proposed hierarchy, which is our focus.

Reviewer comment (Theory):
What is the cost of the proposed models as the number of nodes grows?

Author response:
Our proposed generative models are based on maximal clique enumeration, which determines their scalability. This is theoretically intractable in the worst case, but often practical for small-to-mid-sized real-world networks, such as the ones in this paper. For applications to larger networks, we note that our models only need some (possibly overlapping) groupings of the input graph’s edges, not necessarily its maximal cliques. Therefore, edge groupings like dense subgraph discovery, decomposition methods (Elfarouk et al., 2022), and near-clique discovery (Tsourakakis et al., 2013), which are tractable to compute, could be used instead of max cliques.

Reviewer comment (Theory):
How do you link the bounds to determine what edges have to be sampled to achieve a target structure?

Author response:
The process is quite straightforward given our bounds: the upper bound for the triangle count in terms of volume and overlap of a model can be solved for a lower bound for the overlap in terms of the triangle count and volume. For a given input graph, we know the latter two target quantities, so we can solve for the former.