Paper: Evaluating graph generative models with graph kernels: what structural characteristics are captured? (ICLR-2024)

Reviewer comment (Evaluation):
The paper claims that graph kernels are commonly used to evaluate the performance of graph generative models, however, no references are given. The authors should add some references and also motivate the use of graph kernels for this task. What are the advantages of graph kernels over competing evaluation approaches?

Author response:
We refer to Section 2.2 which discusses the evaluation of graph generative models. In particular, O’Bray et al. (2022) focus on this pipeline of evaluating generative models (but they use very simple kernels). Thompson et al. (2022) go beyond kernel-based evaluation, but their proposed approach can also be fitted to the kernel-based framework as we do in the paper.

Reviewer comment (Evaluation):
The paper focuses on 7 graph properties, but it is not clear why those 7 specific properties were chosen and not others. Are there any applications of graph generative models where graphs that exhibit those properties need to be generated? Please provide more details.

Author response:
We have chosen high-level properties that are meaningful, diverse, and easy to interpret. Of course, one can expect a good graph kernel to be sensitive to all of them, but there is usually a trade-off. Regarding potential applications, for molecular modeling it can be important that the measure is sensitive to clustering or complementarity as the presence of certain small substructures is crucial. For ego-network modeling, heterogeneity, and community structure are important.

Reviewer comment (Theory):
Since no theoretical results are provided, I would expect the authors to provide some explanation or intuitions about why some kernels fail to capture some of the considered properties. An explanation is provided in the case of the WL kernel, but not for all the kernels.

Author response:
Let us note that most of the kernels are intractable for theoretical analysis. For Graphlet3, we might be able to provide the expected values of kernel values, but getting the variances is already much less tractable. Regarding the intuitive explanations, we will try to improve this part in the revised version.

Reviewer comment (Experiments):
The experimental evaluation is not fully convincing because all the generated graphs consist of 50 nodes and 190 edges (in expectation). It is not thus clear whether the results generalize to graphs of different sizes. I would suggest the authors construct other datasets where graphs are of different size and different density.

Author response:
Thank you for this suggestion, the experiments on larger graphs (1000 nodes) are currently running, we will update the paper when they are ready. 

The main reason that we focus on small graphs, is that these methods (evaluating graph generative models using graph kernels) are typically applied to relatively small networks. For example, in molecular modeling, the graphs consist of a relatively small number of atoms. Other applications where small graphs are relevant include modeling ego networks in sociology.

Additionally, let us note that not all kernels are scalable: for instance, the graphlet kernel has a prohibitively high computational cost. We chose these smaller graphs so that we can include all popular graph kernels in our experiment.

Also, as suggested, we conduct additional experiments where we consider density as a characteristic that we vary. The results are the following:

| Kernel | Density |
|---------|----------|
| SP | 0.993 / 0.996 |
|WL | 0.996 / 0.996 |
|WL-OA | 0.996 / 0.996 |
| Graphlet-3 | 0.996 / 0.996 |
| NSPDK | 0.389 / 0.341 |
| PM | 0.982 / 0.979 |
| NetLSD | 0.996 / 0.996 |
| RandGIN | 0.903 / 0.973 |

We see that detecting density is an easy task for most of the kernels, while there are some exceptions, e.g., NSPDK.

Reviewer comment (Reproducibility):
In page 9, it is mentioned that "there are only two different graphlets of size k=3: wedges and triangles". This is true if connected graphlets are only considered. However, the standard graphlet kernel also counts disconnected graphlets and there are 4 such graphlets for k=3. Did the authors use an implementation that counts connected graphlets?

Author response:
Yes, we use the GraKeL implementation that counts connected graphlets.

Reviewer comment (Evaluation):
I find the paper shows some relationship between generative models of graphs and kernels, it is hard to see what are the applications of these findings in real life. The conclusion is rather qualitative.

Author response:
Kernels can be used to evaluate graph generative models. However, some kernels are insensitive to certain graph characteristics. For example, our experiments show that the widely-used Weisfeiler-Lehman kernel is insensitive to geometry. In turn, geometry can be especially relevant, e.g., for the application of molecular modeling because atoms have spatial locations. Such findings help practitioners decide which kernels they should and shouldn't use for their evaluations.

Reviewer comment (Theory):
One of the key step of the paper is to interpolate between generative models, its formula is not clearly stated. It seems to "interpolate" probabilities of different models, and the idea of "interpolating graph characteristics" has to be properly defined as the characteristics are not mutually exclusive. It is difficult to understand what it means by having "100%" of a characteristic and "0%" of another when the two are related.

Author response:
We agree that varying one characteristic may also affect other graph properties as it can be hard (or even impossible) to vary only one property in isolation. For each property, we have chosen a graph generator that produces graphs in which this property is strongly present (we try to consider the simplest possible model). Then, we fix a starting point (usually the ER graph) and an ending point (a given model where the property is present to some extent) and make a transition between the two. Section 3.2 states how the parameters of the graph generators depend on the interpolation parameter. If some particular transitions are not clear, we are happy to clarify.

Reviewer comment (Evaluation):
The "performance" of kernels is difficult to grasp and it is related to what is needed to be performed as in also the last point.

Author response:
Following your suggestions, we will extend the discussion to clearly state some practical conclusions that follow from our analysis. For instance, we observe that random GIN is insensitive to communities, while WL is insensitive to latent geometry. If the generator is supposed to generate graphs that match a certain sample of graphs in terms of having a similar community structure or geometry, then these kernels should be avoided.

Reviewer comment (Evaluation):
Overall, I'd suggest to go on this direction with a clear, deeper study of what exactly are the different between generative models. At the moments, it doesn't show concrete conclusion yet for any practical purpose.

Author response:
Following your suggestions, we will extend the discussion to clearly state some practical conclusions that follow from our analysis. For instance, we observe that random GIN is insensitive to communities, while WL is insensitive to latent geometry. If the generator is supposed to generate graphs that match a certain sample of graphs in terms of having a similar community structure or geometry, then these kernels should be avoided.

Reviewer comment (Experiments):
From my point of view, the paper's main weakness lies in the numerical part: the only results presented concern graphs with 50 nodes and 190 edges (on average). We would like to know whether the results obtained are robust when we vary the size of the graph (in number of nodes and number of edges).

Author response:
**W1. The only results presented concern graphs with 50 nodes and 190 edges (on average). We would like to know whether the results obtained are robust when we vary the size of the graph (in number of nodes and number of edges).**

Thank you for this suggestion, the experiments on larger graphs (1000 nodes) are currently running, we will update the paper when they are ready.

The main reason that we focus on small graphs, is that these methods (evaluating graph generative models using graph kernels) are typically applied to relatively small networks. For example, in molecular modeling, the graphs consist of a relatively small number of atoms. Other applications where small graphs are relevant include modeling ego networks in sociology.

Additionally, let us note that not all kernels are scalable: for instance, the graphlet kernel has a prohibitively high computational cost. We chose these smaller graphs so that we can include all popular graph kernels in our experiment.

Reviewer comment (Experiments):
Another limitation is that the results obtained depend on the probabilistic models chosen. For instance, each may introduce random features other than the one targeted, which may bias the results obtained. Figure 2 gives some interesting examples of this phenomenon. This may be an intrinsic limitation of the approach, but it can be overcome by considering different probabilistic models for the same feature.

Author response:
**W2. Another limitation is that the results obtained depend on the probabilistic models chosen. For instance, each may introduce random features other than the one targeted, which may bias the results obtained. Figure 2 gives some interesting examples of this phenomenon. This may be an intrinsic limitation of the approach, but it can be overcome by considering different probabilistic models for the same feature.**

We agree with this comment and plan to add a clarification about this limitation in the revised text. Our aim was to model each of the properties in the most straightforward way, and in a way that minimally affects other graph characteristics. For some graph characteristics, it can be challenging (or even impossible) to vary them in isolation from everything else.

The idea of considering different models for the same property is interesting - we already do this for clustering: this property can be modeled via triadic closure, latent geometry, or community structure. Other possible variations we can think of are: modeling complementarity by disassortative community structure and modeling degree inhomogeneity by preferential attachment. We consider adding these experiments in the revised version of the paper. However, we may not have time to finalize them during the discussion period as we also need to conduct experiments on larger graphs, as discussed above.

Reviewer comment (Writing):
To have a self-contained paper, the Erdös-Rényi and Chung-Lu models should be described.

Author response:
**Q1. To have a self-contained paper, the Erdös-Rényi and Chung-Lu models should be described.**

Thank you for pointing this out, we will add the definitions in the revised version.

Reviewer comment (Reproducibility):
For some models, the authors explain precisely the link between the parameters and the average number of edges (triadic closure in Appendix B.1 and dimensionality in Appendix B.2). For others (e.g. latent geometry) there is no explanation at all.

Author response:
**Q2. For some models, the authors explain precisely the link between the parameters and the average number of edges (triadic closure in Appendix B.1 and dimensionality in Appendix B.2). For others (e.g. latent geometry) there is no explanation at all.**

The result for latent geometry also follows from Appendix B.2 by setting h=1. We will add a reference to Appendix B.2 to the paragraph about latent geometry.

Reviewer comment (Theory):
As a reader, we are left a little disappointed by the two correlation measures. A single indicator would greatly simplify the statement of results. The data suggest that, in most cases, they are equal. For features such as the triadic model and for the Graphlet kernel, the link between the two correlations might be studied theoretically, which would provide a valuable piece of information on this question.

Author response:
**Q3. As a reader, we are left a little disappointed by the two correlation measures. A single indicator would greatly simplify the statement of results. The data suggest that, in most cases, they are equal.**

Thank you for this comment - we agree that having two numbers can be somewhat inconvenient and that they are in most cases consistent. We decided to simplify the presentation by reporting the average value in the main table (or average±difference/2). We will update the table in the revised version.

Reviewer comment (Theory):
On page 2, one reads that computing a distance is as hard as graph isomorphism, while a few lines down, it says that any kernel can be easily transformed to a distance measure. Can the authors address this apparent contradiction?

Author response:
**Q4. On page 2, one reads that computing a distance is as hard as graph isomorphism, while a few lines down, it says that any kernel can be easily transformed to a distance measure. Can the authors address this apparent contradiction?**

Thanks for noticing! As we write above, graph ‘distances’ usually violate the positivity axiom, and if we apply the mentioned transformation - positivity can also be violated. We will clarify this in the text.