E VALUATION M ETRICS FOR G RAPH G ENERATIVE M ODELS : P ROBLEMS , P ITFALLS , AND P RACTICAL S OLUTIONS
equally. These jointly supervised this work.
A BSTRACT
Graph generative models are a highly active branch of machine learning.
Given the steady development of new models of ever-increasing complexity, it is necessary to provide a principled way to evaluate and compare them.
In this paper, we enumerate the desirable criteria for such a comparison metric and provide an overview of the status quo of graph generative model comparison in use today, which predominantly relies on maximum mean discrepancy (MMD).
We perform a systematic evaluation of MMD in the context of graph generative model comparison, highlighting some of the challenges and pitfalls researchers inadvertently may encounter.
After conducting a thorough analysis of the behaviour of
MMD on synthetically-generated perturbed graphs as well as on recently-proposed graph generative models, we are able to provide a suitable procedure to mitigate these challenges and pitfalls.
We aggregate our ﬁndings into a list of practical recommendations for researchers to use when evaluating graph generative models.
I NTRODUCTION
Graph generative models have become an active research branch, making it possible to generalise structural patterns inherent to graphs from certain domains—such as chemoinformatics—and actively synthesise new graphs (Liao et al., 2019).
Next to the development of improved models, their evaluation is crucial.
This is a well-studied issue in other domains, leading to metrics such as the ‘Fréchet Inception Distance’ (Heusel et al., 2017) for comparing image-based generative models.
Graphs, however, pose their own challenges, foremost among them being that an evaluation based on visualisations, i.e. on perceived differences, is often not possible.
In addition, virtually all relevant structural properties of graphs exhibit spatial invariances, e.g., connected components and cycles, are invariant with respect to rotations—that have to be taken into account by a comparison metric.
While the community has largely gravitated towards a single comparison metric, the maximum mean discrepancy (MMD) (Chen et al., 2021; Dai et al., 2020; Goyal et al., 2020; Liao et al., 2019; Mi et al., 2021; Niu et al., 2020; Podda & Bacciu, 2021;
You et al., 2018; Zhang et al., 2021), neither its expressive power nor its other properties have been systematically investigated in the context of graph generative model comparison.
The goal of this paper is to provide such an investigation, starting from ﬁrst principles by describing the desired properties of such a comparison metric, providing an overview of what is done in practice today, and systemically assessing MMD’s behaviour using recent graph generative models.
We highlight some of the caveats and shortcomings of the existing status quo, and provide researchers practical recommendations to address these issues.
We note here that our investigations focus on assessing the structural similarity of graphs, which is a necessary ﬁrst step before one can jointly assess graphs based on structural and attribute similarity.
This paper purposefully refrains from developing its own graph generative model to avoid any bias in the comparison. [Figure]
C OMPARING GRAPH DISTRIBUTIONS
In the following, we will deal with undirected graphs.
We denote such a graph as G =
( V, E ) with vertices V and edges E .
We treat graph generative models as black-box models, each of which results in a set 1 of graphs G .
The original distribution of graphs is denoted as G ∗ .
Given models {M 1 , M 2 , . . .
} , with generated sets of graphs {G 1 , G 2 , . . .
} , the goal of generative model evaluation is to assess which model is a better ﬁt, i.e. which distribution is closer to G ∗ .
This requires the use of a (pseudo-)metric d( · , · ) to assess the dissimilarity between G ∗ and generated graphs.
We argue that the desiderata of any such comparison metric are as follows:
Expressivity : if G and G (cid:48) do not arise from the same distribution, a suitable metric should be able to detect this.
Speciﬁcally, d( G , G (cid:48) ) should be monotonically increasing the further apart G and G (cid:48) are from one another.
Robustness : if a distribution G is subject to a perturbation, a suitable metric should be robust to small perturbations.
Changes in the metric should be ideally upper-bounded by a function of the amplitude of the perturbation.
Robust metrics are preferable because of the inherent stochasticity of training generative models.
Efﬁciency : model comparison metrics should be reasonably fast to calculate; even though model evaluation is a post hoc analysis, a metric should scale well with an increasing number of graphs and an increasing size of said graphs.
While there are many ways to compare two distributions, ranging from statistical divergence measures to proper metrics in the mathematical sense, the comparison of distributions of graphs is exacerbated by the fact that individual graphs typically differ in their cardinalities and are only described up to permutation.
With distances such as the graph edit distance being NP-hard in general (Zeng et al., 2009), which precludes using them as an efﬁcient metric, a potential alternative is provided by using descriptor functions.
A descriptor function f maps a graph G to an auxiliary representation in some space Z .
The problem of comparing a generated distribution G = { G 1 , . .
. , G n } to the original distribution G ∗ =
{ G ∗ 1 , . . .
, G ∗ m } thus boils down to comparing the images f ( G ) :
= { f (G 1 ) , . . .
, f (G n ) } ⊆
Z and f ( G ∗ )
⊆
Z by any preferred statistical distance in Z (see Figure 1).
C URRENT STATE OF GRAPH GENERATIVE MODEL EVALUATION : MMD
Of particular interest in previous literature is the case of Z = R d and using the maximum mean discrepancy d MMD ( · , · ) as a metric.
MMD is one of the most versatile and expressive options available for comparing distributions of structured objects such as graphs (Borgwardt et al., 2006; [Table]
Gretton et al., 2007), providing also a principled way to perform two-sample tests (Bounliphone et al., 2016; Gretton et al., 2012a; Lloyd & Ghahramani, 2015).
It enables the comparison of two statistical distributions by means of kernels , i.e. similarity measures for structured objects.
Letting X refer to a non-empty set, a function k: X × X
→ R is a kernel if k( x i , x j )
=
k( x j , x i ) for x i , x j ∈ X and (cid:80) i,j c i c j k( x i , x j ) ≥ 0 for x i , x j ∈ X
and c i , c j ∈ R .
MMD uses such a kernel function to assess the distance between two distributions.
Given n samples X = { x 1 , . . .
, x n } ⊆ X and m samples
Y = { y 1 , . . .
y m } ⊆ X , the empirical estimate of the MMD between X and Y is obtained as
[Equation]
Since MMD is known to be a metric on the space of probability distributions under certain conditions,
Eq. 1 is often treated as a metric as well (Liao et al., 2019; Niu et al., 2020; You et al., 2018), even though it would be more correct to only use it for hypothesis/two-sample testing, since as the authors of MMD themselves note, the empirical estimate of MMD can even take negative values (Gretton et al., 2012a), which we also observed in our experiments.
2 Nevertheless, MMD has been adopted by the community and requires (i) choosing a descriptor function f as described above, and (ii) choosing a kernel on R d such as an RBF kernel.
One then evaluates d MMD ( G , G ∗ ) :
=
MMD( f ( G ) , f ( G ∗ )) for a distribution G .
Given multiple distributions {G 1 , G 2 , . . . } , the values d MMD ( G i , G ∗ ) can be used to rank models: smaller values are assumed to indicate a larger agreement with the original distribution G ∗ .
We will now describe this procedure in more detail and highlight some of its pitfalls.
3. K ERNELS & D ESCRIPTOR FUNCTIONS
Before calculating the MMD distance between two distributions, we need to deﬁne both the kernel function k and the descriptor function f that will convert a graph G to a representation in Z = R d , for use in the MMD calculation in Eq.
We observed a variety of kernel choices in the existing literature.
In fact, in three of the most popular graph generative models in use today, which we explore in detail in this paper, a different kernel was chosen for each one.
These include a kernel using the ﬁrst Wasserstein distance (EMD), total variation distance (TV), and the radial basis function kernel (RBF), and are listed in Table 1.
In the current use of MMD, a descriptor function f is used to create a vectorial representation of a graph for use in the kernel computation.
We ﬁnd that several descriptor functions are commonly employed, either based on summary statistics of a graph, such as degree distribution histogram and clustering coefﬁcient histogram, or based on spectral properties of the graph, such as the Laplacian spectrum histogram.
While several papers also consider the orbit as a descriptor function, we do not consider it in depth here due to its computational complexity, which violates the “efﬁciency” property from our desiderata.
We will now provide brief explanations of these prominent descriptor functions.
Degree distribution histogram.
Given a graph G =
( V, E ) , we obtain a histogram by evaluating deg( v ) for v ∈ V , where position i of the resulting histogram is the number of vertices with degree i .
Assuming a maximum degree d and extending the histogram with zeros whenever necessary, we obtain a mapping f : G → R d .
This representation has the advantage of being easy to calculate and easy to compare; by normalising it (so that it sums to 1 ), we obtain a size-invariant descriptor.
Clustering coefﬁcient.
The (local) clustering coefﬁcient of a vertex v is deﬁned as the fraction of edges within its neighbourhood divided by the number of all possible edges between neighbours, i.e.
[Equation]
The value of C ( v ) ∈ [0 , 1] measures to what extent a vertex v forms a clique (Watts & Strogatz, 1998).
The collection of all clustering coefﬁcients of a graph can be binned and converted into a histogram in order to obtain a graph-level descriptor.
This function is also easy to calculate but is inherently local; a graph consisting of disconnected cliques or a fully-connected graph cannot be distinguished, for example.
Laplacian spectrum histogram.
Spectral methods involve assigning a matrix to a graph G , whose spectrum, i.e. its eigenvalues and eigenvectors, is subsequently used as a characterisation of G .
Let
A refer to the adjacency matrix of G , with A ij = 1 if and only if vertices v i and v j are connected by an edge in G (since G is undirected, A is symmetric).
The normalised graph Laplacian is deﬁned as
L := I − D − 12 AD − 12 , where I denotes the identity matrix and D refers to the degree matrix , i.e.
D ii = deg( v i ) for a vertex v i and D ij = 0 for i (cid:54) =
j .
The matrix L is real-valued and symmetric, so it is diagonalisable with a full set of eigenvalues and eigenvectors.
Letting λ 1 ≤ λ 2 ≤ . .
. refer to the eigenvalues of L , we have 0 ≤ λ i ≤ 2 (Chung, 1997, Chapter 1, Lemma 1.7).
This boundedness lends itself naturally to a histogram representation (regardless of the size of G ), making it possible to bin the eigenvalues and use the resulting histogram as a simpliﬁed descriptor of a graph.
The expressivity of such a representation is not clear a priori; the question of whether graphs are fully determined by their spectrum is still open (van Dam & Haemers, 2003) and has only been partially answered in the negative for certain classes of graphs (Schwenk, 1973).
I SSUES WITH THE CURRENT PRACTICE
Common practice in graph generative model papers is to use MMD by ﬁxing a kernel and parameter values for the descriptor functions and kernel values, and then assessing the newly-proposed model as well as its competitor models using said kernel and parameters.
Authors chose different values of σ for the different descriptor functions, but to the best of our knowledge, all parameters are set to a ﬁxed value a priori without any selection procedure.
If MMD were able to give results and rank different models in a stable way across different kernel and parameter choices, this would be inconsequential.
However, as our experiments will demonstrate, the results of MMD are highly sensitive to such choices and can therefore lead to an arbitrary ranking of models.
Subsequently, we use three current real-world models, GraphRNN
(You et al., 2018), GRAN (Liao et al., 2019), and Graph Score Matching (Niu et al., 2020).
We ran the models using the authorprovided implementations to generate new graphs on the Community, Barabási-Albert, ErdösRényi, and Watts-Strogatz graph datasets, and then calculated the MMD distance between the generated graphs and the test graphs, using (i) different kernels that they used (EMD, TV, RBF), (ii) descriptor functions (degree histogram, clustering coefﬁcient histogram, and Laplacian spectrum), and (iii) parameter ranges ( σ, λ ∈ { 10 − 5 , . . .
For simplicity, we will refer to the parameter in all kernels as σ .
We purposefully refrained from using model names , preferring instead ‘A, B, C’ in order to focus on the issues imposed by such an evaluation, rather than providing a commentary on the performance of a speciﬁc model.
In the following, we will delve deeper into issues originating from the individual components of the graph generative model comparison in use today.
Due to space constraints, examples are provided on individual datasets; full results across datasets are available in Appendix A.6.
4. N UANCES OF USING MMD FOR GRAPH GENERATIVE MODEL EVALUATION
While MMD may seem like a reasonable ﬁrst choice as a metric for comparing distributions of graphs, it is worth mentioning two peculiarities of such a choice and how authors are currently applying it in this context.
First, as mentioned before, MMD was originally developed as an approach to perform two-sample testing on structured objects such as graphs.
As such, its suitability was investigated in that context, and not in that of evaluating graph generative models.
This warrants an investigation of the implications of “porting” such a method from one context to another.
1.0 % Perturbation 0.
MM D
D i s t a n ce t
o O r
i g i n a l
G r a ph s (a) 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95 % Perturbation 0.
MM D
D i s t a n ce t
o O r
i g i n a l
G r a ph s
TVEMDRBF (b) 1e-05 8e-04 7e-02 6e+00 5e+02 4e+
N u m b e r o f
B i n s
Model A
Model B
Model C (c) The second peculiarity worth mentioning is that MMD was groundbreaking for its ability to compare distributions of structured objects by means of a kernel, thus bypassing the need to employ an intermediate vector representation. Yet, in its current application, the graphs are ﬁrst being vectorised, and then MMD is used to compare the vectors. MMD is not technically required in this case: any statistical method for comparing vector distributions could be employed, such as Optimal
Transport (OT).
While the use of different evaluators besides MMD for assessing graph generative models is an interesting area for future research, we focus speciﬁcally on MMD, since this is what is in use today, and now highlight two practical issues that can arise from using MMD in this context.
MMD does not always capture differences in distributions.
As two distributions become sufﬁciently dissimilar, we ﬁnd that their distance should monotonically increase as a function of their dissimilarity.
While one can construct speciﬁc scenarios in which two distributions become farther apart but the distance does not monotonically increase (e.g., removing edges from a triangle-free graph, and using the clustering coefﬁcient as the descriptor function), in such cases, the speciﬁc choice of descriptor function f is crucial to ensure that f can capture differences in distribution.
However, it is not guaranteed that MMD will monotonically increase as two distributions become increasingly dissimilar.
Figure 2b depicts this behaviour when focusing on a single descriptor function, the clustering coefﬁcient, with each line representing a unique kernel and parameter choice (see Appendix A.6 for the full results for other datasets and descriptor functions).
We subject an original set of graphs to perturbations of increasing magnitude and then measured the MMD distance to the original distribution.
Despite both distributions becoming progressively dissimilar by experimental design, a large number of kernel/parameter conﬁgurations fail to capture this, showing that MMD is highly sensitive to this choice.
In many instances, the distance remains nearly constant, despite the increased level of perturbation, until the magnitude of the perturbation reaches an extraordinarily high level.
In some cases, we observe that the distance even decreases as the degree of perturbation increases , suggesting that the original data set and its perturbed variant are more similar.
We also ﬁnd that the MMD values as a function of the degree of perturbation are highly sensitive to the kernel and parameter selection, as evidenced by the wide range of different curve shapes observed in Figure 2b.
MMD has no inherent scale.
Another challenge with MMD is that since there is no inherent scale of MMD values, it is difﬁcult to assess whether the smaller MMD distance of one model is substantially improved when compared to the MMD distance of another model.
For instance, suppose that the MMD distance of one model is 5 . 2 × 10 − 8 .
Is this a meaningfully better model than one whose MMD distance is 4 .
6 × 10 − 7 ?
This is further compounded by the choice of kernel and parameter, which also affects the scale of MMD.
Since authors often choose a suboptimal value of
10 4 10 2 10 0 10 2 10 4 log( ) 0.
MM D D
i s t a n ce t
o T e s t
G r a ph s
Model A Model B Model C 10 4 10 2 10 0 10 2 10 4 log( ) 0.
MM D D
i s t a n ce t
o T e s t
G r a ph s
Model A Model B Model C 10 4 10 2 10 0 10 2 10 4 log( ) 0.
MM D D
i s t a n ce t
o T e s t
G r a ph s
Model A Model B Model C 10 4 10 2 10 0 10 2 10 4 log( ) 0.
MM D D
i s t a n ce t
o T e s t
G r a ph s
Model A Model B Model C
[Figure]
EMD,
[Figure]
σ for MMD (Sutherland et al., 2017), they may obtain an arbitrarily low value of MMD, as seen in
Figure 3c and 3d.
As MMD results are typically reported just in a table, the lack of a scale hinders the reader’s ability to effectively assess a model’s performance.
4. C ONSEQUENCES OF THE CHOICE OF KERNEL
MMD requires the choice of a kernel function, yet there is no agreed-upon method to choose one.
We ﬁnd that each of the three models considered in this paper choose a different kernel for its evaluation, as evidenced in Table 1.
There are several potential issues with the current practice, which we will subsequently discuss, namely (i) the selected kernel might be computationally inefﬁcient to compute, as is the case for the previously-used Earth Mover’s Distance (EMD) kernel function, (ii) the use of positive deﬁnite kernel functions is a limitation, with previous work employing functions that are not valid kernels, and (iii) ﬁnally, the kernel choice may result in arbitrary rankings, and there is currently little intention paid to this choice.
Computational cost of kernel computation.
Issue (i) might prevent an evaluation metric to be used in practice, thus stymieing graph generative model development.
While the choice of a kernel using the ﬁrst Wasserstein distance is a valid choice, it is extremely slow to compute, as noted by
Liao et al.
From this perspective, it violates the third quality of our desiderata: efﬁciency.
Kernels need to be p.s.d.
To reduce the aforementioned computational costs, previous work (Liao et al., 2019) used a kernel based on the total variation distance between histograms, which has since been used in subsequent publications as one of the ways to evaluate different models.
As we show in
Appendix A.1, this approach leads to an indeﬁnite kernel (i.e. the kernel is neither positive deﬁnite nor negative deﬁnite), whose behaviour in the context of MMD is not well-deﬁned.
MMD necessitates the use of p.s.d.
kernels, and care must be taken when it comes to interpreting the respective results.
Arbitrary ranking purely based on kernel choice.
Issue (iii) relates to the fact that changing the kernel can lead to different results of model evaluation, which is problematic since each paper we considered used a different kernel.
For instance, with the degree distribution as the graph descriptor function, simply changing the choice of kernel from EMD to RBF (while holding the parameters constant) leads to a different ranking of the models, which can be seen in Figure 3a–Figure 3b, where the best performing model changes merely by changing the kernel choice (!).
This type of behaviour is highly undesired and problematic, as it implies that model performance (in terms of the evaluation) can be improved by choosing a different kernel.
4. E FFECT OF THE CHOICE OF HYPERPARAMETERS
While the choice of which kernel to use in the MMD calculation is itself an a priori design choice without clear justiﬁcation, it is further exacerbated by the fact that many kernels require picking parameters, without any clear process by which to choose them.
To the best of our knowledge, this selection of parameters is glossed over in publications at present.
For instance, Table 1 shows how authors are setting the value of σ differently for different descriptor functions, yet there is no discussion nor established best practice of how such parameters were or should be set.
Hyperparameter selection is known to be a crucial issue in machine learning that mandates clear selection algorithms in order to avoid biasing the results.
If the choice of parameters—similar to the choice of kernel—had no bearing on the outcome, this would not be problematic, but our empirical experiments prove that drastic differences in model evaluation performance can occur.
Arbitrary ranking based on parameter choice.
The small colour bars underneath each plot of
Figure 3 show the model that achieves the lowest MMD for a given value of σ .
Changes in colour highlight a sensitivity to speciﬁc parameter values.
Even though the plots showing MMD seem to have a general trend of which model is best in the peak of the curves, in the other regions of σ , the ranking switches, with the effect that a different model would be declared “best.”
This is particularly the case in Subﬁgures 3a, 3c, and 3d, where Model B appears to be the best, yet for much of σ , a different model ranks ﬁrst.
This sensitivity is further exacerbated by the fact that some descriptor functions require a parameter choice as well, such as the bin size, n bin for a histogram.
Figure 2c shows how the best-ranking model on the Barabási-Albert Graphs is entirely dependent upon the choice of parameters ( n bin , σ ).
The colour in each grid cell corresponds to the best-ranking model for a given σ and n bin ; we ﬁnd that this is wildly unstable across both n bin and σ .
The consequence of this is alarming: any model could rank ﬁrst if the right parameters are chosen.
Choice of σ by authors does not align with maximum discrimination in MMD.
Figures 3c and 3d show MMD for different values of σ with the clustering coefﬁcient descriptor function in the
Community Graphs dataset for Models A, B and C.
The value of σ as selected by the authors (without justiﬁcation or discussion) is indicated by the grey line; however, this choice corresponds to a regions of low activity in the MMD curve, suggesting a poor parameter choice (also shown in Sutherland et al.
While Model B seems to be the clear winner, the choice of σ by the authors resulted in
Model A having the best performance.
Furthermore, it is not clear that choosing the same σ across different kernels, as is currently done, makes sense; whereas σ = 10 3 would be sensible for EMD in
Figure 3c, for the Gaussian kernel in Figure 3d, such a choice too far beyond the discriminative peak.
H OW TO USE MMD FOR GRAPH GENERATIVE MODEL EVALUATION
Having understood the potential pitfalls of using MMD, we now turn to suggestions on how to better leverage MMD for graph generative model evaluation.
Provide a sense of scale. As mentioned in Section 4.1, MMD does not have an inherent scale, making it difﬁcult to assess what is ‘good.’
To endow their results with some meaning, practitioners should calculate MMD between the test and training graphs, and then include this in the results table/ﬁgures alongside the other MMD results.
This will provide a meaningful bound on what two ‘indistinguishable’ sets of graphs look like in a given dataset (see Figures 10–13 in Appendix A.6).
Choose valid and efﬁcient kernel candidates. We strongly recommend to avoid the EMD-based kernel due to the computational burden (see Appendix A.8), and the total variation kernel for its non-p.s.d nature.
Instead, we suggest using either an RBF kernel, since it is a universal kernel, or a
Laplacian kernel, or a linear kernel, i.e. the canonical inner product on R d , since it is parameter-free.
As all of these kernels are p.s.d.
, 4 and fast to compute, they satisfy the efﬁciency desiderata criteria, and thus only require analysis of their expressivity and robustness.
Utilize meaningful descriptor functions. Different descriptor functions measure different aspects of the graph and are often domain-speciﬁc.
As a general recommendation, we propose using previously-described (Liao et al., 2019; You et al., 2018) graph-level descriptor functions, namely (i) the degree distribution, (ii) the clustering coefﬁcient, and (iii) the Laplacian spectrum histograms, and recommend that the practitioner make domain-speciﬁc adjustments based on what is appropriate.
5. S ELECTING AN APPROPRIATE KERNEL AND HYPERPARAMETERS
The kernel choice and descriptor functions require hyperparameter selection.
We recommend assessing the performance of MMD in a controlled setting to elucidate some of the properties speciﬁc to the dataset and the descriptor functions of interest to a given application.
In doing so, it becomes possible to choose a kernel and parameter combination that will yield informative results.
Notice that in contrast to images, where visualisation provides a meaningful evaluation of whether two images are similar or not, graphs cannot be assessed in this manner.
It is thus necessary to have a principled approach where the degree of difference between two distributions can be controlled.
We subject a set of graphs to perturbations (edge insertions, removals, etc.) of increasing magnitude, thus enabling us to assess the expected degree of difference to the original graphs.
We ideally want an evaluation metric to effectively reﬂect the degree of perturbations.
Hence, with an increasing degree of perturbation of graphs, the distance to the original distribution G ∗ of unperturbed graphs should increase.
We can therefore assess both the expressivity of the evaluation metric, i.e., its ability to distinguish two distributions when they are different, and its robustness (or stability) based on how rapidly such a metric changes when subject to small perturbations.
Succinctly, we would like to see a clear correlation of the metric with the degree of perturbation, thus indicating both robustness and expressivity.
Perturbation experiments are particularly appealing because they do not require access to other models but rather only an initial distribution G ∗ .
This procedure therefore does not leak any information from models and is unbiased.
Moreover, researchers have more control over the ground truth in this scenario, as they can adjust the desired degree of dissimilarity.
While there are many perturbations that we will consider (adding edges, removing edges, rewiring edges, and adding connected nodes), we focus primarily on progressively adding or removing edges.
This is the graph analogue to adding “salt-and-pepper” noise to an image, and in its most extreme form (100% perturbation) corresponds to a fully-connected graph and fully-disconnected graph, respectively.
Creating dissimilarity via perturbations. For each perturbation type, i.e., (i) random edge insertions, (ii) random edge deletions, (iii) random rewiring operations (‘swapping’ edges), (iv) random node additions, we progressively perturb the set of graphs, using the relevant perturbation parameters, in order to obtain multiple sets of graphs that are increasingly dissimilar from the original set of graphs.
Each perturbation is parametrised by at least one parameter.
When removing edges, for instance, the parameter is the probability of removing an edge in the graph.
Thus for a graph with 100 edges and p remove = 0 .
1 we would expect on average 90 edges to remain in the graph.
Similar parametrisations apply for the other perturbations, i.e. the probability of adding an edge for edge insertions, the probability of rewiring an edge for edge rewiring, and the number of nodes to add to a graph, as well as the probability of an edge between the new node and the other nodes in the graph for adding connected nodes.
We provide a formal description of each process in Appendix A.2 and A.3.
Correlation analysis to choose a kernel and hyperparameters.
For each of the aforementioned perturbation types, we compared the graph distribution of the perturbed graphs with the original graphs using the MMD.
We repeated this for different scenarios, comprising different kernels, different descriptor functions, and where applicable, parameters.
For a speed up trick to efﬁciently calculate MMD over a range of σ , please see Appendix A.5.
Since these experiments resulted in hundreds of conﬁgurations, due to the choice of kernel, descriptor function, and parameter choices, we relegated most of the visualisations to the Appendix (see Section A.6).
To compare the different conﬁgurations effectively, we needed a way to condense the multitude of results into a more interpretable and comparable visualisation.
We therefore calculated Pearson’s correlation coefﬁcient between the degree of perturbation and the resulting MMD distance, obtaining two heatmaps.
The ﬁrst one shows the best parameter choice , the second one shows the worst parameter choice , both measured in terms of Pearson’s correlation coefﬁcient (Figure 4).
In the absence of an agreed-upon procedure to choose such parameters, the heatmaps effectively depict the extremes of what will happen if one is particularly “lucky” or “unlucky” in the choice of parameters. A robust combination of descriptor function and
BA
Community
ER
WS
CC D e g r ee L a p l ac i a n
BA
Community
ER
WS (a) AddEdges
BA
Community
ER
WS
CC D e g r ee L a p l ac i a n
BA
Community
ER
WS (b) RemoveEdges
BA
Community
ER
WS
CC D e g r ee L a p l ac i a n
BA
Community
ER
WS (c) RewireEdges
BA
Community
ER
WS
CC D e g r ee L a p l ac i a n
BA
Community
ER
WS (d) AddConnectedNodes 1.
comparison function is characterised by both heatmaps exhibiting high correlation with the degree of perturbation.
In the bottom chart, the MMD distance in many cases no longer shows any correlation with the degree of perturbation, and in the case of the clustering coefﬁcient (CC), is even negatively correlated with the degree of perturbation.
Such behaviour is undesired, showcasing a potential pitfall for authors if they inadvertently fail to pick a “good” parameter or kernel combination.
While we chose the Pearson correlation coefﬁcient for its simplicity and interpretability, other measures of dependence could be used instead if they are domain-appropriate (see Appendix A.7).
To choose a kernel and parameter combination, we suggest picking the parameters with the highest correlation for the perturbation that is most meaningful in the given domain.
Lacking this, the practitioner could choose the combination that has the highest average correlation across perturbations.
C ONCLUSION
We provided a thorough analysis of how graph generative models are being currently assessed by means of MMD.
While MMD itself is powerful and expressive, its use has certain idiosyncratic issues that need to be avoided in order to obtain fair and reproducible comparisons.
We highlighted some of these issues, most critical of which are that the choice of kernel and parameters can result in different rankings of different models, and that MMD may not monotonically increase as two graph distributions become increasingly dissimilar.
As a mitigation strategy, we propose running a perturbation experiment as described in this paper to select a kernel and parameter combination that is highly correlated with the degree of perturbation.
This way, the choice of parameters does not depend on the candidate models but only on the initial distribution of graphs.
Future work.
This work gives an overview of the current situation, illuminates some issues with the status quo, and provides practical solutions.
We hope that this will serve as a starting point for the community to further develop methods to assess graph generative models.
Future work could investigate the use of efﬁcient graph kernels in combination with MMD, as described in a recent review (Borgwardt et al., 2020).
This would reduce the comparison pipeline in that graph kernels can be directly used with MMD, making graph descriptor functions unnecessary.
Another approach could be to investigate alternative or new descriptor functions, such as the geodesic distance, or alternative evaluation methods, such as the multivariate Kolmogorov–Smirnov test (Justel et al., 1997), or even develop totally novel evaluation strategies, preferably those that go beyond the currently-employed vectorial representations of graphs.
R EPRODUCIBILITY S TATEMENT
We have provided the code for our experiments in order to make our work fully reproducible.
Details can be found in Appendix A.2-A.4, which includes a link to our GitHub repository.
During the review process our code is available as a Supplementary Material, in order to preserve anonymity
A CKNOWLEDGMENTS
This work was supported in part by the Alfried Krupp Prize for Young University Teachers of the
Alfried Krupp von
Bohlen und Halbach-Stiftung (K.B.).
A A PPENDIX
The following sections provide additional details about the issues with existing methods.
We also show additional plots of our ranking and perturbation experiments.
A. K ERNELS BASED ON TOTAL VARIATION DISTANCE
Previous work used kernels based on the total variation distance in order to compare evaluation functions via MMD.
The choice of this distance, however, requires subtle changes in the selection of kernels for MMD—it turns out that the usual RBF kernel must not be used here!
We brieﬂy recapitulate the deﬁnition of the total variation distance before explaining its use in the kernel context: given two ﬁnite-dimensional real-valued histograms X := { x 1 , . . .
, x n } and
Y :=
{ y 1 , . . .
, y n } , their total variation distance is deﬁned as
[Equation]
This distance induces a metric space that is not ﬂat .
In other words, the metric space induced by Eq.has non-zero curvature (a fact that precludes certain kernels from being used together with Eq. 3).
We can formalise this by showing that the induced metric space cannot have a bound on its curvature.
Theorem 1.
The metric space X TV induced by the total variation distance between two histograms is not in CAT( k ) for k > 0 , where CAT( k ) refers to the category of metric spaces with curvature bounded from above by k (Gromov, 1987).
Proof.
Let x 1 = (1 , 0 , . .
. , 0) and x 2 = (0 , 1 , 0 , . .
There are at least two geodesics—shortest paths—of the same length, one that ﬁrst decreases the ﬁrst coordinate and subsequently increases the second one, whereas for the second geodesic this order is switched.
More precisely, the ﬁrst geodesic proceeds from x 1 to x 1 − ( (cid:15), 0 , . .
. , 0) for an inﬁnitesimal (cid:15) > 0 , until x 0 = (0 , . .
. , 0) has been reached.
Following this, the geodesic continues from x 0 to x 0 + (0 , (cid:15), 0 , . . . , 0) in inﬁnitesimal steps until x 2 has been reached.
The order of these two operations can be switched, such that the geodesic goes from x 1 to (1 , 1 , 0 , . . . , 0) , from which it ﬁnally continues to x 2 .
Both of these geodesics have a length of 1 .
Since geodesics in a CAT( k ) space for k > 0 are unique (Bridson & Haeﬂiger, 1999,
Proposition 2.11, p. 23), X TV is not in CAT( k ) for k > 0 .
Since every CAT( k ) space is also a CAT( l ) space for all l > k , this theorem has the consequence that X TV cannot be a CAT(0) space.
Moreover, as every ﬂat metric space is in particular a CAT(0) space, X TV is not ﬂat.
According to Theorem 1 of Feragen et al.
, the associated geodesic
Gaussian kernel, i.e. the kernel that we obtain by writing
[Equation]
is not positive deﬁnite and should therefore not be used with MMD.
One potential ﬁx for this speciﬁc distance involves using the Laplacian kernel, i.e.
[Equation]
The subtle difference between these kernel functions—only an exponent is being changed— demonstrate that care must be taken when selecting kernels for use with MMD.
A. E XPERIMENTAL SETUP
We can analyse the desiderata outlined above using an experimental setting.
In the following, we will assess expressivity, robustness, and efﬁciency for a set of common perturbations, i.e. (i) random edge insertions, (ii) random edge deletions, (iii) random rewiring operations, i.e. ‘swapping’ edges, and (iv) random node additions.
For each perturbation type, we will investigate how the metric changes for an ever-increasing degree of perturbation, where each perturbation is parametrized by at least one parameter.
When removing edges, for instance, this is the probability of removing an edge in the graph.
Thus for a graph with 100 edges and p remove = 0 .
1 we would expect on average 90 edges to remain in the graph.
Similar parametrizations apply for the other perturbations, for an more detailed description we refer to Appendix A.3.
All these operations are inherently small-scale (though not necessarily localised to speciﬁc regions within a graph), but turn into large-scale perturbations of a graph depending on the strength of the perturbation performed.
We performed these perturbations using our own Python -based framework for graph generative model comparison and evaluation.
Our framework additionally permits the simple integration of additional descriptor functions and evaluators.
A. D ETAILS ON GRAPH PERTURBATIONS
We describe all graph perturbations used in this work on the example of a single graph G :
= ( V, E ) where V refers to the vertices of the graph and E to the edges.
Add Edges For each v i , v j ∈ V with v i (cid:54) = v j a sample from a Bernoulli distribution x ij ∼
Ber ( p add ) is drawn.
Samples for which x ij = 1 , are added to the list of edges such that E (cid:48) =
E ∪ { ( v i , v j ) | x ij = 1 } .
Remove Edges For each e i ∈
E a sample from a Bernoulli distribution x i ∼ Ber ( p remove ) is draw, samples with x i = 1 are removed from the edge list, such that E (cid:48)
=
E ∩ { e i | x i (cid:54) = 1 } .
Rewire Edges For each e i ∈
E a sample from a Bernoulli distribution x i ∼ Ber ( p rewire ) is drawn, samples with x i = 1 are rewired.
For rewiring a further random variable y i ∼ Ber (0 . 5) is drawn which determines which node e i [ y i ] of the edge e
i is kept.
The node to which the edge is connected is chosen uniformly from the set of vertices v i ∈ V , where v i / ∈
e i , i.e. avoiding self-loops and reconnecting the original edge.
Finally the original edge is removed and the new edge e (cid:48)
i =
( e i
[ y i ] , v i ) is added to the graph E (cid:48)
=
E ∩ { e i | x i (cid:54) = 1 } ∪ { e (cid:48) i | x i = 1 } .
Add Connected Node We deﬁne a set of vertices to be added V ∗ =
{ v i | | V | < i ≤ | V | + n } , where n represents the number of nodes to add.
For each v i ∈ V and v j ∈ V ∗ we draw a sample from a Bernoulli distribution x ij ∼ Ber ( p connect_node ) and an edge between v i and v j to the graph if x ij = 1 .
Thus E (cid:48)
=
E ∪ { ( v i , v j ) | v i ∈ V, v j ∈ V ∗ , x ij = 1 } .
A. I MPLEMENTATION DETAILS
We used the ofﬁcial implementations of GraphRNN, GRAN and Graph Score Matching in our experiments.
GraphRNN and GRAN both have an MIT License, and Graph Score Matching is licensed under GNU General Public License v3.0.
To preserve anonymity in the review process, we will censor the GitHub link ( https:/www.github.com/BorgwardtLab/ggme ), which will be available upon publication under a BSD 3-Clause license.
Compute resources.
All the jobs were run on our internal cluster, comprising 64 physical cores ( Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz ) with 8 GeForce GTX
GPUs.
We stress that the main component of this paper, i.e. the evaluation itself, do not necessarily require a cluster environment.
The cluster was chosen because individual generative models had to be trained in order to obtain generated graphs, which we can subsequently analyse and rank.
A. S PEED UP TRICK
As the combination of kernels and hyperparameters yielded hundreds of combinations, it is worth mentioning a worthwhile speedup trick to reduce the complexity of assessing the Gaussian and
Laplacian kernel combinations.
Considering they have a shared intermediate value, namely the
Euclidean distance, it is possible to return intermediate values in the MMD computation ( K XX ,
K Y Y , and K XY , prior to exponentiation, potential squaring, and scaling by σ .
Storing these intermediate results allows one to rapidly iterate over a grid of values for σ without needed to recalculate MMD, leading to a worthwhile speedup.
A. E XPERIMENTAL RESULTS
We now provide the results presented in the paper across all the datasets.
[Figure]
MM
D D i s t a n ce fn = degree   0.
fn = clustering   0.
k e r n e l = e m d fn = laplacian   0.
MM
D D i s t a n ce   0.
k e r n e l = t v   log( ) 0.
MM
D D i s t a n ce   log( ) 0.
log( ) 0.
k e r n e l = g a u
ss i a n modelModel A Model B
Model C train
MM
D D i s t a n ce fn = degree   0.
fn = clustering   0.
k e r n e l = e m d fn = laplacian   0.
MM
D D i s t a n ce   0.
k e r n e l = t v   log( ) 0.
MM
D D i s t a n ce   log( ) 0.
log( ) 0.
k e r n e l = g a u
ss i a n modelModel A Model B
Model C train
MM
D D i s t a n ce fn = degree   0.
fn = clustering   0.
k e r n e l = e m d fn = laplacian   0.
MM
D D i s t a n ce   0.
k e r n e l = t v   log( ) 0.
MM
D D i s t a n ce   log( ) 0.
log( ) 0.
k e r n e l = g a u
ss i a n modelModel A Model B
Model C train
MM
D D i s t a n ce fn = degree   0.
fn = clustering   0.
k e r n e l = e m d fn = laplacian   0.
MM
D D i s t a n ce   0.
k e r n e l = t v   log( ) 0.
MM
D D i s t a n ce   log( ) 0.
log( ) 0.
k e r n e l = g a u
ss i a n modelModel A Model B
Model C train
[Table]
t o
[Table]
r i g i n a l
[Table]
r a
[Table]
D i s t a n ce t
o O r
i g i n a l
G r a ph s 0.
1.0 % Pertubation 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s 0.
1.0 % Pertubation 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s 0.
1.0 % Pertubation 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s
D e s c
r
i p t
o r
F un c ti on =
L a p l ac i a n
[Table]
t r i g i
[Table]
[Table]
[Figure]
D i s t a n ce t
o O r
i g i n a l
G
[Figure]
a
[Figure]
s [Figure] i s t a n ce t
[Figure]
O r i g i n a l G r a
[Figure]
s [Figure] t [Figure]
r i g i n a l
G r a ph s
[Figure]
D i s t a n ce t
o O r
i g i n a l
G r a ph s
D e s c
r
i p t
o r
F un c ti on =
L a p l ac i a n
[Table]
t a
[Table]
t
[Table]
r i g i n a l
[Table]
a
[Table]
i
[Table]
t
[Table]
t
[Table]
r i g i n a l
[Table]
r a
[Table]
s
[Table]
[Figure]
[Figure]
[Figure]
o O r i g i n a l
G r a ph s 0.
1.0 % Pertubation
D i s t a n ce t
o O r
i g i n a l
G r a ph s 1e 0.
1.0 % Pertubation
D i s t a n ce t
o O r
i g i n a l
G r a ph s 1e 0.
1.0 % Pertubation 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s D e s c r i p t
o r
F un c ti on =
L a p l ac i a n
[Table]
t
[Table]
t
[Table]
r i g
[Table]
n
[Table]
l
[Table] D i s t a n ce t
o O r
i g i n a l
G r a ph s
[Table]
n
[Table]
[Figure]
[Table]
[Figure]
D i s t a n ce t
o O r
i g i n a l
G r a ph s    14 # Nodes Added
D i s t a n ce t
o O r
i g i n a l
G r a ph s    14 # Nodes Added 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s    14 # Nodes Added 0.
D i s t a n ce t
o O r
i g i n a l
G r a ph s
D e s c
r
i p t
o r
F un c ti on =
L a p l ac i a n
A. A LTERNATIVE MEASURES OF CORRELATION
As a general recommendation, we chose to use the Pearson correlation coefﬁcient to select a good kernel-hyperparameter combination, due to its simplicity and ease of interpretation.
It reﬂects the behavior we expect from perturbations: as the graphs are increasingly perturbed, the distance to the original graphs should in a similar manner grow as well.
However, there could be scenarios in which the distance to the original graphs should not grow linearly with the degree of perturbation, in which case the Pearson correlation coefﬁcient would not be the best choice.
We present here two plug-in alternatives for the Pearson correlation coefﬁcient, namely the Spearman rank correlation coefﬁcient, and mutual information, which can easily be integrated into our framework.
We add one word of caution when using the mutual information, which is the fact that it does not capture the directionality of dependence.
This is only an issue if there are scenarios in which the MMD distance decreases as the degree of perturbation increases.
Since we observed this in some of our datasets, it would not be the most appropriate to use in this speciﬁc case.
We now present the results from using these measures of dependence below.
[Figure]
[Figure]
BA
Community
ER
WS
CC D e g r ee L a p l a c i a n
BA
Community
ER
WS (a) AddEdges
BA
Community
ER
WS
CC D e g r ee L a p l a c i a n
BA
Community
ER
WS (b) RemoveEdges
BA
Community
ER
WS
CC D e g r ee L a p l a c i a n
BA
Community
ER
WS (c) RewireEdges
BA
Community
ER
WS
CC D e g r ee L a p l a c i a n
BA
Community
ER
WS (d) AddConnectedNodes 0.
[Figure]
A. C OMPUTATIONAL RUNTIME OF DIFFERENT KERNELS
In the following, we assess the empirical CPU runtime of the linear kernel, RBF kernel, and the
EMD-based kernel in the MMD calculation.
As noted by ourselves and Liao et al.
the
EMD-based kernel is computationally expensive, hindering its suitability for use in graph generative model evaluation.
We investigate this effect in more detail by doing a runtime comparison of the three kernels in a simulated environment.
We generate two sets of Erdös-Rényi Graphs with the probability of an edge p = 0 . 3 , and then calculate the MMD distance between the two sets using the degree distribution as the descriptor function for a varying dataset size ( n graphs ), graph size ( n nodes ), and histogram bin size ( n bins ).
As a default, we set n graphs = 100 , n nodes = 100 , and n bins = 100 .
We then change one variable at a time, iterating through values { 100 , 200 , . .
. , 1000 } while keeping the other two variables ﬁxed, and measured the time it took to calculate the MMD distance.
We report the average runtime over ten repetitions to obtain more stable results.
Our results can be seen in Figure 16.
At the default setting of 100 graphs, 100 nodes per graph, and 100 bins in the histogram, we observed that the EMD-based kernel took more than 50times longer to compute compared to the RBF and linear kernels respectively.
This difference was exacerbated when the size of the dataset ( n graphs ) increased as well as when the size of the graphs in the dataset ( n nodes ) increased.
For a dataset of 1,000 graphs, the EMD-based kernel took 27 minutes to run, whereas the linear kernel took 6 seconds and the RBF kernel took 21 seconds.
Doing a simple extrapolation of the runtime of a dataset comprising 10,000 graphs, we estimate the
EMD-based kernel would take 50 hours to run for a single hyperparameter combination, showcasing the computational limitation of this choice of kernel for larger dataset sizes.
We observed a similar, albeit less extreme, increase in runtime when the size of the graphs increased.
While the linear time approximation of MMD (Gretton et al., 2012a) could mitigate some of the runtime challenges, in general we recommend that practitioners use efﬁcient kernels such as the linear kernel or RBF kernel for graph generative model evaluation.