O N E VALUATION M ETRICS FOR G RAPH G ENERATIVE M ODELS
A BSTRACT
In image generation, generative models can be evaluated naturally by visually inspecting model outputs.
However, this is not always the case for graph generative models (GGMs), making their evaluation challenging.
Currently, the standard process for evaluating GGMs suffers from three critical limitations: i) it does not produce a single score which makes model selection challenging, ii) in many cases it fails to consider underlying edge and node features, and iii) it is prohibitively slow to perform.
In this work, we mitigate these issues by searching for scalar domain - agnostic and scalable metrics for evaluating and ranking
GGMs To this end, we study existing GGM metrics and neural-network-based metrics emerging from generative models of images that use embeddings extracted from a task-speciﬁc network.
Motivated by the power of Graph Neural Networks (GNNs) to extract meaningful graph representations without any training , we introduce several metrics based on the features extracted by an untrained random GNN.
We design ex - periments to thoroughly test and objectively score metrics on their ability to measure the diversity and ﬁdelity of generated graphs as well as their sample and com putational efﬁciency
Depending on the quantity of samples, we recommend one of two metrics from our collection of random-GNN-based metrics. We show these two metrics to be more expressive than pre -existing and alternative random -GNN- based metrics using our objective scoring.
While we focus on applying these met - rics to GGM evaluation, in practice this enables the ability to easily compute the dissimilarity between any two sets of graphs regardless of domain . Our code is released at: https://github.com/uoguelph-mlrg/GGM-metrics .
I NTRODUCTION
Graph generation is a key problem in a wide range of domains such as molecule generation (Samanta et al., 2020;
Popova et al., 2019; Li et al., 2018; Kong et al., 2021; Jin et al., 2020) and structure generation (Bapst et al., 2019; Thompson et al., 2020).
An evaluation metric that is capable of accurately measuring the distance between a set of generated and reference graphs is critical for advancing research on graph generative models (GGMs).
This is frequently done by comparing empirical distributions of graph statistics such as orbit counts, degree coefﬁcients, and clustering coefﬁcients through Maximum Mean Discrepancy (MMD)
(You et al., 2018; Gretton et al., 2006).
While these metrics are capable of making a meaningful comparison between generated and real graphs (You et al., 2018), this evaluation method yields a metric for each individual statistic.
In addition, recent works have further increased the number of metrics by performing MMD directly with node and edge feature distributions (Goyal et al., 2020), or on alternative graph statistics such as graph spectra (Liao et al., 2019).
While this is not an issue provided there is a primary statistic of interest, all metrics are frequently displayed together to approximate generation quality and evaluate
GGMs (You et al., 2018; Liao et al., 2019).
This process makes it challenging to measure progress as the ranking of generative models may vary between metrics.
In addition, the computation of the metrics from You et al.
(2018) can be prohibitively slow (Liao et al., 2019; O’Bray et al., 2022), and they are based only on graph structure, meaning they do not incorporate edge and node features.
Therefore, they are less applicable in speciﬁc domains such as molecule generation where such features are essential.
This particular limitation has led to the use of the Neighborhood Subgraph
Pairwise Distance kernel (NSPDK) (Costa & Grave, 2010) in GGM evaluation (Goyal et al., 2020;
Podda & Bacciu, 2021; Kawai et al., 2019) as it naturally incorporates edge and node features.
However, this metric is still unable to incorporate continuous features in evaluation (Costa & Grave, 2010).
Faced with a wide array of metrics and ambiguity regarding when each should be the focus, the community needs robust and scalable standalone metrics that can consistently rank GGMs.
While less popular, metrics from image generation literature have been successfully utilized in GGM evaluation.
These metrics rely on the use of a task-speciﬁc neural network to extract meaningful representations of samples, enabling a more straightforward comparison between generated and reference distributions (Preuer et al., 2018;
Liu et al., 2019; Thompson et al., 2020).
Although these metrics have been validated empirically in the image domain, they are not universally applicable to
GGMs.
For example, Fr´echet Chemnet Distance (Preuer et al., 2018) uses a language model trained on SMILES strings, rendering it unusable for evaluation of GGMs in other domains.
Furthermore, a pretrained GNN cannot be applied to datasets with a different number of edge or node labels.
Pretraining a GNN for every dataset can be prohibitive, making the use of such metrics in GGM evaluation less appealing than in the more established and standardized image domain.
In image generation evaluation, classiﬁers trained on ImageNet (Deng et al., 2009) are frequently used to extract image embeddings (Bi´nkowski et al., 2018; Heusel et al., 2017; Kynk¨a¨anniemi et al., 2019; Xu et al., 2018; Naeem et al., 2020).
While classiﬁers such as Inception v3 (Szegedy et al., 2016) are consistently used, recent works have investigated the use of randomly-initialized CNNs with no further training (hereafter referred to as a random network ) in generative model evaluation.
Xu et al.
; Naeem et al.
(2020) found that a random CNN performs similarly to ImageNet classiﬁers on natural images and is superior outside of the natural image domain.
In the graph domain, random GNNs have been shown to extract meaningful features to solve downstream graph tasks without training (Kipf & Welling, 2017; Morris et al., 2019; Xu et al., 2019).
However, the applicability of random GNNs for the evaluation of GGMs remains unexplored.
In this work, we aim to identify one or more scalar metrics that accurately measures the dissimilarity between two sets of graphs to simplify the ranking of GGMs regardless of domain.
We tackle this problem by exploring the use of random GNNs in the evaluation of GGMs using metrics that were developed in the image domain.
In addition, we perform objective evaluation of a large number of possible evaluation metrics.
We design experiments to thoroughly test each metric on its ability to measure the diversity and ﬁdelity (realism) of generated graphs, as well as their sample and computational efﬁciency.
We study three families of metrics: existing GGM evaluation metrics based on graph statistics and graph kernels, which we call classical metrics ; image domain metrics using a random GNN; and image domain metrics using a pretrained GNN.
We aim to answer the following questions empirically: ( Q1 )
What are the strengths and limitations of each metric?
( Q2 )
Is pretraining a GNN necessary to accurately evaluate GGMs with image domain metrics?
( Q3 )
Is there a strong scalar and domain-agnostic metric for evaluating and ranking GGMs?
Addressing these questions enabled us to reveal several surprising ﬁndings that have implications for GGM evaluation in practice.
For example, regarding Q1, we identify a failure mode in the classical metrics in that they are poor at measuring the diversity of generated graphs.
Consequently, we ﬁnd several metrics that are more expressive.
In terms of Q2, we determine that pretraining is unnecessary to utilize neural-network-based (NN-based) metrics.
Regarding Q3, we ﬁnd two scalar metrics that are appropriate for evaluating and ranking GGMs in certain scenarios; they are scalable, powerful, and can easily incorporate continuous or discrete node and edge features.
These ﬁndings enable computationally inexpensive and domain-agnostic GGM evaluation.
B ACKGROUND & RELATED WORK
Evaluating generative models in any domain is a notoriously difﬁcult task (Theis et al., 2016).
Previous work on generative models has typically relied on two families of evaluation metrics: samplebased (Heusel et al., 2017) and likelihood-based (Theis et al., 2016).
However, comparing the log-likelihood of autoregressive GGMs is intractable as it requires marginalizing over all possible node orderings (Chen et al., 2021).
While recent work learns an optimal ordering and estimates this marginal Chen et al.
(2021), it has been shown previously that likelihood may not be indicative of generation quality (Theis et al., 2016). Sample-based evaluation metrics estimate the distance ρ between real and generated distributions P r and P g by drawing random samples (Heusel et al., 2017; You et al., 2018; Bi´nkowski et al., 2018).
That is, they compute ˆ ρ ( S g , S r )
≈
ρ
( P g , P r ) , with S r = { x r 1 , . . .
, x rm } ∼ P r and S g = { x g 1 , . . .
, x gn } ∼ P g , where x i is deﬁned as some feature vector extracted from a corresponding graph G i .We use sample-based metrics throughout as they are model agnostic and therefore applicable to all GGMs.
2. C LASSICAL METRICS
Metrics based on graph statistics (You et al., 2018) are standard in evaluating GGMs (Liao et al., 2019; Dai et al., 2020).
These metrics set x i to be the clustering coefﬁcient, node degree, or 4-node orbit count histograms 1 that are then used to compute the empirical MMD between generated and reference sets S g , S r (Gretton et al., 2006):
[Equation]
where k ( · , · ) is a general kernel function.
You et al.
(2018) proposed a form of the RBF kernel:
[Equation]
where d ( · , · ) computes pairwise distance, and in that work was chosen to be the Earth Mover’s
Distance (EMD).
This yields three metrics, one for each graph statistic.
The computational cost of these metrics may be decreased by using the total variation distance as d ( · , · ) in Equation 1 (Liao et al., 2019).
However, this change leads to an indeﬁnite kernel and undeﬁned behaviour (O’Bray et al., 2022).
Therefore, we only compute these metrics using EMD (You et al., 2018).
In addition, several works (Goyal et al., 2020; Podda & Bacciu, 2021; Kawai et al., 2019) evaluate GGMs by replacing k ( · , · ) with the Neighborhood Subgraph Pairwise Distance graph kernel (NSPDK).
This metric has the beneﬁt of incorporating discrete edge and node features along with the underlying graph structure in evaluation.
Similar to the metrics proposed by You et al.
(2018), Moreno et al.
(2018) extract graph structure properties such as node degree, clustering coefﬁcient, and geodesic distance.
However, these properties are then combined into a scalar metric through the KolmorogovSmirnov (KS) multidimensional distance (Justel et al., 1997).
We exclude KS from our experiments as it is unable to incorporate edge and node features, which is one of the key properties we seek.
Finally, note that other domain-speciﬁc metrics such as “percentage of valid graphs” exist.
Our goal is not to incorporate, eliminate, or evaluate such metrics; they are properties of generated graphs, and unlike the metrics described above do not provide a comparison to a reference distribution.
We believe that such metrics can still provide valuable information in GGM evaluation.
2. G RAPH NEURAL NETWORKS
We denote a graph as G =
( V , E ) with vertices V and edges E
= { ( i, j ) | i, j ∈ { 1 , . . , | V |}} .
GNNs allow the extraction of a ﬁxed size representation x
i of an arbitrary graph G i .
While many
GNN formulations exist (Wu et al., 2020), we consider Graph Isomorphism Networks (GINs) (Xu et al., 2019) as a common GNN.
GINs consist of L propagation layers followed by a graph readout layer to obtain x i .
For node v ∈ V , the node embeddings h ( l ) v at layer l ∈ [1 , L ] are computed as:
[Equation]
where h (0) v is the input feature of node v , h ( l ) v ∈ R d ∀ l > 0 denotes a d -dimensional embedding of node v after the l -th graph propagation layer; N ( v ) denotes the neighbors of node v ; MLP ( l ) is a fully-connected neural network; f ( l ) is some aggregation function over nodes such as mean, max or sum.
A graph readout layer with skip connections aggregates features from all nodes at each layer l ∈ [1 , L ] and concatenates them into a single L ·
d dimensional vector x i (Xu et al., 2019):
[Equation]
where READOUT is similar to f ( l ) and is often chosen as the mean, max, or sum operation.
2. N EURAL - NETWORK - BASED METRICS
NN-based metrics utilize a task-speciﬁc network to extract descriptive multidimensional embeddings of the input data.
In image generation evaluation, the activations of a hidden layer in Inception [Figure] v3 (Szegedy et al., 2016) are frequently used as vector representations of the input images (Heusel et al., 2017).
These NN-based metrics can be applied to GGM evaluation by replacing Inception v3 with a GNN (Thompson et al., 2020; Liu et al., 2019).
This prompts the use of a wide range of evaluation metrics that have been studied extensively in the image domain.
Computation of these metrics follow a common setup and differ only in how the distance between two sets of data are determined (Figure 1).
Fr´echet Distance (FD) (Heusel et al., 2017) is one of the most popular generative metrics.
FD approximates the graph embeddings as continuous multivariate Gaussians with sample mean and covariance µ, C .
The distance between distributions is computed as an approximate measure of sample quality: FD
[Equation]
Improved Precision & Recall (P&R) (Kynk¨a¨anniemi et al., 2019) decouples the quality of a generator into two separate values to aid in the detection of mode collapse and mode dropping .
Mode dropping refers to the case wherein modes of P r are underrepresented by P g , while mode collapse describes a lack of diversity within the modes of P g .
Manifolds are constructed by extending a radius from each embedded sample in a set to its k th nearest neighbour to form a hypersphere, with the union of all hyperspheres representing a manifold.
Precision is the percentage of generated samples that fall within the manifold of real samples, while Recall is the percentage of real samples that fall within the manifold of generated samples.
The harmonic mean (“F1 PR”) of P&R is a scalar metric that can be decomposed into more meaningful values in experiments (Lucic et al., 2018).
Density & Coverage (D&C) (Naeem et al., 2020) have recently been introduced as robust alternatives for Precision and Recall, respectively.
As opposed to P&R which take the union of all hyperspheres to create a single manifold for each set, D&C operate on each samples hypersphere independently.
Density is calculated as the number of real hyperspheres a generated sample falls within on average.
Coverage is described as the percentage of real hyperspheres that contain a generated sample.
The hyperspheres used are found using the k th nearest neighbour as in P&R. As with P&R, the harmonic mean (“F1 DC”) of D&C can be used to create a scalar metric (Lucic et al., 2018).
MMD (Gretton et al., 2006) (Equation 1) can also be used to measure the dissimilarity between graph embedding distributions.
The original Kernel Inception Distance (KID) (Bi ´ nkowski et al., 2018) proposed a polynomial kernel with MMD, k
( x i , x j ) =
( 1 d x (cid:62) i x j + 1) 3 , where d is the embedding dimension.
The linear kernel k ( x i , x j ) =
x i · x j is another parameter-free kernel used with
MMD to evaluate generative models (O’Bray et al., 2022).
In addition, the RBF kernel (Equation 2) with d ( · , · ) as the Euclidean distance is widely used (Xu et al., 2018; Gretton et al., 2006).
The choice of σ in Equation 2 has a signiﬁcant impact on the output of the RBF kernel, and methods for ﬁnding and selecting an optimal value is an important area of research (Bach et al., 2004; Gretton et al., 2012a;b).
A common strategy is to select σ as the median pairwise distance between the two comparison sets (Garreau et al., 2018; Gretton et al., 2006; Sch¨olkopf et al., 1998).
Another strategy is to evaluate MMD using a small set of σ values and select the value that maximizes MMD: σ = arg max σ ∈ Σ MMD( S g , S r ; σ ) (Sriperumbudur et al., 2009).
We combine these ideas in our σ selection process, and more details are available in Appendix A.
We refer to these metrics as “KD,” “MMD Linear,” and “MMD RBF.”
2. B ENCHMARKING EVALUATION METRICS
Our work is closely related to other works benchmarking evaluation metrics of generative models (O’Bray et al., 2022; Xu et al., 2018).
A consistent method for assessing evaluation metrics is to start with the creation of “reference” and “generated” sets S r and S g that originate from the same real distribution P r .
Then, one gradually increases distortion to the generated set while recording each metric’s response to the distortion.
In the image domain, Xu et al.
(2018) develop multiple experiments to test metrics for key properties.
The metrics are then subjectively evaluated for these proper-
[Figure]
h
[Figure]
hhhhhhhhhhhhhhhhhhhhhhhhh [Figure] [Figure] ties by manually analyzing each metric’s response across experiments.
In concurrent work, O’Bray et al.
(2022) introduced additional GGM evaluation metrics.
Similar to Xu et al.
(2018), the metrics are validated through experiments that slowly increase the level of perturbation in each graph in the generated set.
However, O’Bray et al.
(2022) incorporated an objective evaluation score by computing the Pearson correlation of each metric with the degree of perturbation.
Although objective, evaluation using Pearson correlation is biased in that the relationship between the metric and degree of perturbation need not be linear.
Our work is also unique in considering random embeddings.
T HE EFFECTIVENESS OF RANDOM GNN S
While Inception v3 (Szegedy et al., 2016) has found widespread use in evaluation of generated samples (Salimans et al., 2016; Cai et al., 2018; Lunz et al., 2020), there is no analogue in graph-based tasks.
This prevents a standardized analysis of GGMs.
To tackle this, a single GNN may be pretrained on multiple datasets such that it extracts meaningful embeddings on new datasets (Hu et al., 2020).
However, node and edge features often have incompatible dimensions across graph datasets.
In addition, the distributions of graphs in the pretraining and target tasks can vary drastically, degrading the performance of a pretrained network.
However, random GNNs are capable of extracting meaningful features and solving many graph tasks (Kipf & Welling, 2017; Morris et al., 2019; Xu et al., 2019).
Thus, avoiding pretraining and instead utilizing random GNNs may be a viable strategy to bypass these issues.
To preliminarily test this approach, we apply permutations to Grid graphs by randomly rewiring edges with probability p .
Therefore, as we increase p , the dissimilarity between the original graphs and permuted graphs increases.
Thus, if a GNN is capable of extracting strong representations from graphs, the dissimilarity between graph embeddings should also increase with p .
We visualize the embeddings extracted from a pretrained GIN (Xu et al., 2019) and a random GIN in Figure 2, and ﬁnd that they extract very similar representations throughout this experiment.
This indicates that both random and pretrained GINs may be capable of evaluating GGMs.
We use GIN throughout all experiments due to its theoretical ability to detect graph isomorphism (Xu et al., 2019), however, we also provide a comparison to other common GNNs in Appendix C.6.
E XPERIMENTS
In this section, we describe key properties of a strong GGM evaluation metric and thoroughly test each metric for these properties.
These properties include a metric’s ability to correlate with the ﬁdelity and diversity of generated graphs, its sample efﬁciency, and its computational efﬁciency.
We argue that these properties capture many desired characteristics of a strong evaluation metric and enable reliable ranking of GGMs.
In many cases, we adapt the experiment design of Xu et al.
(2018) to the graph domain.
Datasets.
We experiment using six diverse graph datasets to test each metric’s ability to evaluate GGMs across graph domains (Table 1).
In particular, we include common GGM datasets such as Lobster, Grid, Proteins, Community, and Ego (You et al., 2018; Liao et al., 2019; Dai et al., 2020).
In addition, we utilize the molecular dataset ZINC (Irwin et al., 2012) strictly to demonstrate the ability of each metric to detect changes in node and edge feature distributions (Section 4.3).
We provide thorough descriptions of the datasets in Appendix B.
[Table]
GNN feature extractor.
As the GGM literature frequently uses small datasets, the sample efﬁciency of each metric is extremely important.
Furthermore, as the dimensionality of the graph embedding x is a key factor in many of the metrics in Section 2.3, there is a bias towards minimizing the length of x while retaining discriminability.
As seen in Equation 4, the number of propagation rounds L and the node embedding size d are directly responsible for determining the dimensionality of x .
In addition, You et al.
(2020) demonstrate that the choice of L is one of the most critical for performance across diverse graph tasks.
Thus, in our experiments we consider GIN models (Equations 3 and 4) with L ∈ [2 , 3 , . . .
, 7] , and d ∈ [5 , 10 , . . .
We randomly select 20 architectures inside these ranges to test in our experiments using both randomly initialized and pretrained GINs.
The pretraining process tasks each network with graph classiﬁcation and the methodology is described in Appendix B. Results for metrics computed using a pretrained GIN in individual experiments are left to Appendix C.1.
However, we summarize these results in Table 3 in Section 5 to facilitate discussion.
We use node degree features expressed as an integer as an inexpensive way to improve discriminability in both random and pretrained networks.
In practice, we utilize orthogonal weight initialization (Saxe et al., 2014) in the random networks as it produces metrics with slightly lower variance across initializations.
Evaluating the evaluation metrics.
All experiments are designed to begin with P g ≈ P r , and to have a monotonically increasing degree of perturbation t ∈ [0 , 1] that is a measure of the dissimilarity between S g and S r .
We evaluate each metric objectively by computing the Spearman rank correlation between the metric scores ˆ ρ and the degree of perturbation t .
Spearman rank correlation is preferable to Pearson correlation as it avoids any bias towards a linear relationship.
All metrics are normalized such that ˆ ρ = 0 if P r = P g meaning that ˆ ρ should increase with t for a strong metric and a rank correlation of 1.0 is assumed to be ideal.
We test each metric and GIN architecture combination across 10 random seeds which affects GIN model weights (if applicable) and perturbations applied.
To report the results for a given metric, we ﬁrst compute the rank correlation for a single random seed (which varies model weights and perturbations applied, if applicable), experiment (e.g. edge rewiring), dataset (e.g. Grid) and GIN conﬁguration (if applicable, e.g. L = 4 , d = 25 ).
We then aggregate the rank correlation scores across any combination of these factors of variation.
4. M EASURING FIDELITY
One of the most important properties of a metric is its ability to measure the ﬁdelity of generated samples.
To test metrics for ﬁdelity we construct two experiments.
The ﬁrst experiment tests the metric’s ability to detect various amounts of random samples mixed with real samples (Xu et al., 2018), while the second experiment slowly decreases the quality of graphs by randomly rewiring edges (O’Bray et al., 2022).
Each of these experiments begin with S g as a copy of S r , which is itself a copy of the dataset.
In the ﬁrst experiment, we utilize random graphs to impact the quality of S g .
To decrease the similarity between S r and S g , we slowly mix random graphs by increasing the ratio t of random graphs to real graphs in S g ( t ) .
We simultaneously remove real graphs such that | S g | is constant throughout.
The random graphs are Erd˝os-R´enyi (E-R) graphs (Erd˝os & R´enyi, 1960) with sizes and p values chosen to resemble S r : for every G r ∈ S r , there is a corresponding E-R graph G g with | V | g =
| V | r and p = | E | r | V | 2 r , which is the sparsity of G r .
The second experiment increases distance between P r and P g by randomly rewiring edges in S g .
Here, the degree of perturbation t is the probability of rewiring each edge ( i, j ) ∈ E .
For each
G ∈ S g and each ( i, j ) ∈ E , a sample x i,j ∼ Bernoulli( t ) is drawn.
Edges with x i,j = 1 are rewired, another sample y i,j ∼ Bernoulli(0 .
5) is drawn to decide which node { i, j } of the edge is kept, and a new connection for this edge is chosen uniformly from V .
P r e c i s i
[Figure]
n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F
[Figure]
C F
[Figure]
K
[Figure]
D L i n e a r
[Figure]
R
[Figure]
F
[Figure]
u
[Figure]
t e r i n [Figure]
MM
D
[Figure]
g r ee
[Figure]
r
[Figure]
i t s
[Figure]
Results. With the exception of Recall and Orbits MMD, the majority of tested metrics excel in these experiments as indicated by a rank correlation close to 1.0 (Figure 3, top).
However, Recall is speciﬁcally designed to measure diversity of S g rather than ﬁdelity, and its low sensitivity to ﬁdelity here is expected.
Surprisingly, Coverage demonstrates strong sensitivity to the ﬁdelity of S g although it is also designed to measure the diversity of S g .
In addition, we repeat the mixing experiment with graphs generated by GRAN (Li et al., 2018) and obtain similar results (see Appendix C.4).
4. M EASURING DIVERSITY
The next property we investigate is a metric’s ability to measure the diversity of generated samples in S g .
We focus on two common pitfalls of generative models that a strong metric must be sensitive to: mode dropping and mode collapse.
We test each metric for both sensitivities independently by adapting two experiments from Xu et al.
(2018) to the graph domain, both of which begin by clustering the dataset using Afﬁnity Propagation (Frey & Dueck, 2007) to identify the modes of P r .
Both of these experiments begin with S r and S g as disjoint halves of the dataset.
To simulate mode collapse , we progressively replace each datapoint with its cluster centre.
The degree of perturbation t represents the ratio of clusters that have been collapsed in this manner.
To simulate mode dropping , we progressively remove clusters from S g .
To keep | S g | constant, we randomly select samples from the remaining clusters to duplicate.
In this experiment, the degree of perturbation t is the ratio of clusters that have been deleted from S g . Results.
In the mode collapse experiment, all classical metrics (You et al., 2018) perform poorly with a rank correlation less than 0.5 (Figure 3, bottom).
Classical metrics obtain slightly better, though still suboptimal results in the mode dropping experiment.
As expected, Recall and Coverage exhibit strong positive correlation with the diversity of S g , while Precision and Density are negatively correlated.
In addition, several scalar metrics such as MMD RBF and F1 PR exhibit strong correlation with the diversity of S g and outperform classical metrics across both experiments.
4. S ENSITIVITY TO NODE AND EDGE FEATURES
In this experiment, we measure the sensitivity of each metric to changes in node or edge feature distributions while the underlying graph structure is static.
Similar to the rewiring edges experiment, this is performed by randomizing features with a monotonically increasing probability t , and we provide more thorough descriptions of these experiments in Appendix B.
We exclude metrics from You et al.
(2018) in these experiments as they are unable to incorporate both edge and node features in evaluation.
We ﬁnd all NN-based metrics and NSPDK MMD are sensitive to these perturbations, and a summary is provided in Table 3.
4. S AMPLE EFFICIENCY
Small datasets are frequently used in the GGM literature so the notion of sample efﬁciency is important.
In this experiment, we determine the sample efﬁciency of each metric by ﬁnding the minimum number of samples to discriminate a set of random graphs S g from real samples S r .
The random graphs are E-R graphs generated using the same process described in Section 4.1.
We begin this experiment by sampling two disjoint sets S (cid:48) r and S (cid:48)(cid:48) r from S r , and a set of random graphs S (cid:48)
g from
S g with | S (cid:48) r | = | S (cid:48)(cid:48)
r | = | S (cid:48) g | = n and small n .
A metric with high sample efﬁciency should measure ˆ ρ ( S (cid:48) r , S (cid:48)(cid:48) r ) <
ˆ ρ ( S (cid:48) r , S (cid:48) g ) with a small n .
Rather than using rank correlation, in this experiment we record the sample efﬁciency of each metric as the smallest n where ˆ ρ ( S (cid:48) r , S (cid:48)(cid:48) r ) <
ˆ ρ ( S (cid:48) r , S (cid:48) g ) ∀
i ≥ n .
All of the metrics based on K -nearest neighbours and many of the classical metrics exhibit high sample efﬁciency and require a minimal number of samples to correctly score S (cid:48)(cid:48) r and S (cid:48) g
(Table 3).
4. C OMPUTATIONAL EFFICIENCY
Computational efﬁciency is the ﬁnal property of an evaluation metric that we examine.
Metrics that are efﬁcient to compute are ideal as they can be easily used throughout training to measure progress and perform model selection.
Graph datasets can scale in several dimensions: the number of samples, average number of nodes, and average number of edges.
We make use of E-R graphs to generate graphs with an arbitrary number of nodes and edges enabling us to independently scale graphs in each dimension.
The results highlighting each metric’s computational efﬁciency as dataset size increases to 10,000 samples is shown in Table 3, while the results for all dimensions are presented in Figure 10 in Appendix C.
The classical metrics (You et al., 2018) quickly become prohibitive to compute as the number of samples increase, while the NN-based metrics are faster by several orders of magnitude and are inexpensive to compute at any scale.
4. GGM SELECTION
While NN-based metrics such as MMD RBF and F1 PR have performed consistently in our experiments, this does not necessarily mean they are suited for evaluating GGMs.
With the lack of an
Inception v3 (Szegedy et al., 2016) analogue for GGM evaluation, these metrics must be consistent across model parameterizations.
Thus, in this section we evaluate two popular generative models,
GRAN (Liao et al., 2019) and GraphRNN (You et al., 2018) on the Grid dataset.
To measure the variance induced by changing model parameterizations, we compute ˆ ρ ( S g , S r ) across 10 different random GINs.
We use the strongest GIN conﬁguration along with a strong NN-based metric, MMD
RBF, both of which we identify in Section 5.
We simulate the model selection process by evaluating models at various stages of training and ﬁnd that metrics from You et al.
(2018) are unable to unanimously rank GGMs (Table 2).
As GGMs are frequently evaluated by considering all of these metrics together, this highlights the difﬁculty that may arise during model selection with this evaluation process.
Furthermore, we ﬁnd that MMD RBF is extremely low variance across random GINs indicating its promise for evaluating GGMs.
Additional strong NN-based metrics such as F1 PR and F1 DC are tested in Table 12 in Appendix C.8 and are also found to be low variance.
Note that we also compute all metrics using a 50/50 split of the dataset.
This provides information regarding what score two sets of indistinguishable graphs may receive and represents an ideal value for the given dataset.
We suggest that future work also follows this process as it provides a sense of scale and improves interpretability of the results.
D ISCUSSION
We aggregate the results across experiments for a high-level overview of each metric’s strengths and weaknesses in Table 3.
Metrics computed using a pretrained GIN are nearly indistinguishable from those using a random GIN across rank correlation experiments (columns 3–4).
Although metrics using a pretrained GIN beneﬁt from slightly higher sample efﬁciency, they can have higher variance across model parameters than a simple random initialization (Table 12 in Appendix C.8).
Similar to
[Table] [Table]
Table 3, we aggregate the results across all experiments and metrics for a speciﬁc GIN conﬁguration in Table 4b in Appendix C.
We ﬁnd that the mean rank correlation taken across NN-based metrics and all experiments is approximately 10× higher variance than the mean taken across GIN conﬁgurations (0.078 and 0.007, respectively).
In general, our ﬁndings indicate that the key to strong
GIN-based GGM evaluation metrics is the choice of metric rather than the speciﬁc GNN used to extract graph embeddings .
[Table]
In practice, to measure the distance between two sets of graphs we recommend the use of either the MMD RBF or F1 PR metrics.
Although we found the choice of GIN architecture to be unimportant, we suggest the use of a random GIN with 3 rounds of graph propagation and a node embedding size of 35 for consistency in future work.
This is the strongest GIN conﬁguration across all experiments (Table 4b in Appendix C.2).
Both metrics have been shown to be sensitive to changes in both ﬁdelity and diversity while having minimal variance across random initializations.
In addition, they are capable of detecting changes in node and edge feature distributions, making them useful for a wide array of datasets.
3 To highlight this, we provide results for individual datasets in Appendix B.1 and ﬁnd the performance of GNN-based metrics to be consistent across all datasets.
While MMD RBF has slightly stronger correlation with the ﬁdelity and diversity of generated graphs, F1 PR has superior sample and computational efﬁciency.
Thus, we suggest the use of F1 PR in cases where there are fewer samples than the measured sample size of MMD RBF (i.e., 42) or MMD RBF is extremely prohibitive to compute, otherwise, we suggest MMD RBF.
C ONCLUSION
We introduced the use of an untrained random GIN in GGM evaluation, inspired by metrics popular in the image domain.
We discovered that pre-existing GGM metrics fail to capture the diversity of generated graphs, and ﬁnd several random GIN-based metrics that are more expressive while being domain-agnostic at a signiﬁcantly reduced computational cost.
An interesting direction for future work is to provide theoretical justiﬁcation supporting the use of random GINs, as well as the exploration of more sophisticated GNNs to improve sample efﬁciency. A CKNOWLEDGMENTS
RT and BK received support from NSERC.
RT received funding from the Vector Scholarship in
Artiﬁcial Intelligence. BK received support from the Ontario Graduate Scholarship. GWT acknowledges support from CIFAR and the Canada Foundation for Innovation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through
CIFAR, and companies sponsoring the Vector Institute: http://www.vectorinstitute.
ai/#partners .
Appendices A Additional metric details
B Additional experimental details
B.1 Datasets
B.2 Measuring diversity
B.3 Randomizing edge features . . .
B.4 Randomizing node features . . .
B.5 Computational efﬁciency . B.
[Table]
B. GNN pretraining
C Additional results
C.1 Individual experiments . .
C.2 All experiments
grouped by GIN conﬁguration
C.3 All results
grouped by dataset
C.4 Mixing generated graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.5 Computational efﬁciency . .
C.6 Comparing other GNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.7 Comparing σ selection strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . C.8 GGM selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.
Stability of
GGM
rankings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A A DDITIONAL METRIC DETAILS
For Precision, Recall, Density, and Coverage, we set k = 5 for all experiments (Naeem et al., 2020).
For the MMD RBF metric, we compute MMD as MMD( S g , S r ) =
max { MMD( S g , S r ; σ ) | σ ∈ Σ } , where the values of Σ are multiplied by the mean pairwise distance of S g and S r .
While the median pairwise distance is the heuristic (Garreau et al., 2018), we ﬁnd the mean to be more robust in our experiments.
Before this scaling factor is applied, we use Σ = { 0 . 01 , 0 . 1 , 0 . 25 , 0 . 5 , 0 .
For all baseline metrics from You et al.
(2018), we use the hyperparameters chosen in their open-source code.
For the NSPDK MMD metric we use the open-source code from Goyal et al.
B A DDITIONAL EXPERIMENTAL DETAILS
Following Section 2.2, we denote a graph as G =
( V , E ) with vertices V and edges E
= { ( i, j ) | i, j ∈ { 1 , . . , | V |}} .
Node features for node i ∈ V are denoted as h i ∈
R b ( h (0) i in Section 2.2).
Similarly, A ∈ R n × n × a stores edge information, with the ( i, j ) th entry corresponding to the edge features for edge ( i, j ) .
Unless otherwise speciﬁed in Appendix B.1, A is not utilized and we set h i to the degree of node i expressed as an integer as an inexpensive way to improve discriminability.
We handle edge features by concatenating A i,j, : to messages from node i to node j at each round of graph propagation (Hu et al., 2020).
B. D ATASETS
We perform experiments using a variety of datasets with varying sizes and characteristics to thoroughly test each metric’s ability to evaluate GGMs across domains.
We place a slight emphasis on datasets that are frequently used in the GGM literature.
Lobster.
A set of 100 stochastic graphs where each node is at most 2 hops away from a backbone path.
We generate lobster graphs with 10 ≤ | V | ≤ 100 (Dai et al., 2020).
Grid.
100 2D grid graphs generated with 100 ≤ | V | ≤ 400 (Dai et al., 2020; You et al.
Liao et al.,
Proteins.
918 protein graphs where nodes are amino acids and two nodes are connected by an edge if they are less than 6 Angstroms away (Dobson & Doig, 2003).
We select graphs with 100 ≤ | V | ≤ 500 (You et al., 2018;
Liao et al., 2019; Dai et al., 2020).
Ego.
757 3-hop ego networks with 50 ≤ | V | ≤ 399 (You et al., 2018).
Graphs are extracted from the
CiteSeer network where nodes represent documents and edges represent citation relationships (Sen et al., 2008).
Community.
500 two-community graphs with 60 ≤ | V | ≤ 160 .
Each community is generated by the Erd˝os-R´enyi model (E-R) (Erd˝os & R´enyi, 1960) with n = | V | / 2 and p = 0 .
We then add 0 .
| V | inter-community edges with uniform probability (You et al., 2018).
ZINC.
250k real-world molecular graphs (Irwin et al., 2012) of which we randomly selectsamples for efﬁciency with 10 ≤ | V | ≤ 50 .
We set h i to be a one-hot encoding of the element of node i , and A i,j, : to be a one-hot encoding a ∈ R a of the bond type between nodes i and j .
This dataset is unique in that it is the only one to naturally contain node and edge features.
Thus, we can use this dataset to determine a metric’s sensitivity to changes in the edge and node feature distributions without generating artiﬁcial labels.
B. M EASURING DIVERSITY
We employ Afﬁnity Propagation (Frey & Dueck, 2007) to automatically determine the number of clusters, and set the similarity between graphs to be the value obtained from the WL-subtree kernel (Shervashidze et al., 2011) using the GraKel Python library (Siglidis et al., 2020).
This clustering method avoids any dependency on the GNN model architecture and ensures the clusters found are consistent across all experiments.
B. R ANDOMIZING EDGE FEATURES
This experiment is performed exclusively on the ZINC dataset as it naturally contains edge features.
Here, the degree of perturbation t is the probability of randomizing each edge feature A
i,j, : , ∀ ( i, j ) ∈
E .
For each G ∈ S g and each edge ( i, j ) ∈ E , a sample x i,j ∼ Bernoulli( t ) is drawn.
Edges with x i,j = 1 have their edge feature randomized.
As these features are a one-hot encoding, we sample the new edge feature uniformly from the set of valid edge features.
The results of this experiment are shown in Figure 9.
B. R ANDOMIZING NODE FEATURES
This experiment is performed exclusively on the ZINC dataset as it naturally contains node features.
Here, the degree of perturbation t is the probability of randomizing each node feature h
i , ∀ i ∈ V .
For each G ∈ S g and each node i ∈ V , a sample x i ∼ Bernoulli( t ) is drawn.
Nodes with x i =have their feature randomized.
As these features are a one-hot encoding, we sample the new node feature uniformly from the set of valid node features.
The results of this experiment are shown in
Figure 9.
B. C OMPUTATIONAL EFFICIENCY
All experiments in this section were conducted on an Intel Platinum 8160F Skylake @
2.1Ghz with 4 CPU cores.
We benchmark NN-based metrics using the most expensive GIN conﬁguration tested in our experiments.
This was a conﬁguration with 7 propagation layers and a node embedding size of 20.
The results for this experiment on CPU are shown in Figure 10 and on GPU in Figure 11.
In addition, the approximate RAM usage for each metric throughout these experiments are shown in
Figure 12 and Table 9.
Scaling number of samples.
We generate various sets of E-R graphs (Erd˝os & R´enyi, 1960) to scale the number of samples in a dataset independently from the number of nodes and number of edges per graph.
We generate sets of graphs with the number of samples in [100 , 1000 , 2000 , . .
, 10 k ] with | V | = 50 for all graphs.
We choose p = | E | avg | V | 2 avg for all graphs, where the averages are taken across the Proteins dataset.
Scaling number of edges.
To scale the number of edges independently, we generate 50 E-R graphs per set with | V | = 1000 .
The graphs are generated with p in [0 . 01 , 0 . 1 , 0 . 2 , . . .
Scaling number of nodes.
We generate 50 graphs of a constant size per set with | V | in [1000 , 10 k , 20 k , . .
, 100 k ] .
For each set, we use p = 10000 | V | 2 to generate graphs with approximately 10000 edges per graph.
B. GGM SELECTION
We train GRAN (Liao et al., 2019) and GraphRNN (You et al., 2018) with 80% of the graphs randomly selected for training.
We use the implementations in the ofﬁcial GitHub repositories and train using the recommended hyperparameters.
We then generate n graphs from each model where n is the size of the dataset, and use the remaining 20% of graphs as S r .
B. GNN PRETRAINING
We isolate the comparison between randomly initialized and pretrained weights to datasets without node or edge features (i.e. excluding ZINC).
The GNN is trained under the multiclass classiﬁcation setting to classify a given graph into the remaining datasets.
To increase the difﬁculty of the problem, for each graph G r in the remaining datasets, we generate an E-R graph (Erd˝os & R´enyi, 1960)
G g with | V | g = | V | r and p = | E | r | V | 2 r .
These E-R graphs have separate labels, thereby doubling the number of classes present.
We use the same optimizer across all experiments and select the model with the lowest validation loss.
We train each conﬁguration across 10 random seeds to perform a fair comparison with the 10 different random initializations used in our experiments.
P r e c
i s i o n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F 1 D C F D K D
MM D L i n e a r
MM D R B F 1.
[Figure]
[Figure] [Figure]
[Figure]
[Figure] [Figure]
R a n k c o rr e l a t i o n
[Figure]
P r e c
i s i o n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F 1 D C F D K D
MM D L i n e a r
MM D R B F
Pretrained weights
C l u s t e r i n g MM
D
D e g r ee
MM D
O r b i t s
MM D
N S P D K
MM D
Baseline metrics
[Figure]
Baseline (
[Figure]
metrics
[Figure]
C A DDITIONAL RESULTS
Here we provide a variety of supplementary experiments that were not included in the main body due to page limits.
Unless otherwise speciﬁed, results for all GIN-based metrics utilize a random
GIN, are aggregated across all GIN conﬁgurations, and all GIN conﬁgurations use summation aggregation.
C. I NDIVIDUAL EXPERIMENTS
We provide violin plots for each individual experiment in Figures 4 through 8.
C. A LL EXPERIMENTS GROUPED BY GIN CONFIGURATION
We aggregate results by GIN conﬁguration across all experiments in Tables 4.
In addition, we aggregate results by GIN conﬁguration and metric combination and highlight the top 20 results in
Table 5.
C. A LL RESULTS GROUPED BY DATASET
For a more in-depth comparison of metrics, here we provide results for all experiments grouped by the dataset used. The results are shown in Tables 6 and 7. We ﬁnd that while the performance of each metric does ﬂuctuate slightly across datasets, all of our conclusions drawn in Section 5 still hold regardless of dataset and there are no obvious outliers. F1 PR and MMD RBF are consistently the highest scoring metrics, and with the exception of the Community dataset all baseline metrics (You et al., 2018) are poor at measuring the diversity of generated graphs.
Notably, the performance of
Clustering MMD on diversity experiments is affected by the Lobster and Grid datasets; these graphs are triangle-free and thus all nodes have a clustering coefﬁcient of zero.
However, even if these datasets are excluded from the analysis, Clustering MMD is still suboptimal on the remainder of the datasets.
P r e c
i s i o n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F
[Figure]
F [Figure] K D MM
[Figure]
L i n e a r
MM
[Figure] F [Figure] i
[Figure]
n
[Figure]
n
[Figure]
i t y
[Figure]
c a
[Figure]
C [Figure] [Figure] n k c o rr e l a t i o n
[Figure]
P r e c
i s i o n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F 1 D C F D K D
MM D L i n e a r
MM D R B F
[Figure]
C l u s t e r i n g MM
D
D e g r ee
M M D
O r b i t s M M D N S P D K MM D
[Figure]
P r e c i s i o n D e n s i t y R e c a ll C o v e r a g e
F 1 P R F 1 D C F D K D MM D L i n e a r MM [Figure]
R
[Figure]
F
[Figure]
P r e c
i s i o n
[Figure]
e n s i t y R e c a ll C [Figure] e
[Figure]
i
[Figure]
s
[Figure]
r
[Figure]
ee
[Figure]
i
[Figure]
[Figure]
S
[Figure]
m p l e s i z e (
[Figure]
P r e c
i s i o n
D e n s i t y R e c a ll
C o v e r a g e
F 1 P R F 1 D C F D K D
MM D L i n e a r
MM D R B F
Pretrained weights
C l u s t e r i n g MM
D
D e g r ee
MM D
O r b i t s
MM D
N S P D K
MM D
Baseline metrics
[Table]
[Table]
0 .7 ±
[Table]
[Table]
[Table]
[Table]
[Table]
P r e c
i s i o n D e n s i
t y R e c a ll C o v e r a g e F 1 P R F 1 D C F D K D MM D L i n e a r
MM D R B F
N S P D K
MM D 0.
R a n k c o rr e l a t i o n
Randomize edges
P r e c
i s i o n D
e
n s i
t y R e c a ll
C o v e r a g e
F 1 P R F 1 D C F D K D
MM D L i n e a r MM D R B F N S P D K MM D 0.
R a n k c o rr e l a t i o n
Randomize nodes
[Figure]
g
[Figure]
Precision
[Figure]
[Figure]
[Table] [Figure] [Table] C. M IXING GENERATED GRAPHS
As opposed to the E-R graphs used in Section 4. [Table] here we perform the mixing experiment using graphs generated by GRAN (Liao et al., 2019) for each dataset.
As these graphs are a better representation of the dataset of interest [Table] this represents a much harder problem.
However, as seen in
Table [Table] the results across metrics are fairly consistent regardless of the type of graph used.
C. C OMPUTATIONAL EFFICIENCY
In Figure 12 we plot each metric’s RAM usage throughout the computational efﬁciency experiments in Section 4.5 by tracking the memory usage of the main Python process.
While this is likely imperfect and slightly noisy, it provides an idea of the memory requirements of each metric.
Unfortunately all classiﬁer-based metrics are grouped into the same experiment and more in-depth breakdowns are unavailable for this experiment.
However, it is likely that the bulk of the memory requirements spe-
[Figure] [Figure] [Figure] ciﬁc to these metrics come from storing the GIN parameters and extracted graph embeddings.
The maximum recorded value for each metric in each experiment is recorded in Table [Figure]
C. C OMPARING OTHER GNN S
In the main article [Figure] all experiments were performed using GIN with summation neighborhood aggregation and concatenating graph embeddings at each round of graph propagation into a ﬁnal graph embedding (Equation 4).
Here we also provide results for random GINs with mean and max [Table] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure]
neighborhood aggregation.
With minor architecture differences, the former is equivalent to random GCN (Kipf & Welling, 2017), while the latter is equivalent to random GraphSAGE (Hamilton et al., 2018).
In addition, we also provide a comparison to random GIN with summation neighborhood aggregation that does not use the concatenation in Equation 4, i.e. the resultant graph embedding is simply the embedding obtained at graph propagation round L .
We ﬁnd that while GIN is superior across the presented metrics, it is in many cases only by a very narrow margin (Table 10).
The performance of MMD RBF appears to be relatively constant across GNNs, while the performance of F1 PR has higher variance.
C. C OMPARING σ SELECTION STRATEGIES
In order to provide a fairer comparison to pre-existing metrics based on the RBF kernel (You et al., 2018), here we include results for MMD RBF with a static σ = 1 using a random GIN.
The results are shown in Table 11.
While this metric does result in a decrease in performance while measuring diversity, it is still more expressive than baseline metrics.
Note that we could also utilize our σ selection process with pre-existing metrics.
However, we did not experiment with this as these metrics would still suffer from computational efﬁciency issues and be unable to incorporate node and edge features. [Figure] [Table]
C. GGM SELECTION
Here we evaluate GGMs at various stages of training on Grid, Lobster and Proteins datasets using classical metrics (You et al., 2018), NN-based metrics using a random GIN, and NN-based metrics using a pretrained GIN.
The results are shown in Table 12. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
(c) Using the F1 PR metric.
Part A.
[Table]
(d) Using the F1 PR metric.
Part B.
[Table]
[Table]
C. S TABILITY OF GGM RANKINGS
Here, we investigate the stability of GGM rankings across GIN architectures.
For the Grid, Lobster, and Proteins datasets, we again evaluate GGMs at various stages of training.
We evaluate models using a random GIN with each of the 20 architectures we tested in the main-body using both MMD
RBF and F1 PR, and the results are shown in Tables 13 through 15.
We ﬁnd that the rankings are extremely consistent across architectures for the Grid and Proteins datasets.
However, this does not appear to be the case for the Lobster dataset.
One potential explanation is that for the Lobster dataset, there is added difﬁculty from both small extremely sample sizes and poor generative models.
[Table]
165 GRAN-66% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
Using the F1 PR metric.
Part A.
50/50 split 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
995 GraphRNN-100% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 . GraphRNN-66% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 . 792 GRAN-100% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 . 781 GRAN
[Table]
Using
50/50 split 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
99 GraphRNN-100% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
768 GraphRNN
[Table]
[Table]
064 GRAN-66% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
50/50 split 0 .0 .0 .0 .0 .0 .0 .0 .0 .1 . GraphRNN-100% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 . GraphRNN-66% 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 . 969 GRAN-100% 1 .1 .1 .1 .1 .1 .0 .1 .0 .1 .
GRAN-66% 0 .0 .1 .1 .1 .1 .0 .1 .0 .1 .
Using
50/50 split 0 .0 .1 .0 .0 .0 .0 .0 .1 .1 .
GraphRNN-100% 0 .0 .0 .0 .0 .0 .
[Table]