AgraSSt: Approximate Graph Stein Statistics for Interpretable Assessment of Implicit Graph Generators
s)
AffiliationAddress email
Abstract
We propose and analyse a novel statistical procedure, coined AgraSSt , to assessthe quality of graph generators which may not be available in explicit forms.
Inparticular, AgraSSt can be used to determine whether a learned graph generating process is capable of generating graphs which resemble a given input graph.
Inspired by Stein operators for random graphs, the key idea of AgraSSt is theconstruction of a kernel discrepancy based on an operator obtained from the graphgenerator.
AgraSSt can provide interpretable criticisms for a graph generator training procedure and help identify reliable sample batches for downstream tasks.
Wegive theoretical guarantees for a broad class of random graph models.
We provideempirical results on both synthetic input graphs with known graph generationprocedures, and real-world input graphs that the state-of-the-art (deep) generativemodels for graphs are trained on.
Introduction
Generative models for graphs have received increasing attention in the statistics and machine learningcommunities.
Recently, deep neural networks have been utilised to learn rich representations fromgraph structures and generate graphs [Dai et al., 2020, Li et al., 2018, Liao et al., 2019, You et al.
However, due to the often opaque deep learning procedures, these deep generative models areusually implicit, which hinders theoretical analysis to assess how close the generated samples are intheir distributional properties to the graph distribution they are meant to be sampled from.
Learning parametric models requires an explicit pre-specified probability distribution class, and thelearned parameters can be used for model assessment.
However, parametric models may only capturea fraction of the graph features and have restricted modelling power.
Due to dependency betweenedges, parameter estimation can be inconsistent [Shalizi and Rinaldo, 2013] even for well-specifiedmodels, and may lead to wrong conclusions in model assessments.
Although recent advances indeep generative models for graphs may surpass some of the above-mentioned issues by learning richgraph representations, methods to assess the quality of such implicit graph generators are lacking.
In principle, nonparametric hypothesis tests can be useful to assess complex models, such as by thepopular kernel-based tests procedures that utilise functions in a reproducing kernel Hilbert space(RKHS)
[Berlinet and Thomas, 2004].
When the models are described in the form of explicitprobabilities, goodness-of-fit tests [Chwialkowski et al., 2016, Liu et al., 2016] may apply; however,goodness-of-fit testing procedures are generally not applicable for implicit models.
Instead, if a largeset of samples from the target distribution is observed, one may generate samples from the implicit
[Figure]
model and perform a two-sample test such as a maximum mean discrepancy (MMD) test [Grettonet al., 2007] for model assessment [Jitkrittum et al., 2017, Xu and Matsuda, 2020, 2021].
However, in real-world applications, often only a single graph from the target distribution is observed[Bresler and Nagaraj, 2018, Reinert and Ross, 2019].
In this case, the MMD methods for modelassessment via two-sample testing procedures cannot be used to assess implicit graph generators.
To the best of our knowledge, beyond Monte Carlo tests based on a simple test statistic with oftenpoor power, no principled test is available for assessing the quality of implicit graph generators. In
Figure.1, the task is illustrated: we are given a graph generator G which is learned from a set oftraining samples and can generate samples of user-defined size.
The task is to assess whether G cangenerate samples from the same distribution that generates the observed graph, for example, Padgett’s
Florentine marriage network [Padgett and Ansell, 1993].
In Figure.1, G is a Cross-Entropy Low-rank
Logit (CELL) model [Rendsburg et al., 2020] trained on Padgett’s Florentine marriage network, butany graph generator could be used.
This paper makes three main contributions.
(1) We introduce Approximate graph Stein Statistics(AgraSSt) in Section 3, which opens up a principled way to understand implicit graph generators.
AgraSSt is a variant of a kernel Stein discrepancy based on an empirical Stein operator for conditionaldistributions of general random graph models.
The testing procedure is inspired by gKSS, a goodnessof-fit testing procedure for explicit exponential random graph models (ERGMs)
[Xu and Reinert,2021].
(2) We provide theoretical guarantees for AgraSSt (Section 3).
(3) We propose interpretablemodel criticisms when there is a model misfit and identify representative synthetic sample batcheswhen the model is a good fit (Section 4).
Further, in Section 2 we review gKSS.
We provide empirical results in Section 5 and a discussion in Section 6.
Proof details, more background, theoretical and empirical results, and implementationdetails are found in the Supplementary Information (SI) text.
Code is attached as a folder in the SI.
Background: graph kernel Stein statistics
In [Xu and Reinert, 2021] a goodness-of-fit testing procedure called gKSS for exponential randomgraph models is introduced, which assumes that only a single network may be observed in the sample.
As AgraSSt is inspired by gKSS, and gKSS serves as comparison method when the observed networkis known to be generated from an exponential random graph model (ERGM), we briefly review gKSS.
The notation introduced in this section is used throughout the paper.
The class of ERGMs is a rich model class which includes Bernoulli random graphs and which isextensively used for social network analysis [Wasserman and Faust, 1994, Holland and Leinhardt,1981, Frank and Strauss, 1986].
ERGMs model random graphs via a Gibbs measure with respectto a chosen set of network statistics, such as number of edges, 2-stars, and triangles.
Denote by G lab n , the set of vertex-labeled graphs on n vertices, with N = n ( n − 1) / 2 possible undirected edges.
Encode x ∈ G lab n by an ordered collection of { 0 , 1 } -valued variables x = ( x ( ij ) ) 1 ≤ i<j ≤ n ∈ { 0 , 1 } Nwhere x ( ij ) = {+
+}
if and only if there is an edge between i and j .
We denote an (ordered) vertex-pairindex s =
( i, j ) by s ∈ [ N ] := { 1 , . . .
, N } .
For a collection of graphs H 1 , . . . , H k on at most nvertices, with H 1 denoting a single edge, let t ℓ counts β
= ( β 1 , . . . , β k )
⊤ ∈ R k and t ( x ) =
( t 1 ( x ) , . . .
, t k ( x ))
⊤ ∈
R
k
we say that X ∈ G labn follows theexponential random graph model X ∼ ERGM( β, t ) if for ∀ x ∈ G labn ,
[Equation]
Here κ n ( β ) is a normalisation constant.
In this model, t ℓ ( x ) , ℓ = 1 , . . .
, k , are sufficient statistics.
Parameter estimation ˆ β l for β l is only possible when the graphs H 2 , . . .
, H k are specified a priori;see SI.B.1 for estimation details.
In modern graph learning procedures, e.g. deep generative learning,the sufficient statistics t l of Eq.(1) may not be obtained explicitly.
In [Reinert and Ross, 2019] the exponential random graph distribution in Eq.
1 is characterisedby a so-called Stein operator , as follows.
Let e s ∈ { 0 , 1 } N be a vector with 1 in coordinate sand
in all others
; x ( s, 1) = x + (1 − x s ) e s has the s -entry replaced of x by the value 1, andx ( s, 0) = x − x s e s has the s -entry of x replaced by the value 0; moreover, x − s is the set of edgeindicators with entry s removed.
For a function h : { 0 , 1 } N → R , let ∆ s h ( x ) =
h ( x ( s, 1) ) − h
( x ( s, 0) ) .
Set q X ( x ( s, 1) | x − s ) =
P ( X s = 1 | X − s = x − s ) .
Define the operator
[Equation]
Then under mild conditions Reinert and Ross [2019] show that if E p [ A β,t f ] = 0 for all smooth test functions f , then p must be the distribution of ERGM ( β, t ) .
Thus, this operator characterises
ERGM ( β, t ) .
For the derivation of AgraSSt it is of interest to see how this operator is obtained.
It is indeed the generator of a so-called Glauber Markov chain on G labn with transition probabilities
[Equation]
With the ERGM Stein operator in Eq.(2) and a rich-enough RKHS test function class H , Xu and
Reinert [2021] propose a graph kernel Stein statistics (gKSS) to perform goodness-of-fit testing on anexplicit ERGM when a single network sample is observed.
With the summand components in Eq.(2),the Stein operator can be seen as taking expectation over vertex-pair variables S ∈ [ N ] with uniformprobability P ( S = s ) ≡ N − 1 independently of x , namely
[Equation]
For a fixed graph x , gKSS is defined as
[Equation]
where the function f is chosen to best distinguish q from x .
For an RKHS H associated withkernel K , by the reproducing property of H , the squared version of gKSS admits a quadraticform representation gKSS 2 ( q ; x ) =
(cid:68) E S
[ A ( S ) q K
( x, · )] , E S
[ A ( S ) q K
( x, · )] (cid:69) , which can be computedreadily.
More background can be found in Appendix B.2.
AgraSSt: Approximate Graph Stein Statistic
An implicit graph generator may not admit a probability distribution in the form of Eq.(1); however,the idea of constructing Stein operators based on Glauber dynamics using conditional probability distribution for ERGM is inspiring.
Here we propose Stein operators for conditional graph distributions,to facilitate an (approximate) characterisation for implicit random graph models.
3.1 Stein operators for conditional graph distributions
Let q ( x ) =
P ( X = x ) be any distribution with support G labn .
Let t ( x ) denote a statistic on graphswhich takes on finitely many values k and let q k ( x )
=
P ( X = x
| t ( x ) = k ) .
We assume that q k ( x ) > 0 for all k under consideration.
For a generic outcome we write q t .
We introduce a Markovchain on G labn which transitions from x to x s, 1 with probability
[Equation]
and which transitions from x to x s, 0 with probability q ( x s, 0 | t ( x s ))
= 1 − q t ( x s, 1 ); no other transitionsoccur.
Let
[Equation]
For an ERGM, t ( x ) could be taken as a sufficient vector of statistics 1 , but here we do not assume aparametric network model q ( x ) , and t ( x ) does not have to be sufficient statistics for q ( x ) .
Recall that an operator is a Stein operator for a distribution µ if its expectation under µ is zero.
Thefollowing result provides a theoretical foundation for AgraSSt and is proven in SI.A.
Lemma 3.1.
In this setting, A ( s ) q,t = k is a Stein operator for the conditional distribution of X givent ( X ) =
k , and (cid:80) s A ( s ) q,t
= k is a Stein operator for the conditional distribution of X given t ( X ) = k .
In particular, E q t = k (cid:104) A ( s ) q,t = k (cid:105) = 0 .
Intuitively, if ˜ X has distribution which is close to that of X , thenwith ˜ Y t
=
k denoting the corresponding random graph with distribution that of ˜ X given t ( ˜ X ) =
k,it should hold that E (cid:104) A ( s ) q,t =
k ( ˜ Y t = k ) (cid:105)
≈ 0 .
In this way the Stein operator in Eq.(5) can be used toassess similarity between distributions.
3.2 Approximate Stein operators
For implicit models and graph generators G , the Stein operator A ( s ) q,t in Eq.(5) cannot be obtainedwithout explicit knowledge of q t ( x s, 1 ) .
However, given a large number for samples from the graphgenerator G , the conditional edge probabilities q t ( x s, 1 ) can be estimated.
Here we denote by (cid:98) q t ( x s, 1 )an estimate of q t ( x s, 1 ) ; some estimators will be suggested in Section 3.3.
AgraSSt performs model assessment using an operator which approximates the Stein operator A ( s ) q,t .
We define the approximate Stein operator for the conditional random graph by
[Equation]
The vertex-pair averaged approximate Stein operator is
[Equation]
3.3 Estimation with chosen statistics on graphs
Using the Stein operator for conditional graph distributions, we can obtain the approximate Steinoperators Eq.(6) and (7) for an implicit graph generator G by estimating q t ( x s, 1 ) .
Here t ( x ) areuser-defined statistics.
In principle, any multivariate statistic t ( x ) can be used in this formalism.
However, estimating the conditional probabilities using relative frequencies can be computationallyprohibitive when the graphs are very large and specific frequencies are rarely observed.
Instead,here we consider simple summary statistics, such as edge density, degree statistics or the number ofneighbours connected to both vertices of s .
The estimation procedure is presented in Algorithm 1.
To estimate q t ( x s, 1 ) in Step 3 of Algorithm 1, if the underlying graph has exchangeable edgeindicators then q t ( x s, 1 ) does not depend on the choice of vertex-pair s , an intuitive way is to use alookup table . If t ( x ) is a possibly multivariate statistic with a discrete number of outcomes, countn ( k, s ) , the number of times that vertex-pair s is present in the simulated graphs and t ( x − s )
=
k ;set n ( k ) =
(cid:80) s n ( k, s ) , and
N k = (cid:80) Li =1 (cid:80) s 1 ( t ( x − s )
= k ) ; if t = k then estimate (cid:98) q t ( x s, 1 ) by
[Equation]
If k min , k max denote the minimum and maximum values of statistics from simulated graph samples,then for k outside this set, the lookup table estimator Eq.(8) is set to estimate ˆ g ( k ) = 0 .
If the underlying graph cannot be assumed to have exchangeable edge indicators or if the statistic t ishigh dimensional, then any particular n ( k, s ) may not be observed very often.
In such a situation wecan learn ˆ g t ( s ; k ) using kernel ridge regression so that the conditional probabilities for similar t ( x − s )are predicted in a smooth manner.
Estimating q
x s | t ( x − s ) ≤ k ) instead of q ( x s | t ( x − s )
= k ) mayprovide an alternative, smoother estimate for the conditional probabilities.
The next result shows that the approximate Stein operator achieves the Stein identity asymptotically.
For this result, which is proved in SI.A, we use the notation ∥
∆ f ∥ = sup s ∈ [ N ] ,x ∥ ∆ s f ( x ) ∥ .
Theorem 3.2.
Assume (cid:98) q t ( x s, 1 ) is a consistent estimator for q t ( x s, 1 ) as L → ∞ .
Then for any fsuch that || ∆ f || < ∞ we have E q [ A (cid:98) q,t f ( x )]
→ E q [ A q,t f ( x )] = 0 as L → ∞ .
Section A.2 in SI.A provides refined results for ERGMs, including a Gaussian approximation.
3. AgraSSt for implicit graph generators
The estimated conditional probabilities give an approximate Stein operator for Eq.(5).
With the appropriately defined Stein operator from an implicit model given in Eq.(6), we can define AgraSSt, a kernelbased statistic analogous to gKSS in Eq.(3), as AgraSSt( (cid:98)
[Equation]
In SI.A we prove the following result.
Theorem 3.3.
If the graph is edge-exchangeable, then AgraSSt 2 ( (cid:98) q, t ; x ) is a consistent estimator of
[Equation]
Re-sampling Stein statistic A computationally efficient operator for large N can be derived via resampling vertex-pairs s , which creates a randomised operator.
Let B be the fixed size to be re-sampled. The re-sampled operator is (cid:98) A B (cid:98) q,t f ( x ) = 1 B (cid:80) b ∈ [ B ] A ( s b ) (cid:98) q,t f ( x ) , where s b are vertex-pair samples from{ 1 , . . .
, N } , chosen uniformly with replacement, independent of each other and of x .
The expectationof (cid:98) A B (cid:98) q,t f ( x ) with respect to re-sampling is E B [ (cid:98) A B (cid:98) q,t f
[Equation]
x )]
[Equation] S [
[Equation]
) (cid:98) q [Equation] ( x )] = A (cid:98) q
[Equation]
f ( x ) .
The corresponding re-sampled AgraSSt is (cid:92) AgraSSt
[Equation]
= [Equation] Similar to Eq.(9), the squared version of (cid:92) AgraSSt admits a representation in a quadratic form, [Equation]
[Equation]
where (cid:98) h x ( s, s ′ )
= (cid:68) A ( s ) (cid:98) q,t K ( x, · ) , A ( s ′ ) (cid:98) q,t K ( · , x ) (cid:69) H .
We note that the randomised operator obtainedvia re-sampling is a form of stochastic Stein discrepancy as introduced in [Gorham et al., 2020].
For fixed x , under mild conditions the consistency of (cid:92) AgraSSt 2 ( (cid:98) q, t ; x ) as B → ∞ is ensured by thefollowing normal approximation, which follows from Proposition 2 in [Xu and Reinert, 2021].
Proposition 3.4.
Assume that (cid:98)
h
x
( s, s ′ ) in (10) is bounded and that (cid:92)
AgraSSt( (cid:98) q, t ; x ) has non-zero variance σ 2 .
Let Z be a normal variable with mean AgraSSt( (cid:98) q, t ; x ) and variance σ 2 .
Then there exists an explicitly computable constant C > 0 such that for all 3 times continuously differentiable functions g with bounded derivatives up to order 3,
[Equation]
Applications of AgraSSt
4.1 Assessing graph generators
AgraSSt measures the distributional difference between the underlying distribution of an implicitgraph generator G and an observed graph x , which is useful to assess the quality of the generator G .
The hypothesis testing procedure for the null hypothesis that the observed graph x comes from thesame distribution that generates the samples, against the general alternative, is shown in Algorithm2.
We emphasise two features of this procedure.
Firstly, for a given generator G , AgraSSt directlyassesses the quality of the implicit model represented via samples from G .
Secondly, the generator
G can be trained on the observed graph x , for example through a deep neural network generator.
By learning a deep neural network generator with training samples from the same distribution thatgenerate x , AgraSSt can assess the quality of the training procedure, i.e. whether the deep neuralnetwork is capable of learning the desired distributions.
Additional details are discussed in SI.C.
4.2 Interpreting trained graph generators
If the procedure in Algorithm 2 rejects the null hypothesis, the generator may not be suitable forgenerating samples from the distribution that generates the one observed graph.
Hence, understandingwhere the misfit comes from can be very useful, especially for models trained from black-box deepneural networks.
AgraSSt provides an interpretable model criticism by comparing the learned (cid:98) q t ( x s, 1 )with the underlying q t ( x s, 1 ) when available, such as in synthetic experiments from a specified ERGM.
Such an interpretation can be also useful to re-calibrate training procedures.
4.3 Identifying reliable graph samples
If the procedure in Algorithm 2 does not reject the null hypothesis, there is not enough evidence toreject the hypothesis that the generator is capable of generating graphs that resembles the observedgraph.
If a generator G has passed this hurdle then it can be recommended for generating graphsamples of the desired type.
AgraSSt can also be put to use for the task of sample batch selection.
Inreal scientific studies, only a small batch of representative graph samples may need to be generatedfor downstream tasks such as privacy-preserving methods where users only access a small number ofgraph data, or a randomised experimental design for community interaction.
To quantify the qualityof sample batches via p -values : (1) generate a sample batch of size m , say; (2) perform the steps in
Procedure from Algorithm 2; (3) compute the p -value m − 1 (cid:80) mi
=1 1 ( τ > τ i ) , with τ as in step 3 and{ τ 1 , . .
, τ m } as in step 5.
If the p -value is smaller than a pre-specified threshold, generate anothersample batch; otherwise accept the current sample batch.
Empirical results
We first illustrate the performance of AgraSSt on synthetic data, where the null distribution is knownand we have control of the set-up; in particular we can illustrate the use of AgraSSt for interpretablemodel criticism.
Then we show the performance of AgraSSt on a real-world data application to assessgraph generators trained via various deep generative models.
5.1 Synthetic experiments
Only few competing approaches are available for our task and many of them are devised specificallyfor ERGMs.
Hence here we use an ERGM, namely the Edge-2Star-Triangle (E2ST) model with
[Equation]
where E d ( x ) denotes the number of edges of x , S 2 ( x ) denotes the 2-Star statistics and T r ( x ) denotesthe triangle statistics.
Here, β = ( − 2 . 00 , 0 . 00 , 0 . 01) is chosen as the null model, while alternativemodels are constructed by perturbing the coefficient β 2 as in Yang et al.
[2018], Xu and Reinert[2021].
This particular ERGM is chosen because it is the currently most complex ERGM for which athorough theoretical analysis for parameter estimation is available, see Mukherjee and Xu [2013] 2 .
5.1.1 Related approaches for comparisons
To assess the performance of AgraSSt, we consider the following existing test statistics which areeither tailored or modified to perform assessment for implicit graph generators: Deg is a degree-basedstatistics for goodness-of-fit test of exchangeable random graphs [Ouadah et al., 2020] based onthe estimated variance of the degree distribution.
The statistics can be obtained from empiricaldegrees from samples generated from the implicit model.
TV_deg denotes the Total-Variation (TV)distance between degree distributions.
Hunter et al.
[2008] proposes a simulation-based approach toconstruct graphical goodness-of-fit tests.
Xu and Reinert [2021] quantifies this approach using thetotal-variation distance between the distributions of chosen network statistics; see SI.D.2 for details.
MDdeg is the Mahalanobis distance between degree distributions [Lospinoso and Snijders, 2019].
In the synthetic experiment where the parametric model is known (Eq.(11)), the coefficients β areestimated from generated samples for model assessment to provide our baseline approach, denotedby Param .
Details can be found in SI.B.1.
Knowing the explicit null model in the synthetic setting,we can also compute gKSS in Eq.(3), denoted by Exact as our benchmark.
The Weisfeiler-Lehmangraph kernel [Shervashidze et al., 2011] with height parameter 3 is used for kernel-based approaches.
5.1.2 Simulation results
The rejection rates for various settings are shown in Figure.2.
As the null model is relatively sparse(edge density 11 .
2% ), the sparser alternatives are much harder to distinguish while the denser onesare easier problems.
In Figure.2(a), we compare the AgraSSt procedure with the approaches from Section 5.1.1. From the results, we see that AgraSSt performs competitively to the benchmark Exact , which is only available when the model is known explicitly, and outperforms other assessmentprocedures for implicit models.
TV_deg is slightly less powerful than AgraSSt but outperforms Deg .
Due to the small perturbation of the model parameter for the alternative, creating a hardproblem for sparser alternatives, MDdeg and Param have a much lower rejection rate and thus less
[Figure]
j
[Figure]
c t i o n R a t e
DegMDdegTV
[Figure]
AgraSStParamExact
[Figure]
Different [Figure] approaches
[Figure]
powerful for sparser alternatives.
In Figure.2(b), we compare the performance of AgraSSt withdifferent estimation methods for (cid:98) q t ( x s, 1 ) .
The degree deg ( k ) of a vertex k is calculated excludingthe vertex-pair s =
( i, j ) ; Sum_deg : for s =
( i, j ) we set t =
deg ( i ) + deg ( j ) ; Cum_deg : thecumulative distribution function of sum of degrees are used; Bi_deg : for s =
( i, j ) the 2-dimensionalvector t =
( deg ( i ) , deg ( j )) is used; Edges : t is the edge density after removing vertex-pair s .
Fromthe results, we see that Edges outperforms the other estimates, which echos the theoretical resultsshown in Theorem A.5 in SI, as the coefficient β for E2ST satisfies its assumptions.
We also see thatusing both vertex degrees as 2 d vector predicts substantially better than using predictors based onsum of degrees of two vertices.
In Figure.2(c), the comparison with re-sampling is shown.
Withincrease in re-sampling size, the power of AgraSSt increases.
5.2 Real-world applications on deep graph generators
We now assess the performances of a set of state-of-the-art deep generative models for graphs trainedon ERGMs and the Karate Club network collected by Zachary [1977].
The Karate Club network has34 vertices and 78 edges representing friendships.
Soon after the data collection the Karate Clubseparated into two factions.
This graph is a benchmark graph for community detection.
One wouldnot expect this graph to be close to an G ( n, p ) 3 graph or to be well modelled by an ERGM.
5.2.1 Graph generation methods
GraphRNN
[You et al., 2018] is an architecture to generate graphs from learning two recurrent neuralnetworks (RNN), one a vertex-level RNN and the other an edge-level RNN.
The procedure startsfrom a breadth-first-search for vertex ordering; two RNNs are trained from a sequential procedure.
NetGAN [Bojchevski et al., 2018] utilises an adversarial approach by training an interplay between agenerator and a discriminator neural network on graph data.
CELL [Rendsburg et al., 2020] improveson the NetGAN idea by solving a low-rank approximation problem based on a cross-entropy objective.
MC is the standard Monte-Carlo network sampling in the ergm suite in R and is used as a baselinewhen the simulated network is known to follow the model in Eq.(1); q ( x ) needs to be known.
5.2.2 Generator assessment results
[Table]
We first train the generative models withsamples from ERGMs to assess their ability to generate ERGMs.
The test resultsare shown in Table.1.
From the result,we see that for the “reliable” MC generator all the assessment statistics presentedhave well-controlled type-I error.
Samplesgenerated from CELL deviate not too farfrom the test level, indicating a good generative model for ERGMs.
NetGAN and
GraphRNN both encounter a high rejection rate, implying that the generated samples that are notclose to the training E2ST model.
From the density based AgraSSt , taking a G ( n, q ) model, we can interpret the model misfit bychecking the estimated (cid:98) q .
For the true E2ST model to generate training samples, q = 0 . 112 ,while CELL has (cid:98) q = 0 . 116 which is close to the null.
GraphRNN estimates (cid:98) q = 0 .
128 whichis substantially higher than the null.
Although GraphRNN is good in learning local patterns andstructures for neighbourhoods [You et al., 2018], it does not take the overall density sufficiently intoaccount.
Due to its limited “look back” and absence of “look forward” on the ordered vertex setduring training, the over-generation of edges may have caused this significant difference for learning
ERGMs.
NetGAN, on the other hand, produces a close estimate (cid:98) q = 0 .
However, countingtriangles, it only has on average 12 .
6 triangles , which is far less from the null with expected numberof triangles 46 .
NetGAN, due to its random walk adversarial procedure, may not be effective inlearning such clustered patterns.
5.2.3 Case study: Karate Club network
[Table]
Next, we assess the performances of thesegenerative models by training on the Karate
Club network [Zachary, 1977].
The pvalues for different testing procedures areshown in Table.2.
From the results, wesee that AgraSSt rejects samples generatedfrom both GraphRNN and NetGAN trained with the Karate Club network.
Although the edgedensities generated from the trained GraphRNN (edge density 15 .
3% ) and NetGAN (edge density13 .
4% ) are comparable with the Karate Club edge density of 13 .
9% , both GraphRNN and NetGANsamples exhibit a single large component rather than two fairly separated communities in the Karate Club network.
This difference is picked up by AgraSSt, Deg and TV_deg, which all reject bothmodels.
On the other hand, CELL generates samples that are not rejected by all tests at significancelevel α = 0 .
In Figure.7 in the SI, the Karate Club network is shown in Figure.7(a).
Samples from GraphRNN, NetGAN and CELL are shown in Figure.7(b), 7(c) and 7(d) respectively.
SI D includes additional results and visualisations; a second case study — the Florentine marriagenetwork from Padgett and Ansell [1993] — is presented in SI D.4; additional visualisations of thereliable sample batch selection procedure described in Section 4.3 are also included.
Discussions and future directions
In this paper, we propose AgraSSt, a unique general purpose model assessment and criticismprocedure for implicit random graph models.
As it is based on a kernel Stein statistic, we are able togive theoretical guarantees.
AgraSSt not only solves an important problem but also opens up a wholeset of follow-up research problems of which we list a few here.
(i).
Currently AgraSSt is only appliedto undirected and unweighted graphs.
Extensions to more general graphs as well as to time series ofgraphs will be interesting to explore in follow-up work.
(ii).
AgraSSt could be also helpful to improvedesign and training of deep graph generative models, e.g. by regularising graph features if there is amisfit.
(iii).
ERGMs allow for exogenous features to be included in the sufficient statistics.
AgraSStcan be based on a variety of statistics t ( x ) ; further examples are found in
Appendix E.
It would alsobe possible to incorporate exogenous features in the statistics t ( x ) in AgraSSt, for example usingideas from graph attention networks [Veliˇckovi´c et al., 2018].
Exploring this idea in more detail willbe another topic of further research.
As AgraSSt depends on the chosen summary statistic t ( x ) , results have to be interpreted with regardsto the respective conditional distributions.
Also multiple tests will have the p -values to be adjusted toavoid misinterpretation of tests, which could have serious consequences for example in the area ofpersonal health.