AgraSSt: Approximate Graph Stein Statistics for Interpretable Assessment of Implicit Graph Generators
s)
AffiliationAddress email
Abstract
We propose and analyse a novel statistical procedure, coined AgraSSt , to assessthe quality of graph generators which may not be available in explicit forms.
Inparticular, AgraSSt can be used to determine whether a learned graph generating process is capable of generating graphs which resemble a given input graph.
Inspired by Stein operators for random graphs, the key idea of AgraSSt is theconstruction of a kernel discrepancy based on an operator obtained from the graphgenerator.
AgraSSt can provide interpretable criticisms for a graph generator training procedure and help identify reliable sample batches for downstream tasks.
Wegive theoretical guarantees for a broad class of random graph models.
We provideempirical results on both synthetic input graphs with known graph generationprocedures, and real-world input graphs that the state-of-the-art (deep) generativemodels for graphs are trained on.
Introduction
Generative models for graphs have received increasing attention in the statistics and machine learningcommunities.
Recently, deep neural networks have been utilised to learn rich representations fromgraph structures and generate graphs [Dai et al., 2020, Li et al., 2018, Liao et al., 2019, You et al.
However, due to the often opaque deep learning procedures, these deep generative models areusually implicit, which hinders theoretical analysis to assess how close the generated samples are intheir distributional properties to the graph distribution they are meant to be sampled from.
Learning parametric models requires an explicit pre-specified probability distribution class, and thelearned parameters can be used for model assessment.
However, parametric models may only capturea fraction of the graph features and have restricted modelling power.
Due to dependency betweenedges, parameter estimation can be inconsistent [Shalizi and Rinaldo, 2013] even for well-specifiedmodels, and may lead to wrong conclusions in model assessments.
Although recent advances indeep generative models for graphs may surpass some of the above-mentioned issues by learning richgraph representations, methods to assess the quality of such implicit graph generators are lacking.
In principle, nonparametric hypothesis tests can be useful to assess complex models, such as by thepopular kernel-based tests procedures that utilise functions in a reproducing kernel Hilbert space(RKHS)
[Berlinet and Thomas, 2004].
When the models are described in the form of explicitprobabilities, goodness-of-fit tests [Chwialkowski et al., 2016, Liu et al., 2016] may apply; however,goodness-of-fit testing procedures are generally not applicable for implicit models.
Instead, if a largeset of samples from the target distribution is observed, one may generate samples from the implicit
[Figure]
model and perform a two-sample test such as a maximum mean discrepancy (MMD) test [Grettonet al., 2007] for model assessment [Jitkrittum et al., 2017, Xu and Matsuda, 2020, 2021].
However, in real-world applications, often only a single graph from the target distribution is observed[Bresler and Nagaraj, 2018, Reinert and Ross, 2019].
In this case, the MMD methods for modelassessment via two-sample testing procedures cannot be used to assess implicit graph generators.
Toour knowledge, beyond simple Monte Carlo tests with often poor power, no principled test is availablefor assessing the quality of implicit graph generators.
In Figure.1, the task is illustrated: we are givena graph generator G which is learned from a set of training samples and can generate samples ofuser-defined size.
The task is to assess whether G can generate samples from the same distributionthat generates the observed graph, for example, Padgett’s Florentine marriage network [Padgett and
Ansell, 1993].
In Figure.1, G is a Cross-Entropy Low-rank Logit (CELL) model
[Rendsburg et al. ,2020] trained on Padgett’s Florentine marriage network, but any graph generator could be used.
This paper makes three main contributions.
(1) We introduce Approximate graph Stein Statistics(AgraSSt) in Section 3, which opens up a principled way to understand implicit graph generators.
AgraSSt is a variant of a kernel Stein discrepancy based on an empirical Stein operator for conditionaldistributions of general random graph models.
The testing procedure is inspired by gKSS, a goodnessof-fit testing procedure for explicit exponential random graph models (ERGMs)
[Xu and Reinert,2021].
(2) We provide theoretical guarantees for AgraSSt (Section 3).
(3) We propose interpretablemodel criticisms when there is a model misfit and identify representative synthetic sample batcheswhen the model is a good fit (Section 4).
Further, in Section 2 we review gKSS.
We provide empiricalresults in Section 5 and a discussion in Section 6.
Proof details, more background, theoretical andempirical results, and implementation details are found in the Supplementary Information (SI) text.
Code is attached as a folder in the SI.
Background: graph kernel Stein statistics
In [Xu and Reinert, 2021] a goodness-of-fit testing procedure called gKSS for exponential randomgraph models is introduced, which assumes that only a single network may be observed in the sample.
As AgraSSt is inspired by gKSS, and gKSS serves as comparison method when the observed networkis known to be generated from an exponential random graph model (ERGM), we briefly review gKSS.
The notation introduced in this section is used throughout the paper.
ERGMs are extensively used for social network analysis [Wasserman and Faust, 1994, Holland and Leinhardt, 1981, Frank and Strauss, 1986]; a special case are Bernoulli random graphs.
ERGMsmodel random graphs via a Gibbs measure with respect to a chosen set of network statistics, suchas number of edges, 2-stars, and triangles.
Denote by G labn , the set of vertex-labeled graphs on nvertices, with N = n ( n − 1) / 2 possible undirected edges.
Encode x ∈ G labn by an ordered collectionof { 0 , 1 } -valued variables x = ( x ( ij ) ) 1 ≤ i<j ≤ n ∈ { 0 , 1 } N where x ( ij ) =[-
-]
if and only if there is anedge between i and j .
We denote an (ordered) vertex-pair index s =
( i, j ) by s ∈ [ N ] := { 1 , . . .
, N } .
Let t 1 ( x ) count the number of edges in x and let t 2 ( x ) , . . . , t k ( x ) count other small graphs on atmost n vertices in x (possibly scaled; for details including a precise definition, Definition A.3, see
SI A.2). For β = ( β 1 , . . .
, β k ) ⊤ ∈ R k and t ( x ) =
( t 1 ( x ) , . . .
, t k ( x ))
⊤ ∈
R
k
we say that X ∈ G labnfollows the exponential random graph model X ∼ ERGM( β, t ) if for ∀ x ∈ G labn ,
[Equation]
Here κ n ( β ) is a normalisation constant.
In this model, t ℓ , ℓ = 1 , . . .
, k , are sufficient statistics.
Parameter estimation ˆ β l for β l is only possible when the sufficient statistics are specified a priori; see
SI.B.1 for estimation details.
In modern graph learning procedures, e.g. deep generative learning, thesufficient statistics t l of Eq.(1) may not be obtained explicitly.
In [Reinert and Ross, 2019] the exponential random graph distribution in Eq.
1 is characterisedby a so-called Stein operator , as follows.
Let e s ∈ { 0 , 1 } N be a vector with 1 in coordinate sand
in all others
; x ( s, 1) = x + (1 − x s ) e s has the s -entry replaced of x by the value 1, andx ( s, 0) = x − x s e s has the s -entry of x replaced by the value 0; moreover, x − s is the set of edgeindicators with entry s removed.
For a function h : { 0 , 1 } N → R , let ∆ s h ( x ) =
h ( x ( s, 1) ) − h
( x ( s, 0) ) .
Set q X ( x ( s, 1) | x − s ) =
P ( X s = 1 | X − s = x − s ) .
Define the operator
[Equation]
Then under mild conditions Reinert and Ross [2019] show that if E p [ A β,t f ] = 0 for all smooth test functions f , then p must be the distribution of ERGM ( β, t ) .
Thus, this operator characterises
ERGM ( β, t ) .
For the derivation of AgraSSt it is of interest to see how this operator is obtained.
It is indeed the generator of a so-called Glauber Markov chain on G labn with transition probabilities
[Equation]
With the ERGM Stein operator in Eq.(2) and a rich-enough RKHS test function class H , Xu and
Reinert [2021] propose a graph kernel Stein statistics (gKSS) to perform goodness-of-fit testing on anexplicit ERGM when a single network sample is observed.
With the summand components in Eq.(2),the Stein operator can be seen as taking expectation over vertex-pair variables S ∈ [ N ] with uniformprobability P ( S = s ) ≡ N − 1 independently of x , namely
[Equation]
For a fixed graph x , gKSS is defined as
[Equation]
where the function f is chosen to best distinguish q from x .
For an RKHS H associated withkernel K , by the reproducing property of H , the squared version of gKSS admits a quadratic formrepresentation gKSS 2 ( q ; x ) = ⟨ E S
[ A ( S ) q K
( x, · )] , E S
[ A ( S ) q K
( x, · )] ⟩ , which can be computed readily.
More background can be found in SI B.2.
AgraSSt: Approximate Graph Stein Statistic
An implicit graph generator may not admit a probability distribution in the form of Eq.(1); however,the idea of constructing Stein operators based on Glauber dynamics using conditional probability distribution for ERGM is inspiring.
Here we propose Stein operators for conditional graph distributions,to facilitate an (approximate) characterisation for implicit random graph models.
3.1 Stein operators for conditional graph distributions
Let q ( x ) =
P ( X = x ) be any distribution with support G labn .
Let t ( x ) denote a statistic on graphswhich takes on finitely many values k and let q k ( x )
=
P ( X = x
| t ( x ) = k ) .
We assume thatq k ( x ) > 0 for all k under consideration.
For a generic outcome we write q t .
We introduce a Markovchain on G labn which transitions from x to x ( s, 1) with probability [Equation]
[Equation]
and which transitions from x to x ( s, 0) with probability q ( x ( s, 0) | t [Equation] x
[Equation]
s )) = 1 − q t ( x ( s, 1) ); no othertransitions occur.
Let
[Equation]
For an ERGM, t ( x ) could be taken as a sufficient vector of statistics 1 , but here we do not assume aparametric network model q ( x ) , and t ( x ) does not have to be sufficient statistics for q ( x ) .
Recall that an operator is a Stein operator for a distribution µ if its expectation under µ is zero.
Thefollowing result provides a theoretical foundation for AgraSSt and is proven in SI.A.
Lemma 3.1.
If q k ( x ) > 0 for all k , then A ( s ) q,t = k is a Stein operator for the conditional distributionof X given t ( X ) =
k , and (cid:80) s A ( s ) q,t
= k is a Stein operator for the conditional distribution of X givent ( X ) = k .
In particular, E q t = k [ A ( s ) q,t = k ] = 0 .
Intuitively, if the distribution of ˜ X is close to that of X , thenwith ˜ Y t
=
k denoting the corresponding random graph with distribution that of ˜ X given t ( ˜ X ) =
k, itshould hold that E [ A ( s ) q,t = k
( ˜ Y t = k )]
≈ 0 .
In this way the Stein operator in Eq.(5) can be used to assesssimilarity between distributions.
3.2 Approximate Stein operators
For implicit models and graph generators G , the Stein operator A ( s ) q,t in Eq.(5) cannot be obtainedwithout explicit knowledge of q t ( x ( s, 1) ) .
However, given a large number for samples from the graphgenerator G , the conditional edge probabilities q t ( x ( s, 1) ) can be estimated.
Here we denote by(cid:98) q t ( x ( s, 1) ) an estimate of q t ( x ( s, 1) ) ; some estimators will be suggested in Section 3.3.
AgraSSt performs model assessment using an operator which approximates the Stein operator A ( s ) q,t .
We define the approximate Stein operator for the conditional random graph by
[Equation]
The vertex-pair averaged approximate Stein operator is
[Equation]
3.3 Estimation with chosen statistics on graphs
Using the Stein operator for conditional graph distributions, we can obtain the approximate Steinoperators Eq.(6) and (7) for an implicit graph generator G by estimating q t ( x ( s, 1) ) .
Here t ( x ) areuser-defined statistics.
In principle, any multivariate statistic t ( x ) can be used in this formalism.
However, estimating the conditional probabilities using relative frequencies can be computationallyprohibitive when the graphs are very large and specific frequencies are rarely observed.
Instead,here we consider simple summary statistics, such as edge density, degree statistics or the number ofneighbours connected to both vertices of s .
The estimation procedure is presented in Algorithm 1.
To estimate q t ( x ( s, 1) ) in Step 3 of Algorithm 1, a look-up table can be used: If t ( x ) is a possiblymultivariate statistic with a discrete number of outcomes, generate many independent copies from the synthetic graph generator, and count n ( k, s ) , the number of times that vertex-pair s is present inthe simulated graphs and t ( x − s )
=
k . Consistency follows from the law of large numbers. Whenusing smoothing to estimate the conditional probability, consistency will depend on the smoothingmethod. If the underlying graph has exchangeable edge indicators then q t ( x ( s, 1) ) does not depend onthe choice of vertex-pair s ; set n ( k ) =
(cid:80) s n ( k, s ) , and
N k = (cid:80) Li =1 (cid:80) s 1 ( t ( x − s ) =
k ) ; if t = kthen estimate (cid:98) q t ( x ( s, 1) ) by [Equation]
[Equation]
If k min , k max denote the minimum and maximum values of statistics from simulated graph samples,then for k outside this set, the lookup table estimator Eq.(8) is set to estimate ˆ g ( k ) = 0 .
If the underlying graph cannot be assumed to have exchangeable edge indicators or if the statistic t ishigh dimensional, then any particular n ( k, s ) may not be observed very often.
In such a situation wecan learn ˆ g t ( s ; k ) using kernel ridge regression so that the conditional probabilities for similar t ( x − s )are predicted in a smooth manner.
Estimating q
x s | t ( x − s ) ≤ k ) instead of q ( x s | t ( x − s )
= k ) mayprovide an alternative, smoother estimate for the conditional probabilities.
The next result shows that the approximate Stein operator achieves the Stein identity asymptotically.
For this result, which is proved in SI.A, we use the notation ∥
∆ f ∥ = sup s ∈ [ N ] ,x ∥ ∆ s f ( x ) ∥ .
Theorem 3.2.
Assume (cid:98) q t ( x ( s, 1) ) is a consistent estimator for q t ( x ( s, 1) ) as L → ∞ .
Then for any fsuch that || ∆ f || < ∞ we have E q [ A (cid:98) q,t f ( x )]
→ E q [ A q,t f ( x )] = 0 as L → ∞ .
Proofs and additional analysis are included in SI A. Proposition A.1 in SI A.1 addresses consistentestimation of (cid:98) q t and SI A.2 provides refined results for ERGMs, including a Gaussian approximation.
3. AgraSSt for implicit graph generators
The estimated conditional probabilities give an approximate Stein operator for Eq.(5).
With the appropriately defined Stein operator from an implicit model given in Eq.(6), we can define AgraSSt, a kernelbased statistic analogous to gKSS in Eq.(3), as AgraSSt
[Equation]
In SI.A we prove the following result.
Theorem 3.3.
If the graph is edge-exchangeable, then AgraSSt 2 ( (cid:98) q, t ; x ) is a consistent estimator of [Equation]
[Equation]
Re-sampling Stein statistic A computationally efficient operator for large N can be derived via re-sampling vertex-pairs s , which creates a randomised operator.
Let B be the fixed size to be re-sampled.
The re-sampled operator is
[Equation] where s b are vertex-pair samples from { 1 , . .
. , N } , chosen uniformly with replacement, independent of each other and of x .
The expectation of (cid:98) A B (cid:98) q,t f ( x ) with respect to re-sampling is
[Equation]
The corresponding re-sampled AgraSSt is (cid:92)
AgraSSt( (cid:98)
[Equation]
=
[Equation]
(cid:12)(cid:12)(cid:12) 1 B (cid:80) b ∈
[ B ] A (
[Equation]
b ) (cid:98) q,t f ( x ) (cid:12)(cid:12)(cid:12) .
Similar to Eq.(9), the squared version of (cid:92) AgraSSt admits a representation in a quadratic form,(cid:92) AgraSSt
[Equation]
10) where (cid:98) h x ( s, s ′ )
= ⟨A ( s ) (cid:98) q,t K ( x, · ) , A ( s ′ ) (cid:98) q,t K ( · , x ) ⟩ H .
We note that the randomised operator obtainedvia re-sampling is a form of stochastic Stein discrepancy as introduced in Gorham et al.
For fixed x , under mild conditions the consistency of (cid:92) AgraSSt 2 ( (cid:98) q, t ; x ) as B → ∞ is ensured by thefollowing normal approximation, which follows from Proposition 2 in Xu and Reinert [2021].
Proposition 3.4.
Assume that (cid:98)
h
x
( s, s ′ ) in (10) is bounded and that (cid:92)
AgraSSt( (cid:98) q, t ; x ) has non-zero variance σ 2 .
Let Z be a normal variable with mean AgraSSt( (cid:98) q, t ; x ) and variance σ 2 .
Then there exists an explicitly computable constant C > 0 such that for all 3 times continuously differentiable functions g with bounded derivatives up to order 3,
[Equation]
Applications of AgraSSt
4.1 Assessing graph generators
AgraSSt measures the distributional difference between the underlying distribution of an implicitgraph generator G and an observed graph x , which is useful to assess the quality of the generator G .
The hypothesis testing procedure for the null hypothesis that the observed graph x comes from thesame distribution that generates the samples, against the general alternative, is shown in Algorithm2.
We emphasise two features of this procedure.
Firstly, for a given generator G , AgraSSt directlyassesses the quality of the implicit model represented via samples from G .
Secondly, the generator
G can be trained on the observed graph x , for example through a deep neural network generator.
By learning a deep neural network generator with training samples from the same distribution thatgenerate x , AgraSSt can assess the quality of the training procedure, i.e. whether the deep neuralnetwork is capable of learning the desired distributions.
Additional details are discussed in SI.C.
4.2 Interpreting trained graph generators
If the procedure in Algorithm 2 rejects the null hypothesis, the generator may not be suitable forgenerating samples from the distribution that generates the one observed graph.
Hence, understandingwhere the misfit comes from can be very useful, especially for models trained from black-boxdeep neural networks.
AgraSSt provides an interpretable model criticism by comparing the learned(cid:98) q t ( x ( s, 1) ) with the underlying q t ( x ( s, 1) ) when available, such as in synthetic experiments from aspecified ERGM.
Such an interpretation can be also useful to re-calibrate training procedures.
4.3 Identifying reliable graph samples
If the procedure in Algorithm 2 does not reject the null hypothesis, there is not enough evidence toreject the hypothesis that the generator is capable of generating graphs that resembles the observedgraph.
If a generator G has passed this hurdle then it can be recommended for generating graphsamples of the desired type.
AgraSSt can also be put to use for the task of sample batch selection.
Inreal scientific studies, only a small batch of representative graph samples may need to be generatedfor downstream tasks such as privacy-preserving methods where users only access a small number ofgraph data, or a randomised experimental design for community interaction.
To quantify the qualityof sample batches via p -values : (1) generate a sample batch of size m , say; (2) perform the steps in
Procedure from Algorithm 2; (3) compute the p -value m − 1 (cid:80) mi
=1 1 ( τ > τ i ) , with τ as in step 3 and{ τ 1 , . .
, τ m } as in step 5.
If the p -value is smaller than a pre-specified threshold, generate anothersample batch; otherwise accept the current sample batch.
Empirical results
We first illustrate the performance of AgraSSt on synthetic data, where the null distribution is knownand we have control of the set-up; in particular we can illustrate the use of AgraSSt for interpretablemodel criticism.
Then we show the performance of AgraSSt on a real-world data application to assessgraph generators trained via various deep generative models.
5.1 Synthetic experiments
Only few competing approaches are available for our task and many of them are devised specificallyfor ERGMs.
Hence here we use an ERGM, namely the Edge-2Star-Triangle (E2ST) model with
[Equation]
where E d ( x ) denotes the number of edges of x , S 2 ( x ) denotes the 2-Star statistics and T r ( x ) denotesthe triangle statistics.
Here, β = ( − 2 . 00 , 0 . 00 , 0 . 01) is chosen as the null model, while alternativemodels are constructed by perturbing the coefficient β 2 as in Yang et al.
[2018], Xu and Reinert[2021].
This particular ERGM is chosen because it is the currently most complex ERGM for which athorough theoretical analysis for parameter estimation is available, see Mukherjee and Xu [2013] 2 .
5.1.1 Related approaches for comparisons
To assess the performance of AgraSSt, we consider the following existing test statistics which areeither tailored or modified to perform assessment for implicit graph generators: Deg is a degree-basedstatistics for goodness-of-fit test of exchangeable random graphs [Ouadah et al., 2020] based onthe estimated variance of the degree distribution.
The statistics can be obtained from empiricaldegrees from samples generated from the implicit model.
TV_deg denotes the Total-Variation (TV)distance between degree distributions.
Hunter et al.
[2008] proposes a simulation-based approach toconstruct graphical goodness-of-fit tests.
Xu and Reinert [2021] quantifies this approach using thetotal-variation distance between the distributions of chosen network statistics; see SI.D.4 for details.
MDdeg is the Mahalanobis distance between degree distributions [Lospinoso and Snijders, 2019].
In the synthetic experiment where the parametric model is known (Eq.(11)), the coefficients β areestimated from generated samples for model assessment to provide our baseline approach, denotedby Param .
Details can be found in SI.B.1.
Knowing the explicit null model in the synthetic setting,we can also compute gKSS in Eq.(3), denoted by Exact as our benchmark.
The Weisfeiler-Lehmangraph kernel [Shervashidze et al., 2011] with height parameter 3 is used for kernel-based approaches.
5.1.2 Simulation results
The rejection rates for various settings are shown in Figure.2.
As the null model is relatively sparse(edge density 11 .
2% ), the sparser alternatives are much harder to distinguish while the denser ones areeasier problems.
Figure.2(a) compares the AgraSSt procedure with the approaches from Section 5.1.1;it shows that AgraSSt performs competitively to the benchmark Exact , which is only available whenthe model is known explicitly, and outperforms other assessment procedures for implicit models.
TV_deg is slightly less powerful than AgraSSt but outperforms Deg .
For sparser alternatives (smallerβ 2 ), MDdeg and Param have a much lower rejection rate and thus are less powerful. In Figure.2(b),we compare the performance of AgraSSt with different estimation methods for (cid:98) q t ( x ( s, 1) ) . The degree studied. [Figure]
[Figure]
R
[Figure]
j e c t i o n R a t e
DegMDdegTV
[Figure]
deg AgraSStParamExact
[Figure]
Different assessment approaches
[Figure]
deg ( k ) of a vertex k is calculated excluding the vertex-pair s =
( i, j ) ; Sum_deg : for s =
( i, j )we set t =
deg ( i ) + deg ( j ) ; Cum_deg : the cumulative distribution function of sum of degreesare used; Bi_deg : for s =
( i, j ) the 2-dimensional vector t =
( deg ( i ) , deg ( j )) is used; Edges : t isthe edge density after removing vertex-pair s .
From the results, we see that Edges outperforms theother estimates, which echos the theoretical results shown in Theorem A.5 in SI, as the coefficientβ for E2ST satisfies its assumptions.
Hence this is the statistics which is used in our real-worldexperiments, with an exception of SI.D.3.
We also see that using both vertex degrees as 2 d vectorpredicts substantially better than using predictors based on sum of degrees of two vertices.
In Figure.2(c), the comparison with re-sampling is shown.
With increase in re-sampling size, the powerof AgraSSt increases.
5.2 Real-world applications on deep graph generators
We now assess the performances of a set of state-of-the-art deep generative models for graphs trainedon ERGMs and the Karate Club network collected by Zachary [1977].
The Karate Club network has34 vertices and 78 edges representing friendships.
Soon after the data collection the Karate Clubseparated into two factions.
This graph is a benchmark graph for community detection.
One wouldnot expect this graph to be close to an G ( n, p ) 3 graph or to be well modelled by an ERGM.
5.2.1 Graph generation methods
GraphRNN
[You et al., 2018] is an architecture to generate graphs from learning two recurrent neuralnetworks (RNN), one a vertex-level RNN and the other an edge-level RNN.
The procedure startsfrom a breadth-first-search for vertex ordering; two RNNs are trained from a sequential procedure.
NetGAN [Bojchevski et al., 2018] utilises an adversarial approach by training an interplay between agenerator and a discriminator neural network on graph data.
CELL [Rendsburg et al., 2020] improveson the NetGAN idea by solving a low-rank approximation problem based on a cross-entropy objective.
MC is the standard Monte-Carlo network sampling in the ergm suite in R and is used as a baselinewhen the simulated network is known to follow the model in Eq.(1); q ( x ) needs to be known.
5.2.2 Generator assessment results
[Table]
We first train the generative models withsamples from ERGMs to assess their ability to generate ERGMs.
The test resultsare shown in Table.1.
From the result,we see that for the “reliable” MC generator all the assessment statistics presentedhave well-controlled type-I error.
Samplesgenerated from CELL deviate not too farfrom the test level, indicating a good generative model for ERGMs.
NetGAN and
GraphRNN both encounter a high rejection rate, implying that the generated samples that are notclose to the training E2ST model.
From the density based AgraSSt , taking a G ( n, q ) model, we can interpret the model misfit bychecking the estimated (cid:98) q .
For the true E2ST model to generate training samples, q = 0 . 112 ,while CELL has (cid:98) q = 0 . 116 which is close to the null.
GraphRNN estimates (cid:98) q = 0 .
128 whichis substantially higher than the null.
Although GraphRNN is good in learning local patterns andstructures for neighbourhoods [You et al., 2018], it does not take the overall density sufficiently intoaccount.
Due to its limited “look back” and absence of “look forward” on the ordered vertex setduring training, the over-generation of edges may have caused this significant difference for learning
ERGMs.
NetGAN, on the other hand, produces a close estimate (cid:98) q = 0 .
However, countingtriangles, it only has on average 12 .
6 triangles , which is far less from the null with expected numberof triangles 46 .
NetGAN, due to its random walk adversarial procedure, may not be effective inlearning such clustered patterns.
5.2.3 Case study: Karate Club network
[Table]
Next, we assess the performances of thesegenerative models by training on the Karate
Club network [Zachary, 1977].
The pvalues for different testing procedures areshown in Table.2.
From the results, wesee that AgraSSt rejects samples generatedfrom both GraphRNN and NetGAN trained with the Karate Club network.
Although the edgedensities generated from the trained GraphRNN (edge density 15 .
3% ) and NetGAN (edge density13 .
4% ) are comparable with the Karate Club edge density of 13 .
9% , visual inspection indicatesthat both GraphRNN and NetGAN samples exhibit a single large component rather than two fairlyseparated communities in the Karate Club network.
This difference is picked up by AgraSSt, Degand TV_deg, which all reject both models.
On the other hand, CELL generates samples that are notrejected by all tests at significance level α = 0 .
In Figure.5 in the SI, the Karate Club network isshown in Figure.5(a).
Samples from GraphRNN, NetGAN and CELL are shown in Figure.5(b), 5(c)and 5(d) respectively.
SI D includes additional results, including on efficiency, and visualisations; a second case study —the Florentine marriage network from Padgett and Ansell [1993] — is presented in SI D.2; additionalvisualisations of the reliable sample batch selection procedure described in Section 4.3 are also given.
Discussions and future directions
In this paper, we propose AgraSSt, a unique general purpose model assessment and criticismprocedure for implicit random graph models.
As it is based on a kernel Stein statistic, we are ableto give theoretical guarantees.
AgraSSt not only solves an important problem but also opens up awhole set of follow-up research problems of which we list a few here.
(i).
Currently AgraSSt is onlyapplied to undirected and unweighted graphs.
Extensions to more general graphs as well as to timeseries of graphs will be interesting to explore in follow-up work.
(ii).
AgraSSt could be also helpfulto improve design and training of deep graph generative models, e.g. by regularising graph features ifthere is a misfit.
(iii).
ERGMs allow for exogenous features to be included in the sufficient statistics.
AgraSSt can be based on a variety of statistics t ( x ) ; further examples are found in SI D.3.
It wouldalso be possible to incorporate exogenous features in the statistics t ( x ) in AgraSSt, for example usingideas from graph attention networks [Veliˇckovi´c et al., 2018].
Exploring this idea in more detail willbe another topic of further research.
As AgraSSt depends on the chosen summary statistic t ( x ) , results have to be interpreted with regardsto the respective conditional distributions.
Also multiple tests will have the p -values to be adjusted toavoid misinterpretation of tests, which could have serious consequences for example in the area ofpersonal health.
Investigating the effect of choice of t ( x ) will be part of future work.
Checklist
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section
(b) Did you include complete proofs of all theoretical results?
[Yes] SI.A in Supplementary3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code attachedas a folder in the Supplementary(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Model details are included in the code.
Additional implementationdetails are included in SID.6.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Random seed are used for experiment and included in thecode.
See SID.3 for error bars for graphical tests.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See SID.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[N/A] Datasets and codes are open source.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
SID
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]