AgraSSt: Approximate Graph Stein Statistics for Interpretable Assessment of Implicit Graph Generators
Abstract
We propose and analyse a novel statistical procedure, coined AgraSSt , to assess the quality of graph generators which may not be available in explicit forms.
In particular, AgraSSt can be used to determine whether a learned graph generating process is capable of generating graphs which resemble a given input graph.
Inspired by Stein operators for random graphs, the key idea of AgraSSt is the construction of a kernel discrepancy based on an operator obtained from the graph generator.
AgraSSt can provide interpretable criticisms for a graph generator training procedure and help identify reliable sample batches for downstream tasks.
We give theoretical guarantees for a broad class of random graph models.
Moreover, we provide empirical results on both synthetic input graphs with known graph generation procedures, and real-world input graphs that the state-of-the-art (deep) generative models for graphs are trained on.
Introduction
Generative models for graphs have received increasing attention in the statistics and machine learning communities.
Recently, deep neural networks have been utilised to learn rich representations from graph structures and generate graphs [Dai et al., 2020, Li et al., 2018, Liao et al., 2019, You et al., 2018].
However, due to the often opaque deep learning procedures, these deep generative models are usually implicit, which hinders theoretical analysis to assess how close the generated samples are in their distributional properties to the graph distribution they are meant to be sampled from.
Learning instead parametric models from an explicit pre-specified probability distribution class would allow to use the learned parameters for model assessment.
However, parametric models may only capture a fraction of the graph features and have restricted modelling power.
Parameter estimation can be inconsistent [Shalizi and Rinaldo, 2013] even for well-specified models and may lead to wrong conclusions in model assessments.
Using instead deep generative models for graphs may surpass some of these issues by learning rich graph representations, but methods to assess the quality of such implicit graph generators are lacking.
In principle, nonparametric hypothesis tests can be useful to assess complex models, such as kernel-based tests procedures that utilise functions in a reproducing kernel Hilbert space (RKHS)
[Berlinet and Thomas, 2004].
When the models are described in the form of explicit probabilities, goodness-of-fit tests [Chwialkowski et al., 2016, Liu et al., 2016] may apply; however, goodness-of-fit testing procedures are generally not applicable for implicit models.
Instead, if a large set of samples from the target distribution is observed, one may generate samples from the implicit model and perform a two-sample test such as a maximum mean discrepancy (MMD) test
[Gretton et al., 2007] for model assessment [Jitkrittum et al., 2017, Xu and Matsuda, 2020, 2021].
[Figure]
Yet, in real-world applications, often only a single graph from the target distribution is observed [Bresler and Nagaraj, 2018, Reinert and Ross, 2019].
In this case, the MMD methods for model assessment via two-sample testing procedures cannot be used to assess implicit graph generators.
To our knowledge, beyond simple Monte Carlo tests with often poor power, no principled test is available for assessing the quality of implicit graph generators.
Figure.1 illustrates the task: a given graph generator G which is learned from a set of training samples can generate samples of user-defined size.
The task is to assess whether G can generate samples from the same distribution that generates the observed graph, for example, Padgett’s Florentine marriage network [Padgett and Ansell, 1993].
The samples in Figure.1 are generated by a Cross-Entropy Low-rank Logit (CELL) model [Rendsburg et al., 2020] trained on Padgett’s Florentine marriage network, but any graph generator could be used.
This paper makes three main contributions.
(1) We introduce Approximate graph Stein Statistics (AgraSSt) in Section 3, which opens up a principled way to understand implicit graph generators.
AgraSSt is a variant of a kernel Stein discrepancy based on an empirical Stein operator for conditional distributions of general random graph models.
The testing procedure is inspired by gKSS, a goodnessof-fit testing procedure for explicit exponential random graph models (ERGMs)
[Xu and Reinert, 2021].
(2) We provide theoretical guarantees for AgraSSt (Section 3).
(3) We propose interpretable model criticisms when there is a model misfit, and we identify representative synthetic sample batches when the model is a good fit (Section 4).
Further, in Section 2 we review gKSS.
We provide empirical results in Section 5 and a discussion in Section 6.
Proof details, more background, theoretical and empirical results, and implementation details are found in the Supplementary Information (SI).The code for the experiments is available at https://github.com/wenkaixl/agrasst .
Background: graph kernel Stein statistics
In [Xu and Reinert, 2021], a goodness-of-fit testing procedure called gKSS for exponential random graph models is introduced, which assumes that only a single network may be observed in the sample.
As AgraSSt is inspired by gKSS, and gKSS serves as comparison method when the observed network is known to be generated from an exponential random graph model (ERGM), we briefly review gKSS.
The notation introduced in this section is used throughout the paper.
ERGMs are extensively used for social network analysis [Wasserman and Faust, 1994, Holland and
Leinhardt, 1981, Frank and Strauss, 1986]; a special case are Bernoulli random graphs.
ERGMs model random graphs via a Gibbs measure with respect to a chosen set of network statistics, such as number of edges, 2-stars, and triangles.
Denote by G labn , the set of vertex-labeled graphs on n vertices, with N = n ( n − 1) / 2 possible undirected edges.
Encode x ∈ G labn by an ordered collection of { 0 , 1 } -valued variables x = ( x ( ij ) )
1 ≤ i<j ≤ n ∈ { 0 , 1 } N where x ( ij ) =
if and only if there is an edge between i and j .
We denote an (ordered) vertex-pair index s =
( i, j ) by s ∈ [ N ] := { 1 , . . .
, N } .
Let t 1 ( x ) count the number of edges in x and let t 2 ( x ) , . .
. , t k ( x ) count other small graphs on at most n vertices in x (possibly scaled; for details including a precise definition, Definition A.3, see
SI A.2).
For β = ( β 1 , . . .
, β k ) ⊤ ∈ R k and t ( x ) =
( t 1 ( x ) , . . .
, t k ( x ))
⊤ ∈
R
k
we say that X ∈ G lab n follows the exponential random graph model X ∼ ERGM( β, t ) if for ∀ x ∈ G labn ,
[Equation]
Here κ n ( β ) is a normalisation constant.
In this model, t ℓ , ℓ = 1 , . . .
, k , are sufficient statistics.
Parameter estimation ˆ β l for β l is only possible when the sufficient statistics are specified a priori; see
SI.B.1 for estimation details.
In modern graph learning procedures, e.g. deep generative learning, the sufficient statistics t l of Eq.(1) may not be obtained explicitly.
In [Reinert and Ross, 2019] the exponential random graph distribution in Eq.
(1) is characterised by a so-called Stein operator , as follows.
Let e s ∈ { 0 , 1 } N be a vector with 1 in coordinate s and 0 in all others; x ( s, 1) =
x + (1 − x s ) e s has the s -entry replaced of x by the value 1, and x ( s, 0) =
x − x s e s has the s -entry of x replaced by the value 0; moreover, x − s is the set of edge indicators with entry s removed.
For a function h : { 0 , 1 } N → R , let ∆ s h ( x ) =
h ( x ( s, 1) ) − h
( x ( s, 0) ) .
Set q X ( x ( s, 1) | x − s ) =
P ( X s = 1 | X − s = x − s ) .
Define the operator
[Equation]
Then under mild conditions Reinert and Ross [2019] show that if E p [ A β,t f ] = 0 for all smooth test functions f , then p must be the distribution of ERGM ( β, t ) .
Thus, this operator characterises
ERGM ( β, t ) .
For the derivation of AgraSSt it is of interest to see how this operator is obtained.
It is indeed the generator of a so-called Glauber Markov chain on G labn with transition probabilities
[Equation]
With the ERGM Stein operator in Eq.(2) and a rich-enough RKHS test function class H , Xu and
Reinert [2021] propose a graph kernel Stein statistics (gKSS) to perform goodness-of-fit testing on an explicit ERGM when a single network sample is observed.
With the summand components in Eq.(2), the Stein operator can be seen as taking expectation over vertex-pair variables S ∈ [ N ] with uniform probability P ( S = s ) ≡ N − 1 independently of x , namely
[Equation]
For a fixed graph x , gKSS is defined as
[Equation]
where the function f is chosen to best distinguish q from x .
For an RKHS H associated with kernel K , by the reproducing property of H , the squared version of gKSS admits a quadratic form representation gKSS 2 ( q ; x ) = ⟨ E S
[ A ( S ) q K
( x, · )] , E S
[ A ( S ) q K
( x, · )] ⟩ , which can be computed readily.
More background can be found in SI B.2.
AgraSSt: Approximate Graph Stein Statistic
An implicit graph generator may not admit a probability distribution in the form of Eq.(1).
However, the idea of constructing Stein operators based on Glauber dynamics using conditional probability distribution for ERGM serves as a motivation for us to propose Stein operators for conditional graph distributions, to facilitate an (approximate) characterisation for implicit random graph models.
3.1 Stein operators for conditional graph distributions
Let q ( x ) =
P ( X = x ) be any distribution with support G labn .
Let t ( x ) denote a statistic on graphs which takes on finitely many values k and let q k ( x ) =
P ( X = x
| t ( x ) = k ) .
We assume that q k ( x ) > 0 for all k under consideration.
For a generic outcome we write q t .
We introduce a Markov chain on G labn which transitions from x to x ( s, 1) with probability
[Equation]
and which transitions from x to x ( s, 0) with probability q ( x ( s, 0) | t ( x − s ))
= 1 − q t ( x ( s, 1) ); no other transitions occur.
Let
[Equation]
For an ERGM, t ( x ) could be taken as a sufficient vector of statistics 1 , but here we do not assume a parametric network model q ( x ) , and t ( x ) does not have to be sufficient statistics for q ( x ) .
Recall that an operator is a Stein operator for a distribution µ if its expectation under µ is zero.
The following result provides a theoretical foundation for AgraSSt and is proven in SI.A.
Lemma 3.1.
If q k ( x ) > 0 for all k , then A ( s ) q,t = k is a Stein operator for the conditional distribution of X given t ( X ) =
k , and (cid:80) s A ( s ) q,t
= k is a Stein operator for the conditional distribution of X given t ( X ) = k .
In particular, E q t = k
[ A ( s ) q,t = k ] = 0 .
Intuitively, if the distribution of ˜ X is close to that of X , then with ˜ Y t =
k denoting the corresponding random graph with distribution that of ˜ X given t ( ˜ X ) =
k, it should hold that E [ A ( s ) q,t = k
( ˜ Y t = k )]
≈ 0 .
In this way the Stein operator in Eq.(5) can be used to assess similarity between distributions.
3.2 Approximate Stein operators
For implicit models and graph generators G , the Stein operator A ( s ) q,t in Eq.(5) cannot be obtained without explicit knowledge of q t ( x ( s, 1) ) .
However, given a large number for samples from the graph generator G , the conditional edge probabilities q t ( x ( s, 1) ) can be estimated.
Here we denote by (cid:98) q t ( x ( s, 1) ) an estimate of q t ( x ( s, 1) ) ; some estimators will be suggested in Section 3.3.
AgraSSt performs model assessment using an operator which approximates the Stein operator A ( s ) q,t .
We define the approximate Stein operator for the conditional random graph by
[Equation]
The vertex-pair averaged approximate Stein operator is
[Equation]
3.3 Estimation with chosen statistics on graphs
Using the Stein operator for conditional graph distributions, we can obtain the approximate Stein operators in Eq.
(6) and Eq.
(7) for an implicit graph generator G by estimating q t ( x ( s, 1) ) .
Here t ( x ) are user-defined statistics.
In principle, any multivariate statistic t ( x ) can be used in this formalism.
However, estimating the conditional probabilities using relative frequencies can be computationally prohibitive when the graphs are very large and specific frequencies are rarely observed.
Instead, here we consider simple summary statistics, such as edge density, degree statistics or the number of neighbours connected to both vertices of s .
The estimation procedure is presented in Algorithm 1.
To estimate q t ( x ( s, 1) ) in Step 3 of Algorithm 1, a look-up table can be used: If t ( x ) is a possibly multivariate statistic with a discrete number of outcomes, generate many independent copies from the synthetic graph generator, and count n ( k, s ) , the number of times that vertex-pair s is present in the simulated graphs and t ( x − s )
=
k .
Consistency follows from the law of large numbers.
When using smoothing to estimate the conditional probability, consistency will depend on the smoothing method.
If the underlying graph has exchangeable edge indicators then q t ( x ( s, 1) ) does not depend on the choice of vertex-pair s .
Then we set set n ( k )
= (cid:80) s n ( k, s ) , and
N k = (cid:80) Li =1 (cid:80) s 1 ( t ( x − s )
= k ) ; if t = k we estimate (cid:98) q t ( x ( s, 1) ) by
[Equation]
t
[Equation] [Equation]
If k min , k max denote the minimum and maximum values of statistics from simulated graph samples, then for k outside this set, the lookup table estimator Eq.(8) is set to estimate ˆ g ( k ) = 0 .
If the underlying graph cannot be assumed to have exchangeable edge indicators or if the statistic t is high dimensional, then any particular n ( k, s ) may not be observed very often.
In such a situation we can learn ˆ g t ( s ; k ) using kernel ridge regression so that the conditional probabilities for similar t ( x − s ) are predicted in a smooth manner.
Estimating q
x s | t ( x − s ) ≤ k ) instead of q ( x s | t ( x − s )
= k ) may provide an alternative, smoother estimate for the conditional probabilities.
The next result shows that the approximate Stein operator achieves the Stein identity asymptotically.
For this result, which is proved in SI.A, we use the notation ∥
∆ f ∥ = sup s ∈ [ N ] ,x ∥ ∆ s f ( x ) ∥ .
Theorem 3.2.
Assume (cid:98) q t ( x ( s, 1) ) is a consistent estimator for q t ( x ( s, 1) ) as L → ∞ .
Then for any f such that || ∆ f || < ∞ we have E q [ A (cid:98) q,t f ( x )]
→ E q [ A q,t f ( x )] = 0 as L → ∞ .
Proofs and additional analysis are included in SI A. Proposition A.1 in SI A.1 addresses consistent estimation of (cid:98) q t and SI A.2 provides refined results for ERGMs, including a Gaussian approximation.
3. AgraSSt for implicit graph generators
The estimated conditional probabilities give an approximate Stein operator for Eq.(5).
With the appropriately defined Stein operator from an implicit model given in Eq.(6), we can define AgraSSt, a kernel-based statistic analogous to gKSS in Eq.(3), as
[Equation]
In SI.A we prove the following result for edge-exchangeable graphs , that is, graphs in which all permutations of the edge indicators have the same distribution.
Theorem 3.3.
If the graph is edge-exchangeable, then AgraSSt 2 ( (cid:98) q, t ; x ) is a consistent estimator of
[Equation]
Re-sampling Stein statistic A computationally efficient operator for large N can be derived via re-sampling B vertex-pairs s b , b = 1 , . . .
, B , from { 1 , . . .
, N } , chosen uniformly with replacement, independent of each other and of x .
This procedure gives rise to a randomised operator; this re-sampled operator is
[Equation]
where the expectation of (cid:98) A B
[Equation]
q,t f ( x ) with respect to re-sampling is
[Equation]
The corresponding re-sampled AgraSSt statistic is
[Equation]
AgraSSt
[Equation]
Similar to Eq.(9), the squared version of (cid:92) AgraSSt admits a representation in a quadratic form,
[Equation]
AgraSSt
[Equation] [Equation]
where (cid:98) h x ( s, s ′ )
= ⟨A ( s ) (cid:98) q,t K ( x, · ) ,
A ( s ′ ) (cid:98) q,t K ( · , x ) ⟩ H .
We note that the randomised operator obtained via re-sampling is a form of stochastic Stein discrepancy as introduced in Gorham et al.
For fixed x , under mild conditions the consistency of (cid:92) AgraSSt 2 ( (cid:98) q, t ; x ) as B → ∞ is ensured by the following normal approximation, which follows from Proposition 2 in Xu and Reinert [2021].
Proposition 3.4.
Assume that (cid:98)
h
x
( s, s ′ ) in (10) is bounded and that (cid:92)
AgraSSt( (cid:98) q, t ; x ) has non-zero variance σ 2 .
Let Z be a normal variable with mean AgraSSt( (cid:98) q, t ; x ) and variance σ 2 .
Then there exists an explicitly computable constant C > 0 such that for all 3 times continuously differentiable functions g with bounded derivatives up to order 3,
[Equation]
Applications of AgraSSt 4.1 Assessing graph generators
AgraSSt measures the distributional difference between the underlying distribution of an implicit graph generator G and an observed graph x , in order to assess the quality of the generator G .
The hypothesis testing procedure for the null hypothesis that the observed graph x comes from the same distribution that generates the samples, against the general alternative, is shown in Algorithm 2.
We emphasise two features of this procedure.
Firstly, for a given generator G , AgraSSt directly assesses the quality of the implicit model represented via samples from G .
Secondly, the generator G can be trained on the observed graph x , for example through a deep neural network generator.
By learning a deep neural network generator with training samples from the same distribution that generate x ,
AgraSSt can assess the quality of the training procedure, i.e. whether the deep neural network is capable of learning the desired distributions.
Additional details are discussed in SI.C.
4.2 Interpreting trained graph generators
If the procedure in Algorithm 2 rejects the null hypothesis, the generator may not be suitable for generating samples from the distribution that generates the one observed graph.
Hence, understanding where the misfit comes from can be very useful, especially for models trained from black-box deep neural networks.
AgraSSt provides an interpretable model criticism by comparing the learned (cid:98) q t ( x ( s, 1) ) with the underlying q t ( x ( s, 1) ) when available, such as in synthetic experiments from a specified ERGM.
Such an interpretation can be also useful to re-calibrate training procedures.
4.3 Identifying reliable graph samples
If the procedure in Algorithm 2 does not reject the null hypothesis, there is not enough evidence to reject the hypothesis that the generator is capable of generating graphs that resembles the observed graph.
If a generator G has passed this hurdle then it can be recommended for generating graph samples of the desired type.
AgraSSt can also be put to use for the task of sample batch selection.
In many scientific studies, only a small batch of representative graph samples may need to be generated for downstream tasks such as privacy-preserving methods where users only access a small number of graph data, or a randomised experimental design for community interaction.
To quantify the quality of sample batches via p -values : (1) we generate a sample batch of size m , say; (2) we perform the steps in Procedure from Algorithm 2; (3) we compute the p -value m − 1 (cid:80) mi
=1 1 ( τ > τ i ) , with τ as in step 3 and { τ 1 , . .
, τ m } as in step 5.
If the p -value is smaller than a pre-specified threshold, we generate another sample batch; otherwise we accept the current sample batch.
Empirical results
We first illustrate the performance of AgraSSt on synthetic data, where the null distribution is known and we have control of the set-up; in particular we can illustrate the use of AgraSSt for interpretable model criticism.
Then we show the performance of AgraSSt on a real-world data application to assess graph generators trained via various deep generative models.
5.1 Synthetic experiments
Only few competing approaches are available for our task and many of them are devised specifically for ERGMs.
Hence here we use an ERGM, namely the Edge-2Star-Triangle (E2ST) model with
[Equation]
(11) where E d ( x ) denotes the number of edges of x , S 2 ( x ) denotes the 2-Star statistics and T r ( x ) denotes the triangle statistics.
The parameter vector β = ( − 2 . 00 , 0 . 00 , 0 . 01) is chosen as the null model, while alternative models are constructed by perturbing the coefficient β 2 as in Yang et al.
[2018], Xu and Reinert [2021].
This particular type of ERGM is chosen because it is the currently most complex
ERGM for which a thorough theoretical analysis for parameter estimation is available, see Mukherjee and Xu [2013] 2 .
5.1.1 Related approaches for comparisons
To assess the performance of AgraSSt, we consider the following test statistics which are either tailored or modified to perform assessment for implicit graph generators:
• Deg is a degree-based statistics for goodness-of-fit test of exchangeable random graphs [Ouadah et al., 2020] based on the estimated variance of the degree distribution.
The statistics can be obtained from empirical degrees from samples generated from the implicit model.
• TV_deg denotes the Total-Variation (TV) distance between degree distributions.
Hunter et al.
[2008] proposes a simulation-based approach to construct graphical goodness-of-fit tests.
Xu and Reinert [2021] quantifies this approach using the total-variation distance between the distributions of chosen network statistics; see SI.D.4 for details.
• MDdeg is the Mahalanobis distance between degree distributions [Lospinoso and Snijders, 2019].
In the synthetic experiment where the parametric model is known (Eq.(11)), the coefficients β are estimated from generated samples for model assessment to provide our baseline approach, denoted by Param .
Details can be found in SI.B.1.
Knowing the explicit null model in the synthetic setting, we can also compute gKSS in Eq.(3), denoted by Exact as our benchmark.
The Weisfeiler-Lehman graph kernel [Shervashidze et al., 2011] with height parameter 3 is used for kernel-based approaches.
5.1.2 Simulation results
The rejection rates for various settings are shown in Figure.2.
As the null model is relatively sparse (edge density 11 .
2% ), the sparser alternatives are much harder to distinguish while the denser ones are
[Figure]
R e j e c t
i
o n R a t e
DegMDdegTV
[Figure]
deg
AgraSStParamExact
[Figure]
easier problems.
Figure.2(a) compares the AgraSSt procedure with the approaches from Section 5.1.1; it shows that AgraSSt performs competitively to the benchmark Exact , which is only available when the model is known explicitly, and outperforms other assessment procedures for implicit models.
TV_deg is slightly less powerful than AgraSSt but outperforms Deg .
For sparser alternatives (smaller β 2 ), MDdeg and Param have a much lower rejection rate and thus are less powerful.
In Figure.2(b), we compare the performance of AgraSSt using different estimation methods for (cid:98) q t ( x ( s, 1) ) ; the degree deg ( k ) of a vertex k is calculated excluding the vertex-pair s =
( i, j ) .
The estimation methods are
• Sum_deg : for s =
( i, j ) we set t =
deg ( i ) + deg ( j ) ; • Cum_deg : the cumulative distribution function of sum of degrees are used; • Bi_deg : for s =
( i, j ) the 2-dimensional vector t =
( deg ( i ) , deg ( j )) is used; • Edges : t is the edge density after removing vertex-pair s .
From the results, we see that Edges outperforms the other estimates, which echos the theoretical results shown in Theorem A.5 in SI, as the coefficient β for E2ST satisfies its assumptions.
Hence this is the statistics which is used in our real-world experiments, with an exception of SI.D.3.
We also see that using both vertex degrees as 2 d vector predicts substantially better than using predictors based on sum of degrees of two vertices.
In Figure.2(c), the comparison with re-sampling is shown.
With increase in re-sampling size, the power of AgraSSt increases.
5.2 Real-world applications on deep graph generators
We now assess the performances of a set of state-of-the-art deep generative models for graphs trained on ERGMs and the Karate Club network collected by Zachary [1977].
The Karate Club network has 34 vertices and 78 edges representing friendships.
Soon after the data collection the Karate Club separated into two factions.
This graph is a benchmark graph for community detection.
One would not expect this graph to be close to an G ( n, p ) 3 graph or to be well modelled by an ERGM.
5.2.1 Graph generation methods
The graph generation methods to which we apply AgraSSt are the following.
• GraphRNN
[You et al., 2018] is an architecture to generate graphs from learning two recurrent neural networks (RNN), one a vertex-level RNN and the other an edge-level RNN.
The procedure starts from a breadth-first-search for vertex ordering; two RNNs are trained from a sequential procedure.
• NetGAN
[Bojchevski et al., 2018] utilises an adversarial approach by training an interplay between a generator and a discriminator neural network on graph data.
• CELL [Rendsburg et al., 2020] improves on the NetGAN idea by solving a low-rank approximation problem based on a cross-entropy objective.
• MC is the standard Monte-Carlo network sampling in the ergm suite in R and is used as a baseline when the simulated network is known to follow the model in Eq.(1); q ( x ) needs to be known.
5.2.2 Generator assessment results
[Table]
We first train the generative models with samples from ERGMs to assess their ability to generate ERGMs.
The test results are shown in Table.1.
From the result, we see that for the “reliable” MC generator all the assessment statistics presented have well-controlled type-I error.
Samples generated from CELL deviate not too far from the test level, indicating a good generative model for ERGMs.
NetGAN and
GraphRNN both encounter a high rejection rate, implying that the generated samples that are not close to the training E2ST model.
From the density based AgraSSt , taking a G ( n, q ) model, we can interpret the model misfit by checking the estimated (cid:98) q .
For the true E2ST model used to generate training samples, we have = 0 .
112 , while CELL has (cid:98) q = 0 .
which is close to the null.
GraphRNN estimates (cid:98) q =
.which is substantially higher than the null.
Although GraphRNN can be powerful in learning local patterns and structures for neighbourhoods [You et al., 2018], here it does not take the overall density sufficiently into account.
Due to its limited “look back” and absence of “look forward” on the ordered vertex set during training, the over-generation of edges may have caused this significant difference for learning ERGMs.
NetGAN, on the other hand, produces a close estimate (cid:98) q = 0 .
However, counting triangles, it only has on average 12 . 6 triangles , which is far less from the null with expected number of triangles 46 . 3 .
NetGAN, due to its random walk adversarial procedure, may not be effective in learning such clustered patterns.
5.2.3 Case study: Karate Club network
[Table]
Next, we assess the performances of these generative models by training on the Karate
Club network [Zachary, 1977].
The pvalues for different testing procedures are shown in Table.2.
From the results, we see that AgraSSt rejects samples generated from both GraphRNN and NetGAN trained with the Karate Club network.
Although the edge densities generated from the trained GraphRNN (edge density 15 .
3% ) and NetGAN (edge density 13 .
4% ) are comparable with the Karate Club edge density of 13 .
9% , visual inspection indicates that both GraphRNN and NetGAN samples exhibit a single large component rather than two fairly separated communities in the Karate Club network.
This difference is picked up by AgraSSt, Deg and TV_deg, which all reject both models.
On the other hand, CELL generates samples that are not rejected by all tests at significance level α = 0 .
In Figure.5 in the SI, the
Karate Club network is shown in Figure.5(a).
Samples from GraphRNN, NetGAN and CELL are shown in Figure.5(b), 5(c) and 5(d) respectively.
SI D includes additional results, including on efficiency, and visualisations.
A second case study — the Florentine marriage network from Padgett and Ansell [1993] — is presented in SI D.2; additional visualisations of the reliable sample batch selection procedure described in Section 4.3 are also given.
Discussions and future directions
In this paper, we propose AgraSSt, a unique general purpose model assessment and criticism procedure for implicit random graph models.
As it is based on a kernel Stein statistic, we are able to give theoretical guarantees.
AgraSSt not only solves an important problem but also opens up a whole set of follow-up research problems of which we list a few here.
(i).
Currently AgraSSt is only applied to undirected and unweighted graphs.
Extensions to more general graphs as well as to time series of graphs will be interesting to explore in follow-up work.
(ii).
AgraSSt could be also helpful to improve design and training of deep graph generative models, e.g. by regularising graph features if there is a misfit.
(iii).
ERGMs allow for exogenous features to be included in the sufficient statistics.
AgraSSt can be based on a variety of statistics t ( x ) ; further examples are found in SI D.3.
It would also be possible to incorporate exogenous features in the statistics t ( x ) in AgraSSt, for example using ideas from graph attention networks [Veliˇckovi´c et al., 2018].
Exploring this idea in more detail will be another topic of further research.
As AgraSSt depends on the chosen summary statistic t ( x ) , results have to be interpreted with regards to the respective conditional distributions.
Also multiple tests will have the p -values to be adjusted to avoid misinterpretation of tests, which could have serious consequences for example in the area of personal health.
Investigating the effect of choice of t ( x ) will be part of future work.
Acknowledgements
The authors thank the anonymous reviewers for their constructive comments and suggestions to improve the paper.
G.R. and W.X. acknowledges the support from EPSRC grant EP/T018445/1.
G.R is also supported in part by EPSRC grants EP/W037211/1, EP/V056883/1, and EP/R018472/1.
Checklist
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Section(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section(b) Did you include complete proofs of all theoretical results?
[Yes] SI.A in Supplementary 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code attached as a folder in the Supplementary (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Model details are included in the code.
Additional implementation details are included in SID.6.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Random seed are used for experiment and included in the code.
See SID.3 for error bars for graphical tests.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See SID.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section(b)
Did you mention the license of the assets?
[N/A] Datasets and codes are open source.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] SID (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proofs and additional theoretical results A.1 Proofs
Here we usually omit the = k suffix in A ( s ) q,t = k and write instead A ( s ) q,t for easier reading.
The operator
A ( s ) q,t is understood as having a fixed value t ( x s ) of t observed.
Similarly we often write E q,t to abbreviate E q,t
= k , with the same understanding.
Proof of Lemma 3.1 For convenience we repeat the statement of the lemma here.
Lemma 3.
If q k ( x ) > 0 for all k , then A ( s ) q,t = k is a Stein operator for the conditional distribution of
X given t ( X ) =
k , and (cid:80) s A ( s ) q,t
= k is a Stein operator for the conditional distribution of X given t ( X ) = k .
Proof.
In order to show the assertion we prove that for E q,t = k denoting the conditional distribution of
X given t ( X ) =
k , the expectation E q,t = k
[ A ( s ) q,t f ] vanishes for all functions for which the expectation exists.
Let f be such a function.
We have
[Equation]
Thus,
E
[Equation]
Proof of Theorem 3.2 For convenience we repeat the theorem here.
Theorem
[Equation]
2 Assume that (cid:98) q t
( x ( s, 1) ) is a consistent estimator for q t ( x ( s, 1) ) as L → ∞ .
Then for any function f such that || ∆ f || < ∞ we have E q [ A (cid:98) q,t f ( x )]
→ E q [ A q,t f ( x )] = 0 as L → ∞ .
Proof.
We recall the notation that Equation (4).
We have that
[Equation]
so that
[Equation]
( x − s ))
∆ s f ( x ) .
Hence |A (cid:98) q,t f ( x
[Equation]
− A
[Equation]
t f
[Equation]
x ) | ≤ || ∆
[Equation]
||
[Equation]
N (cid:88) s ∈ [ N ] | (cid:98) q
( x ( s, 1) | t ( x
[Equation]
s ))
− q ( x ( s, 1) | t ( x − s ))
| .
Thus, if for all s ∈ [ N ] , as L → ∞ we have (cid:98) q
( x ( s, 1) | t ( x − s )) − q ( x ( s, 1) | t ( x − s ))
→ 0 for all s then so does |A (cid:98) q,t f ( x ) −A q,t f ( x ) | .
The assertion follows from the assumption that (cid:98) q t ( x ( s, 1) ) is a consistent estimator for q t ( x ( s, 1) ) as L → ∞ .
Proof of Theorem 3.3 For convenience we repeat the statement of the theorem here.
Recall that a random graph model is edge-exchangeable if its edge indicator variables are finitely exchangeable.
Often we just write edge-exchangeable graph .
Theorem 3.3 If the graph is edge-exchangeable, then AgraSSt 2 ( (cid:98) q, t ; x ) is a consistent estimator of
[Equation]
For easier tractability the proof is organised in two steps.
First, Proposition A.1 shows that in an edge-exchangeable random graph model, g k given in
Equation (8) is a consistent estimator for q ( x ( s, 1) | t ( x − s )) as NL → ∞ .
Theorem A.2 uses these results to obtain a concentration bound for A (cid:98) q from which then
Theorem 3.3 follows.
Moreover theoretical guarantees for fixed L which depend on the model are given.
As the graph generator can generate as large a number L of graphs as desired, these theoretical results can be used to determine L which result in theoretical guarantees on deviations from the mean.
Proposition A.1.
Suppose that X 1 , . . .
, X L are i.i.d.
copies of the adjacency matrix of an edgeexchangeable random graph model.
Let s =
( i, j ) be a fixed vertex-pair.
For l = 1 , . . .
L and for a graph X l let t ( s ) l denote a possibly multivariate statistic which is evaluated on the collection of indicator variables in X l except X s,l .
For a possible t ( s ) l outcome k , let p ( k ) =
P ( t ( s ) l = k ) and let k be such that p ( k ) ̸
= 0 .
Set
[Equation]
); let n ( k, s )
=
L (cid:88)
[Equation]
X s,l 1 ( t ( s ) l = k ) and n ( k ( s ) ) =
L (cid:88) l = 1 ( t ( s ) l = k ); n ( k ) = (cid:88) s ∈ [ N ] n ( k, s ) and
N k = (cid:88) s ∈ [ N ] n ( k ( s ) ); and set g ( k )
=
n ( k ) N k 1 ( N k ≥ 1) .
Then g ( k ) → p in probability as NL → ∞ .
In particular, for all ϵ > 0 ,
[Equation]
Proof.
Due to the exchangeability of the edges we have, with s denoting a generic edge,
E ( n ( k ))
=
N E ( n ( k, s ))
=
NLp (1; k ) p ( k ) ,
[Equation]
as well as
[Equation]
To show convergence in probability, let ϵ > 0 .
Then
[Equation] [Equation]
k
[Equation]
(cid:19)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
ϵ (cid:21) .
Note that n ( k ) 1 [ n ( k ) ≥ 1] = n ( k ) .
By Chebychev’s inequality,
[Equation]
and
[Equation]
Summing the contributions completes the proof
[Equation]
Proposition A.1 shows that in edge-exchangeable graphs, ˆ g ( k ) consistently estimates q ( x ( s, 1) )
=
q ( x ( s, 1) | t ( x − s ) = k ) .
In an expanded version of Theorem 3.3 we show that the approximate Stein operator from Eq.(7),
[Equation]
with
[Equation]
is a consistent estimator of
A
[Equation]
We recall
[Equation]
with
[Equation]
We state the expanded version of Theorem 3.3 here.
Theorem A.2.
If the graph is edge-exchangeable then for any test function f for which the Stein operator A q,t f is well defined, and for all ϵ >
[Equation]
Moreover, AgraSSt 2 (ˆ q, t, x ) is a consistent estimator of
[Equation]
Proof.
We have that
[Equation]
and
[Equation]
with
[Equation]
so that
[Equation]
) f
( x ( s, 0) ) } =
N (cid:88) s ∈ [ N ] ( g ( k ) − q ( x ( s ) | t ( x − s )))
{ f ( x ( s, 1) ) − f ( x ( s, 0) ) } =
N (cid:88) s ∈ [ N ] ( g ( k ) − q ( x ( s ) | t ( x − s )))
∆ s f ( x ) .
Hence
[Equation]
With Proposition A.1 and using the edge-exchangeability
[Equation]
The fact that taking the sup over functions in the Hilbert space H does not spoil the convergence follows from the closed form representation of the sup of AgraSSt 2 , see for example Equation (11) in [Xu and Reinert, 2021].
We have that
[Equation]
where
[Equation]
Hence
[Equation]
AgraSSt
[Equation]
and the first part gives the desired convergence as L → ∞ .
A.2 Gaussian approximation for AgraSSt in ERGMs
If the observed graph x is a realisation of an ERGM then results from Xu and Reinert [2021] can be leveraged to obtain finer theoretical results.
First we detail the scaling for exponential random graph models which is used in the theoretical results which follow.
For a graph H on at most n vertices V ( H ) denote the vertex set, and for x ∈ { 0 , 1 } N , denote by t ( H, x ) the number of edgepreserving injections from V ( H ) to V ( x ) ; an injection σ preserves edges if for all edges vw of H with σ ( v ) <
σ ( w ) , x σ ( v ) σ ( w ) = 1 .
For v H =
| V ( H ) | ≥ 3 set t H ( x ) =
[Equation]
If H = H 1 is a single edge, then t H ( x ) is twice the number of edges of x .
In the exponent this scaling of counts matches Definition 1 in Bhamidi et al.
[2011] and Sections 3 and 4 of Chatterjee and Diaconis [2013].
An ERGM for the collection x ∈ { 0 , 1 } N can be defined as follows.
Definition A.3 ( Definition 1.5 in Reinert and Ross [2019]) .
Fix n ∈ N and k ∈ N .
Let H 1 be a single edge and for l = 2 , . . .
, k let H l be a connected graph on at most n vertices; set t l ( x ) =
t H l ( x ) .
For β = ( β 1 , . . .
, β k ) ⊤ ∈ R k and t ( x ) =
( t 1 ( x ) , . . .
, t k ( x ))
⊤ ∈ R k X ∈ G labn follows the exponential random graph model X ∼ ERGM( β, t ) if for ∀ x ∈ G labn ,
[Equation]
Here κ n ( β ) is a normalisation constant.
In particular, under suitable conditions, the ERGM Glauber Stein operator is close to the G ( n, p ) Stein operator.
This result is already shown in Reinert and Ross [2019], Theorem 1.7, with details provided in the proof of Theorem 1 in Xu and Reinert [2021].
To give the result, a technical assumption is required, which originates in Chatterjee and Diaconis [2013], and is required in Reinert and Ross [2019].
For a ∈ [0 , 1] , define the following functions [Bhamidi et al., 2011, Eldan and Gross, 2018], with the notation in Definition A.3 for ERGM ( β, t ) :
[Equation]
where e l is the number of edges in H l .
Assumption 1 .
(1) 12 | Φ | ′ (1) < 1 .
(2) ∃ a ∗ ∈ [0 , 1] that solves the equation φ ( a ∗ )
= a ∗ .
The value a ∗ will be the edge probability in the approximating Bernoulli random graph, ER( a ∗ ) .
The following result holds.
Proposition A.4.
Let q ( x ) =
ERGM( β, t ) satisfy Assumption 1 and let ˜ q denote the distribution of
ER ( a ∗ ) .
Then there is an explicit constant C = C ( β, t, K ) such that for all ϵ > 0 ,
[Equation]
Moreover, for f ∈ H equipped with kernel K , let f ∗ x ( · ) =
[Equation]
K
[Equation]
x
[Equation]
∥ H .
Then there is an explicit constant C = C ( β, t, K ) such that for all ϵ > 0 ,
[Equation]
Proof.
The assertion follows immediately from the proof of Theorem 1 in Xu and Reinert [2021].
The approximation with a Bernoulli random graph is useful as for a Bernoulli random graphs a normal approximation for its gKSS is available in Xu and Reinert [2021], under suitable assumptions.
Assumption 2 .
Let H be the RKHS associated with the kernel K : { 0 , 1 } N × { 0 , 1 } N → R and for s ∈ [ N ] let H s be the RKHS associated with the kernel l s : { 0 , 1 } × { 0 , 1 } → R .
Then
i)
H is a tensor product RKHS, H = ⊗ s ∈ [ n ] H s ; ii) k is a product kernel, k ( x, y ) =
⊗ s ∈ [ N ] l s ( x s , y s ) ; iii) ⟨ l s ( x s , · ) , l s ( x s , · ) ⟩ H s = 1 ; iv) l s (1 , · ) − l s (0 , · ) ̸ = 0 for all s ∈ [ N ] .
These assumptions are satisfied for example for the suitably standardised Gaussian kernel K ( x, y ) =
exp {− 1 σ 2 (cid:80) s ∈
[ N ] ( x s − y s ) 2 } .
Letting || · || 1 denote L 1 -distance, and L denote the law of a random variable, Xu and Reinert [2021] show the following normal approximation.
Theorem A.5 (Theorem 2 in Xu and Reinert [2021]) .
Let Y have the distribution ˜ q of a Bernoulli random graph ER ( a ∗ ) as in Proposition A.4.
Assume that the conditions i) - iv) in Assumptionhold.
Let µ = E
[gKSS 2 (˜ q, Y )] and σ 2 = V ar [gKSS 2 (˜ q, Y )] .
Set W = 1 σ (gKSS 2 (˜ q, Y )] − µ ) and let Z denote a standard normal variable, Then there is an explicit constant C = C ( a ∗ , l s , s ∈ [ N ]) such that
[Equation]
Thus a normal approximation for the approximating gKSS can then be used to assess the theoretical behaviour of AgraSSt as follows.
Corollary A.6.
Let the assumptions Proposition A.4 and Theorem A.5 be satisfied.
With the notation of Theorem A.5, assume that the RKHS kernel K is such that the right hand side of Proposition A. is o ( n ) .
Then 1 σ (AgraSSt( (cid:101) q ( x ( s ) | t ( x − s )))
− µ ) is approximately standard normally distributed as
N
→ ∞ .
Proof.
For all ϵ > 0 ,
[Equation]
The first summand tends to 0 as N → ∞ due to Theorem 3.2 and the second summand tends todue to Proposition A.4.
That gKSS( a ∗ ) is approximately normally distributed with the appropriate scaling follows from Theorem A.5.
The theoretical behaviour of the subsampling version (cid:92) AgraSSt( (cid:101) q ( x ( s ) | t ( x − s ))) is addressed in
Proposition 3.4.
A detailed examination of the choice of kernel K such that the assumptions of
Corollary A.6 are satisfied is left for future work.
B Additional background
In this section, we present additional background to complement the discussions in the main text.
B.1 Parameter estimation for random graphs
Estimating parameters for parametric models is possible only when the parametric family is explicitly specified .
For instance, in the synthetic example for E2ST model shown in Section 5.1, ˆ β l can be estimated for β l since the edge, 2Star and triangle statistics are specified.
There are various approaches for parameter estimation.
Maximum likelihood Maximum likelihood is a popular approach for parameter estimation in random graph models.
A complication arises because its probability mass function from Eq.(1),
[Equation]
involves a normalisation constant κ n ( β )
= (cid:80) x exp { (cid:80) kl
=1 β l t l ( x ) } which is generally intractable and needs to be estimated for performing MLE.
For this task, Markov chain Monte-Carlo maximum likelihood estimation (MCMCMLE) for ERGM has been developed, see Snijders [2002].
When the network size is large, accurate estimation for the normalised κ n ( β ) requires large amount of
Monte-Carlo samples and is hence computationally expensive.
Maximum pseudo-likelihood estimator To alleviate the problem associated with the normalising constant, Maximum Pseudo-likelihood Estimation (MPLE)
[Besag, 1975] has been suggested for
ERGMs, see Strauss and Ikeda [1990] and also Schmid and Desmarais [2017].
MPLE factorises the conditional edge probability to approximate the exact likelihood,
[Equation]
For ERGMs the conditional distribution q ( x s | x − s ) does not involve the normalising constant and can hence be computed more efficiently than the MLE.
However, in general the MPLS is not consistent for ERGMs as the edge indicator variables are generally not independent.
The consistency of MPLE for Boltzmann machines is shown in Hyvärinen [2006].
A thorough comparison of MCMCMLE and
MPLE estimation in ERGMs can be found in Van Duijn et al.
Contrastive divergence Estimation based on contrastive divergence (CD)
[Hinton, 2002] has also been used for ERGM estimation [Hunter and Handcock, 2006].
Contrastive divergence runs a small number of Markov chains simultaneously for T steps and estimates the gradient based on the differences between initial values and values after T steps in order to find a maximum.
Convergence results for exponential family models are shown in Jiang et al.
CD can provide a useful balance between computationally expensive but accurate MCMCMLE and fast but inconsistent
MPLE.
B.2 Kernel Stein discrepancies and kernel-based nonparametric hypothesis testing
The task of hypothesis testing involves the comparison of distributions p and q that are significantly different with respect to the size of the test, denoted by α .
In nonparametric tests, the distributions are not assumed to be in any parametric families and test statistics are often based on ranking of observations.
In contrast, parametric tests, such as a Student t-test or a normality test, assume a pre-defined parametric family to be tested against and usually employ a particular summary statistics such as means or standard deviations.
Recent advances in nonparametric test procedures introduce
RKHS functions which can be rich enough to distinguish distributions whenever they differ.
Below we detail two instances which are relevant for the main paper.
We start with a terse review of kernel Stein discrepancy (KSD) for continuous distributions developed to compare and test distributions [Gorham and Mackey, 2015, Ley et al., 2017].
Let q be a smooth probability density on R d that vanishes at the boundary.
The operator A q : ( R d →
R d ) → ( R d → R ) is called a Stein operator if the following Stein identity holds: E q [ A q f ] = 0 , where f : R d → R d is a bounded smooth function from a set F which is often called a Stein class of functions.
Many choices of Stein operators and Stein classes are possible, see for example Mijoule et al.
A Stein operator A q for q a continuous differentiable density on R d is defined for differentiable real-valued functions f on R d by
[Equation]
see [Chwialkowski et al., 2016, Liu et al., 2016].
This Stein operator is also called Langevin Stein operator
[Barp et al., 2019].
That the Stein operator A q only requires the derivatives of log q and thus does not involve computing the normalisation constant of q renders it particularly useful for dealing with unnormalised models [Hyvärinen, 2005].
For our purposes it is convenient to take as
Stein class F = B 1 ( H ) , the unit ball of a large enough RKHS with bounded kernel K .
The kernel
Stein discrepancy (KSD) between two densities p and q based on A q is defined as
[Equation]
Under mild regularity conditions, using the Langevin Stein operator, KSD( p ∥ q, H ) ≥ 0 and
KSD( p ∥ q, H ) =
if and only if p = q
[Chwialkowski et al., 2016], in which case KSD is a proper discrepancy measure between probability densities.
The KSD in Eq.(16) can be used to test the model goodness-of-fit as follows.
One can show that
KSD 2 ( p ∥ q, H ) =
E x, ˜ x ∼ p
[ h q ( x, ˜ x )] , where x and ˜ x are independent random variables with density p and h q ( x, ˜ x ) is given in explicit form which does not involve p ,
[Equation]
Given a set of samples { x 1 , . . .
, x n } from an unknown density p on R d , to test whether p = q the statistic KSD 2 ( p ∥ q, H ) can be empirically estimated by independent samples from p using a U - or Vstatistic.
The critical value is determined by bootstrap based on weighted chi-square approximations for U - or V -statistics.
For goodness-of-fit tests of discrete distributions when i.i.d.
samples are available, a kernel discrete Stein discrepancy (KDSD) has been proposed in Yang et al.
Goodness-of-fit Testing assesses the null hypothesis H 0 : p = q against the general alternative
H 1 : p ̸
= q when the target distribution q is explicitly specified.
Given sample(s) from the unknown distribution p and an explicit density q , H 0 is assessed using a chosen test statistic, usually a discrepancy measure, D ( q ∥ p ) , between p and q , which can be estimated empirically.
Kernel-based hypothesis tests on goodness-of-fit for continuous distributions q use the kernel Stein discrepancy (KSD) in Section B.2 as the test statistic.
Given samples x 1 , . . .
, x n from the unknown density p ,
KSD 2 ( p ∥ q, H ) in Eq.(16) is estimated via the V -statistic
[Equation]
with h q ( x i , x j ) = ⟨A q K ( x i , · ) , A q K ( x j , · ) ⟩ H from Eq.(17).
The null distribution of this test statistic involves integral operators that are not available in closed form; often it is simulated using a wildbootstrap procedure [Chwialkowski et al., 2014].
With the (simulated) null distribution, the critical value of the test can be estimated to decide whether the null hypothesis is rejected at test level α .
In this way, a general method for nonparametric testing of goodness-of-fit on R d is obtained, which is applicable even for models with an intractable normalising constant.
Two-sample Testing assesses whether two sets of samples are drawn from the same distribution; instead of q being available in density form as in the goodness-of-fit setting, q is only accessible through samples.
Maximum mean embedding (MMD) test are often used for this two-sample problem [Gretton et al., 2007].
These tests are based on the kernel mean embedding of a distribution,
[Equation]
[Figure] [Figure]
whenever the function µ p exist.
Similar to KSD, MMD takes the supremum over unit ball RKHS functions;
[Equation]
With samples x 1 , . . .
x m ∼ p and y 1 , . . .
, y n ∼ q , MMD can be estimated empirically via U -statistics, (cid:92)
[Equation]
In such kernel-based two-sample tests, the null distribution can be obtained via a permutation procedure [Gretton et al., 2007]; this procedure can be more robust compared to a wild-bootstrap procedure, especially when the kernels need to be optimised [Gretton et al., 2012, Jitkrittum et al., 2016, Liu et al., 2020, 2021].
The two-sample procedure can also be applied to verify model assumptions when the model is not directly accessible through its distribution but through generated samples.
Such a strategy has been considered as benchmark testing procedure in various studies for goodness-of-fit tests [Jitkrittum et al., 2017, Xu and Matsuda, 2020, 2021].
Despite lower test power compared to the corresponding state-ofthe-art KSD-based tests and higher computational cost due to additional empirical estimation for the distribution q , these MMD-based tests are competitive with a simpler derivation in complicated testing scenarios [Xu and Matsuda, 2020, 2021], and they can outperform non-kernel based goodness-of-fit tests as discussed in Xu and Matsuda [2020].
C Visual illustrations of the assessment procedures
While AgraSSt is illustrated in Figure.1, we provide an additional visualisation emphasising different tasks for which AgraSSt can be applied.
In Section 4.1 in the main text, we mentioned two features of our proposed AgraSSt procedure:
Regardless of the learning or training procedures (masked in grey in in Figure.3), AgraSSt can test a given generator G that is only accessible through its generated samples as shown in Figure.3.
In this setting, we do not need to know how the generator G is obtained; the focus is the assessment of a particular generator G itself.
Moreover, we are also interested in understanding the quality and capability of training procedures of (deep) generative models.
As illustrated in Figure.4, a generator G is trained on samples from the same distribution as the input graph, e
g. ERGMs.
The focus in this setting is to assess the training procedure of the generative model.
(The samples generated are masked in grey.)
For instance, for G trained from the Florentine marriage network
[Padgett and Ansell, 1993], we may like to assess whether the generative model can be trained to generate graphs that resemble the Florentine marriage network.
D Additional experimental results and discussions D.1 Generating reliable samples
To illustrate how AgraSSt can be used to select sample batches, Figure.5 shows three sample batches of size 8 for the Karate club network of Zachary [1977], including the corresponding p -values for the displayed sample batches.
Here we would expect to detect some community structure in the networks; visually, only the sample batch from CELL captures this feature at least to some extent and has p -value which would not lead to rejection at the 5% level.
This finding chimes with the results from Table 2; AgraSSt rejects both GraphRNN and NetGAN as synthetic data generators, but does not reject CELL.
D.2 Additional case study: Padgett’s Florentine network
Padgett’s Florentine network [Padgett and Ansell, 1993]. has 16 vertices and 20 edges; in Xu and
Reinert [2021] the hypothesis that it is an instance of a G ( n, p ) model could not be rejected.
[Table]
The p -values for different tests are shown in Table.3.
The Florentine marriage network has edge density q = 0 .
167 , while the trained CELL has (cid:98) q = 0 . 165 which is a close approximation.
GraphRNN generates graphs with higher edge density (cid:98) q = 0 .
NetGAN generate samples with (cid:98) q = 0 .
176 , not too different from the null, which is not rejected at α = 0 .
Sample batch selection With CELL being deemed a good generator for the Florentine marriage network, we generate a sample batch of size 30 and check the sample quality.
Most sample batches produce a p -value above α = 0 .
until the 8th batch, which has p -value 0 .
03 <
α .
AgraSSt would recommend not taking this batch.
A visual illustration is shown in Figure.6.
To investigate these batches, we note that the Florentine marriage network has 3 triangles, while the batch being rejected has a significantly lower average number of triangles, namely 1 . 2 .
Despite a well estimated edge density, this batch produces a low p -value.
This batch, identified by AgraSSt as less reliable, may not be very suitable for downstream tasks and it may be better to generate another batch instead.
In contrast, the batch with p -value 0 .
75 has 2 .
28 triangles on average while the batch with p -value 0 .
37 has 2 .
04 triangles on average; these averages are closer to the observed number of triangles.
D.3 Experiments with other network statistics
AgraSSt can incorporate any user-defined network statistics.
Table 4 and Table 5 show additional results in the settings of Figure 2(b) and Table 1, respectively.
As AgraSSt network statistics t ( x − s ) , we introduce D3, which for s =
( i, j ) is the multivariate statistics t ( x − s )
= ( edges (( i, j )) , deg ( i ) , deg ( j )) , and we introduce Tri, which is the number of common neighbours of i and j ; we use the same notation as in Section 5.1.2, calculating the degree and the edges with ( i, j ) removed.
The edge based AgraSSt from the main text is added in grey for comparison.
In the Florentine network example, D3 has p -values 0.04 for GraphRNN, 0.11 for NetGAN, and 0.
for CELL.
Tri has p -values 0.02 for GraphRNN, 0.01 for NetGAN, and 0.12 for CELL.
Overall, the results are mainly comparable to using AgraSSt based on the number of edges, although Tri rejects
[Figure]
[Table] [Table]
NetGAN for the Florentine marriage network, thus picking up on NetGAN struggling to reproduce local clustering.
D.4 Additional discussions on distance-based test statistics
A classical approach for goodness-of-fit testing in ERGMs is the graphical test by Hunter et al.
The idea is to simulate sample graphs under the null distribution statistics and create box plots of some relevant network statistics; add to these plots the network statistics in the observed network, as a solid line for comparison, as illustrated in Figure.7.
The box plot is used to check whether the observed network is “very different” from the simulated null samples.
This graphical test procedure can be translated into Monte Carlo tests.
It is natural to adapt such procedure to implicit models from which samples can be obtained.
Figure.7 plots standard network statistics from Hunter et al.
[2008] for samples from a fitted G ( n, p ) generator (ER Approximate) and a learned GraphRNN generator of the Florentine marriage network.
The bold black line indicates the distribution of statistics for the
Florentine marriage network.
The distribution of network statistics is then quantified via Total Variation (TV) distance [Xu and
Reinert, 2021], based on which goodness-of-fit testing with p -values can be conducted.
We find that while the fitted ER generator shows a reasonable fit for all summary statistics, the GraphRNN generator does not match the Florentine marriage network very well for dyad-wise shared partners and the triad census.
D.5 Efficiency results
Table.6 presents the computational runtime (RT) and the test construction time (CT) for AgraSSt and its comparison methods from Section 5.1.1 with the simulation setup as in Section 5.1.2.
As a measure of accuracy, the variance (Var) of the simulated (or estimated) test statistics under the null distribution is also included.
The parameter estimation in Param depends on a computationally efficient method which is based on MPLE [Schmid and Desmarais, 2017] in Eq.(14).
AgraSSt takes longer to compute mainly due to the computation of graph kernels; here we use a Weisfeiler-Lehman kernel [Shervashidze et al., 2011].
We note that for implicit models, the estimation step in AgraSSt relies on generating samples from the model so that the the computational advantage 4 of the Stein based test over graphical goodnessof-fit tests 5 reduces compared to gKSS from Xu and Reinert [2021].
MDdeg is computationally expensive due to the estimation of an inverse covariance matrix.
While providing fast computation and estimation, Deg and Param sacrifice test power through a large variance of the test statistics.
Estimating the full degree distribution, the total variation distance method TV_deg , based only on
[Table]
a
[Table]
l l l l l l l l l l l l l l
[Table]
l l l l l l l l l
[Table]
l l l l l l l l l l l l l l l l l
[Table]
l l l
[Table]
l l
[Table]
l l l l
[Table]
l l l l l l l l l l l l l l
[Table]
an edge l l l l l l l l l l l l l l l l l l
[Table]
l l
[Table]
l l l
[Table]
l
[Table]
[Table]
l l l l l l l l l l l l l l l l l l l l l l l l l l l
[Table]
partners l og − odd s f o r an edge
l l l l l l l l l l l l l l l l l l    8 − 2 .− 1 .−
.degree l og − odd s f
o r a node l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l 0 1 2 3 4 5−−− dyad−wise shared partners l og − odd s f o r an edge
l l l l l l l l l l l l l l l l l l l l l
[Figure]
−
[Figure]
triad census l og − odd s f o r a t r i ad l l l l l l l l l l l l −   model statistics l og − odd s f o r t he s t a t i s t
i c l l l
[Figure]
degrees, is competitive with AgraSSt; we recall that in our simulation results from Section 5.1.
TV_deg was less powerful than AgraSSt.
Here MDdeg is outperformed by the other test statistics.
[Table]
D.6 Additional implementation details
For GraphRNN, we use batch size 128, 1000 epochs for training, 100 for testing, and learning rate 0.003.
For CELL, we use learning rate 0.01, and weight decay 10 − 7 .
For NetGAN, we use batch size 128, 50 epochs, generator size and discriminator size both 128, and learning rate 0.0003.
We note that training NetGAN [Bojchevski et al., 2018] with the Florentine network and with the
Karate Club network may encounter some generator instability and hence early stopping can be useful.
Without early stopping, the training loss for the generator increases during training, although it should be decreasing.
Figure.8 shows the training loss on the generator in NetGAN as well as on the critic (or discriminator) in NetGAN.
Figure.8 plots the loss every 200 training epochs.
We can see from Figure.8(a) that the generator loss starts to be unstable and then increases after 50 points, i.e.
10,000 epochs.
Hence we use only 10,000 epochs for training.
[Figure]