A Kernelised Stein Statistic for Assessing Implicit Generative Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Synthetic data generation has become a key ingredient for training machine learningprocedures, addressing tasks such as data augmentation, analysing privacy-sensitivedata, or visualising representative samples.
Assessing the quality of such syntheticdata generators hence has to be addressed.
As (deep) generative models for synthetic data often do not admit explicit probability distributions, classical statisticalprocedures for assessing model goodness-of-fit may not be applicable.
In thispaper, we propose a principled procedure to assess the quality of a synthetic datagenerator.
The procedure is a kernelised Stein discrepancy (KSD)-type test whichis based on a non-parametric Stein operator for the synthetic data generator ofinterest.
This operator is estimated from samples which are obtained from thesynthetic data generator and hence can be applied even when the model is onlyimplicit.
In contrast to classical testing, the sample size from the synthetic datagenerator can be as large as desired, while the size of the observed data which thegenerator aims to emulate is fixed.
Experimental results on synthetic distributionsand trained generative models on synthetic and real datasets illustrate that themethod shows improved power performance compared to existing approaches.
Introduction
Synthetic data capturing main features of the original dataset are of particular interest for machinelearning methods.
The use of original dataset for machine learning tasks can be problematic or evenprohibitive in certain scenarios, e.g. under authority regularisation on privacy-sensitive information,training models on small-sample dataset or calibrating models with imbalanced groups.
High qualitysynthetic data generation procedures surpass some of these challenges by creating de-identified datato preserve privacy and to augment small or imbalance datasets.
Training deep generative modelshas been widely studied in the recent years [Kingma and Welling, 2013, Radford et al., 2015, Songand Kingma, 2021] and methods such as those based on Generative Adversarial Networks (GANs)[Goodfellow et al., 2014] provide powerful approaches that learn to generate synthetic data whichresemble the original data distributions.
However, these deep generative models usually do notprovide theoretical guarantees on the goodness-of-fit to the original data [Creswell et al., 2018].
To the best of our knowledge, existing mainstream developments for deep generative models [Songand Ermon, 2020, Li et al., 2017] do not provide a systematic approach to assess the quality of thesynthetic samples.
Instead, heuristic methods are applied, e.g. for image data, the quality of samples are generally decided via visual comparisons.
The training quality has been studied relying largelyon the specific choice of training loss, which does not directly translate into a measure of sample quality; in the case of the log-likelihood [Theis et al., 2015].
Common quality assessment measuresfor implicit generative models, on images for example, include Inception Scores (IS)
[Salimanset al., 2016] and Fréchet Inception Distance (FID) [Heusel et al., 2017], which are motivated byhuman inception systems in the visual cortex and pooling [Wang et al., 2004].
Bi´nkowski et al.[2018] pointed out issues for IS and FID and developed the Kernel Inception Distance (KID) formore general datasets.
Although these scores can be used for for comparisons, they do not provide astatistical significance test which would assess whether a deemed good generative model is “ goodenough ”.
A key stumbling block is that the distribution from which a synthetic method generatessamples is not available; one only ever observes samples from it.
For models in which the density is known explicitly, at least up to a normalising constant, someassessment methods are available.
Gorham and Mackey [2017] proposed to assess sample qualityusing discrepancy measures called kernelised Stein discrepancy (KSD).
Schrab et al.
[2022] assessesthe quality of generative models on the MNIST image dataset from LeCun et al.
[1995] using anaggregated kernel Stein discrepancy (KSDAgg) test; still an explicit density is required.
The onlyavailable implicit goodness-of-fit test, AgraSSt [Xu and Reinert, 2022], applies only to generators offinite graphs; it is also of KSD form and makes extensive use of the discrete and finite nature of theproblem.
To date, quality assessment procedures of implicit deep generative models for continuousdata remains unresolved.
This paper provides a solution of this problem.
The underlying idea can be sketched as follows.
Traditionally, given a set of n observations, each in
R m , one would estimate the distribution of these observations from the data and then check whetherthe synthetic data can be viewed as coming from the data distribution.
Here instead we characterisethe distribution which is generated possibly implicitly from the synthetic data generator, and thentest whether the observed data can be viewed as coming from the synthetic data distribution.
Theadvantage of this approach is that while the observed sample size n may be fairly small, the syntheticdata distribution can be estimated to any desirable level of accuracy by generating a large number ofsamples.
Similarly to the works mentioned in the previous paragraph for goodness-of-fit tests, we usea KSD approach, based on a Stein operator which characterises the synthetic data distribution.
As thesynthetic data generator is usually implicit, this Stein operator is not available.
We show howeverthat it can be estimated from synthetic data samples to any desired level of accuracy.
Our contributions We introduce a method to assess (deep) generative models, which are oftenblack-box approaches, when the underlying probability distribution is continuous, usually in highdimensions.
To this purpose, we develop a non-parametric Stein operator and the correspondingnon-parametric kernel Stein discrepancies (NP-KSD), based on estimating conditional score functions.
Moreover, we give theoretical guarantees for NP-KSD.
This paper is structured as follows.
We start with a review of Stein’s method and KSD goodness-of-fittests for explicit models in Section 2 before we introduce the NP-KSD in Section 3 and analysethe model assessment procedures.
We show results of experiments in Section 4 and conclude withfuture directions in Section 5.
Theoretical underpinnings, and additional results are provided in thesupplementary material.
Code is also attached in the supplementary material.
Stein’s method and kernel Stein discrepancy tests
Stein identities, equations, and operators Stein’s method [Stein, 1972] provides an elegant toolto characterise distributions via Stein operators , which can be used to assess distances betweenprobability distributions [Barbour and Chen, 2005, Barbour, 2005, Barbour et al., 2018].
Given adistribution q , an operator A q is called a Stein operator w.r.t.
q and Stein class F if the following
Stein identity holds for any test function f ∈ F : E q
[ A q f ] = 0 .
For a test function h one then aims tofind a function f = f h ∈ F which solves the Stein equation
[Equation]
[Equation]
Then for any distribution p , taking expectations E p in Eq.
1 assesses the distance | E p
h − E q h | through| E p
A q f | , an expression in which randomness enters only through the distribution p .
When the density function q is given explicitly, with smooth support Ω q ⊂ R m , is differentiableand vanishes at the boundary of Ω q , a common choice of Stein operator in the literature utilisesthe score-function, see for example Mijoule et al.
The gradient operator is denoted by ∇and taken to be a column vector.
The score function of q is defined as s q = ∇ log q = ∇ qq (withthe convention that s q ≡ 0 outside of Ω q ).
Let f = ( f 1 , . . .
, f m ) ⊤ where f i : R m → R , ∀ i, aredifferentiable.
The score-Stein operator 1 is the vector-valued operator acting on (vector-valued)function f ,
[Equation]
and the Stein identity E q
[ A q f ] = 0 holds for functions f which belong to the so-called canonical
Stein class defined in Mijoule et al.
[2021], Definition 3.2.
As it requires knowledge of the densityq only via its score function, this Stein operator is particularly useful for unnormalised densities[Hyvärinen, 2005], appearing e.g. in energy based models (EBM)
[LeCun et al., 2006].
Kernel Stein discrepancy Stein operators can be used to assess discrepancies between two probability distributions; the Stein discrepancy between probability distribution p and q (w.r.t.
class B ⊂ F )is defined as [Gorham and Mackey, 2015]
[Equation]
As the sup f over a general class B can be difficult to compute, taking B as the unit ball of a reproducing kernel Hilbert space (RKHS) has been considered, resulting in the kernel Stein discrepancy(KSD) defined as [Gorham and Mackey, 2017]
[Equation]
Denoting by k the reproducing kernel associated with the RKHS H over a set X , the reproducingproperty ensures that ∀ f ∈ H , f ( x ) = ⟨ f, k ( x , · ) ⟩ H , ∀ x ∈ X .
Algebraic manipulations yield
[Equation]
where u q ( x , ˜ x ) = ⟨A
q k ( x , · ) , A q k (˜ x , · ) ⟩ H , which takes the exact sup without approximation anddoes not involve the (sample) distribution p .
Then, KSD 2 can be estimated through empirical means,over samples from p , e.g. V-statistic
[Van der Vaart, 2000] and U-statistics [Lee, 1990]
estimates are
[Equation]
KSD has been studied as discrepancy measure between distributions for testing model goodness-of-fit[Chwialkowski et al., 2016, Liu et al., 2016].
KSD testing procedure Suppose we have observed samples x 1 , . . .
, x n from the unknown distribution p .
To test the null hypothesis H 0 : p = q against the (broad class of) alternative hypothesis
H 1 : p ̸ = q , KSD can be empirically estimated via Eq. 6.
The null distribution is usually simulatedvia the wild-bootstrap procedure [Chwialkowski et al., 2014].
Then if the empirical quantile, i.e. theproportion of wild bootstrap samples that are larger than KSD 2 v ( q ∥ p ) , is smaller than the pre-definedtest level (or significance level)
α , the null hypothesis is rejected; otherwise the null hypothesis isnot rejected.
In this way, a systematic non-parametric goodness-of-fit testing procedure is obtained,which is applicable to unnormalised models.
Non-Parametric kernel Stein discrepancies
The construction of a KSD relies on the knowledge of the density model, up to normalisation.
However, for deep generative models where the density function is not explicitly known, the computationfor Stein operator in Eq. 2, which is based on an explicit parametric density, is no longer feasible.
While in principle one could estimate the multivariate density function from synthetic data, densityestimation in high dimensions is known to be problematic, see for example Scott and Sain [2005].
Instead, Stein’s method allows to use a two-step approach: For data in R m , we first pick a coordinatei ∈ [ m ] := { 1 , . . .
, m } , and then we characterize the uni-variate conditional distribution of that coordinate, given the values of the other coordinates.
Using score Stein operators from Ley et al.
[2017],this approach only requires knowledge or estimation of uni-variate conditional score functions.
We denote observed data z 1 , . . .
, z n with z i =
( z (1) i , . . .
, z ( m ) i )
⊤ ∈ R m ; and denoting the generativemodel as G , we write X ∼ G to denote a random R m -valued element from the (often only givenimplicitly) distribution which is underlying G .
Using G , we generate N samples denoted byy 1 , . . .
, y N .
In our case, n is fixed and n ≪ N , allowing N → ∞ in theoretical results.
The kernel ofan RKHS is denoted by k and is assumed to be bounded.
For x ∈ R m , x ∈ R and g ( x ) : R m → R , wewrite g x ( − i ) ( x ) : R → R for the uni-variate function which acts only on the coordinate i and fixes theother coordinates to equal x ( j ) ,
j ̸ = i , so that g x ( − i ) ( x ) =
g ( x (1) , . . .
, x ( i − 1) , x, x ( i +1) , . . .
, x ( m ) ) .
For i ∈ [ m ] let T ( i ) denote a Stein operator for the conditional distribution Q ( i ) =
Q ( i ) x
( − i ) with
E Q ( i ) x ( − i ) g x ( − i ) ( x ) =
E [ g y ( − i ) ( Y ) | Y ( j ) =
y ( j ) , j ̸ = i ] .
The proposed Stein operator A acting onfunctions g : R m → R underlying the non-parametric Stein operator is
[Equation]
We note that for X ∼ q , the Stein identity
E A g ( X ) = 0 holds and thus A is a Stein operator.
Thedomain of the operator will depend on the conditional distribution in question.
Instead of using theweights w i = 1 m , other positive weights which sum to 1 would be possible, but for simplicity we useequal weights.
A more detailed theoretical justification of Eq.
7 is given in Appendix A.
In what follows we use as Stein operator for a differentiable uni-variate density q the score operatorfrom Eq. 2, given by
[Equation]
In Proposition D.1 of Appendix D we shall see that the operator in Eq.
7 equals the score-Steinoperator in Eq.
2; in Appendix D an example is also given.
For the development in this paper, Eq.
7 ismore convenient as it relates directly to conditional distributions.
Other choices of Stein operators arediscussed for example in Ley et al.
[2017], Mijoule et al.
[2021], Xu [2022].
Re-sampling Stein operators The Stein operator Eq.
7 depends on all coordinates i ∈
[ m ] .
Whenm is large we can estimate this operator via re-sampling with replacement, as follows.
We draw Bsamples { i 1 , . . .
, i B } with replacement from [ m ] such that { i 1 , . . .
, i B } ∼ Multinom( B, { 1 m } i ∈ [ m ] ) .
The re-sampled Stein operator acting on f : R m → R is
[Equation]
Then we have E A B f [Equation] B
[Equation]
Bb =1 E A
( i b ) f ( X ) = 0 .
So A B is again a Stein operator.
In practice, when m is large, the stochastic operator in Eq.
9 creates a computationally efficient wayfor comparing distributions.
A similar re-sampling strategy for constructing stochastic operatorsare considered in the context of Bayesian inference [Gorham et al., 2020], where conditional scorefunctions, which are given in parametric form, are re-sampled to derive score-based (or Langevin)
Stein operators for posterior distributions.
The conditional distribution has been considered [Wanget al., 2018] and [Zhuo et al., 2018] in the context of graphical models [Liu and Wang, 2016].
Ingraphical models, the conditional distribution is simplified to conditioning on the Markov blanket[Wang et al., 2018], which is a subset of the full coordinate; however, no random re-sampling is used.
Conditional distributions also apply in message passing, but there, the sequence of updates is ordered.
Estimating Stein operators via score matching Usually the score function q ′ /q
in Eq.
8 is notavailable but needs to be estimated.
An efficient way of estimating the score function is throughscore-matching, see for example [Hyvärinen, 2005, Song and Kingma, 2021, Wenliang et al., 2019].
Score matching relies on the following score-matching (SM) objective [Hyvärinen, 2005],
[Equation]
(10) which is particularly useful for unnormalised models such as EBMs.
Additional details are includedin
Appendix E. Often score matching estimators can be shown to be consistent, see for example Songet al.
Proposition 3.1, proven in Appendix B, gives theoretical guarantees for the consistencyof a general form of Stein operator estimation, as follows.
Proposition 3.1.
Suppose that for i ∈ [ m ] , (cid:98) s ( i ) N is a consistent estimator of the uni-variate scorefunction s ( i ) .
Let T ( i ) be a Stein operator for the uni-variate differentiable probability distribution
Q ( i ) of the generalised density operator form Eq. 8. Let(cid:98)
T ( i ) N g ( x ) =
g ′ ( x ) + g ( x ) (cid:98) s ( i ) N and (cid:98)
A g = (cid:98) T ( I ) N g x ( − I ) .
Then (cid:98) T ( i ) N is a consistent estimator for T ( i ) , and (cid:98) A is a consistent estimator of A .
Non-parametric Stein operators with summary statistics In practice, the data y ( − i ) ∈
R m − 1can be high dimensional, e.g. image pixels, and the observations can be sparse.
Thus, estimationof the conditional distribution can be unstable or exponentially large sample size is required.
Inspired by Xu and Reinert [2021] and Xu and Reinert [2022], we use low-dimensional measurablenon-trivial summary statistics t and the conditional distribution of the data given t as new targetdistributions.
Heuristically, if two distributions match, then so do their conditional distributions.
Thus, the conditional distribution Q ( i ) ( A ) is replaced by Q ( i
) t ( A ) =
P ( X ( i ) ∈
A | t ( x ( − i ) )) .
Settingt ( x ( − i ) )
=
x ( − i ) replicates the actual conditional distribution.
We denote the uni-variate score function of q t ( x | t ( x − i )) by s ( i ) t ( x | t ( x − i )) , or by s ( i ) t ( x ) when the context is clear.
The summary statisticst ( x ( − i ) ) can be uni-variate or multi-variate, and they may attempt to capture useful distributionalfeatures.
Here we consider uni-variate summary statistics such as the sample mean.
The non-parametric Stein operator enables the construction of Stein-based statistics based on Eq.
7with estimated score functions (cid:98) s ( i ) t,N using generated samples from the model G , as shown in Algorithm 1.
The re-sampled non-parametric Stein operator is
[Equation]
Non-parametric kernel Stein discrepancy With the well-defined non-parametric Stein operator,we define the corresponding non-parametric Stein discrepancy (NP-KSD) using the Stein operator in
Eq. 9, the Stein discrepancy notion in Eq. 3 and choosing as set of test functions the unit ball of the
RKHS within unit ball RKHS.
Similarly to Eq.
4, we define the NP-KSD with summary statistic t as
[Equation]
A similar quadratic form as in Eq.
5 applies to give
[Equation] [Equation]
where (cid:98) u Bt,N ( x , ˜ x ) = ⟨ (cid:98)
A Bt,N k
( x , · ) , (cid:98) A Bt,N k (˜ x , · ) ⟩ H .
The empirical estimate is
[Equation]
where S = { z 1 , . . .
, z n } ∼ p .
Thus, NP-KSD allows the computation between a set of samples anda generative model, enabling the quality assessment of synthetic data generators even for implicitmodels.
The relationship between NP-KSD and KSD is clarified in the following result; we use the notationˆ s t,N = (ˆ s t,N ( x ( i ) ) , i ∈ [ m ]) .
Here we set
[Equation]
with
[Equation]
as in Eq. 7, and following Eq.
8, T ( i )
q t g x ( − i )
( x ) =
g ′ x ( − i ) ( x )+ g x ( − i ) ( x ) s ( i ) t ( x | t ( x ( − i ) )) .
More detailsabout the interpretation of this quantity are given in App. B.1.
Theorem 3.2.
Assume that the score function estimator vector ˆ s t,N =
(ˆ s ( i ) t,N , i = 1 , . . .
, m ) ⊤ isasymptotically normal with mean 0 and covariance matrix N − 1 Σ s .
Then NP-KSD 2 t ( G ∥ p ) convergesin probability to KSD 2 t ( q t ∥ p ) at rate at least min( B − 12 , N − 12 ) .
The proof of Theorem 3.2, which is found in App. B, also shows that the distribution
NP-KSD 2 t ( G ∥ p ) − KSD 2 t ( q t ∥ p ) involves mixture of normal variables.
The assumption of asymptoticnormality for score matching estimators is often satisfied, see for example
Song et al.
Model assessment with NP-KSD Given an implicit generative model G and a set of observedsamples S
=
z 1 , . .
z n } , we aim to test the null hypothesis H 0 : S ∼ G versus the alternative
H 1 : S ̸∼ G .
This test assumes that samples generated from G follows some (unknown) distributionq and S are generated according to some (unknown) distribution p .
The null hypothesis is H 0 : p = qwhile the alternative is H 1 : p ̸
= q .
We note that the observed sample size n is fixed.
NP-KSD testing procedures NP-KSD can be applied for testing the above hypothesis using thetesting procedure outlined in Algorithm 2.
In contrast to the KSD testing procedure in Section 2,the NP-KSD test in Algorithm 2 is a Monte Carlo based test [Xu and Reinert, 2021, 2022, Schrabet al., 2022] for which the null distribution is approximated via samples generated from G insteadof the wild bootstrap procedure [Chwialkowski et al., 2014].
The reasons for employing the Monte
Carlo testing strategy instead of the wild-bootstrap are 1).
The non-parametric Stein operator dependson the random function (cid:98) s t so that classical results for V-statistics convergence which assume thatthe sole source of randomness is the bootstrap may not apply 2 ; 2).
While the wild-bootstrap isasymptotically consistent as observed sample size n → ∞ , it may not necessarily control the type-Ierror in a non-asymptotic regime where n is fixed.
More details can be found in Appendix F.
Here we note that any test which is based on the summary statistic t will only be able to test fora distribution up to equivalence of their distributions with respect to the summary statistic t ; twodistributions P and Q are equivalent w.r.t.
the summary statistics t if P ( X | t ( X ))
=
Q ( X | t ( X )) .
Thus the null hypothesis for the NP-KSD test is that the distribution is equivalent to P with respect tot .
Hence, the null hypothesis specifies the conditional distribution, not the unconditional distribution.
Related works To assess whether an implicit generative models can generate samples that aresignificantly good for the desired data model, several hypothesis testing procedures have beenstudied.
Jitkrittum et al.
[2018] has proposed kernel-based test statistics, Relative Unbiased Mean
Embedding (Rel-UME) test and Relative Finite-Set Stein Discrepancy (Rel-FSSD) test for relativemodel goodness-of-fit, i.e. whether model S is a better fit than model R.
While Rel-UME is applicablefor implicit generative models, Rel-FSSD still requires explicit knowledge of the unnormalised density.
The idea for assessing sample quality for implicit generative models is through addressing two-sampleproblem, where samples generated from the implicit model are compared with the observed data.
Inthis sense, maximum-mean-discrepancy (MMD) may also apply for assessing sample qualities forthe implicit models.
With efficient choice of (deep) kernel, Liu et al.
[2020] applied MMD tests toassess the distributional difference for image data, e.g. MNIST [LeCun et al., 1998] v.s. digits imagetrained via deep convolutional GAN (DCGAN)
[Radford et al., 2015]; CIFAR
[Krizhevsky, 2009]v.s.
CIFAR10.
[Recht et al., 2019].
However, as the distribution is represented via samples, thetwo-sample based assessment suffers from limited probabilistic information from the implicit modeland low estimation accuracy when the sample size for observed data is small.
Experiments
4.1 Baseline and competing approaches
We illustrate the proposed NP-KSD testing procedure with different choice of summary statistics.
Wedenote by NP-KSD the version which uses the estimation of the conditional score, i.e. t ( x ( − i ) )
=
x ( − i ) ; by NP-KSD_mean the version which uses conditioning on the mean statistics, i.e. t ( x ( − i ) )
=1 m − 1 (cid:80)
j ̸
= i x ( j ) ; and by NP-KSD_G the version which fits a Gaussian model as conditional density 3 .
Two-sample testing methods can be useful for model assessment, where the observed sample setis tested against sample set generated from the model.
In our setting where n ≪ N , we considera consistent non-asymptotic MMD-based test, MMDAgg [Schrab et al., 2021], as our competingapproach; see Appendix F for more details.
For synthetic distributions where the null models haveexplicit densities, we include the KSD goodness-of-fit testing procedure in Section 2 as the baseline.
Gaussian kernels are used and the median heuristic
[Gretton et al., 2007] is applied for bandwidthselection.
As a caveat, in view of [Gorham and Mackey, 2015], when the kernel decays more rapidlythan the score function grows, then identifiability of q t through a KSD method may not be guaranteed.
Details while MMD is not included in this list are found in Appendix F.
4.2 Experiments on synthetic distributions
Gaussian Variance Difference (GVD)
We first consider a standard synthetic setting, studied in
Jitkrittum et al.
[2017], in which the null distribution is multivariate Gaussian with mean zero and
0.8 Variance perturbation 0.
R e j e c t
i o n r a t e (a) GVD: n =
Generated sample size 0.
R e j e c t
i o n r a t e (b) GVD: σ per = 0 .0.
0.8 Covariance perturbation 0.
R e j e c t
i o n r a t e (c) MoG: n =
NP-KSD
re-sample size 0.1 0.
R e j e c t
i o n r a t e (d) MoG: m = identity covariance matrix.
The alternative is set to perturb the the diagonal terms of the covariancematrix, i.e. the variances, all by the same amount.
The rejection rate against the variances perturbation is shown in Figure 1(a).
From the result, wesee that all the tests presented have controlled type-I error.
For all the tests the power increases withincreased perturbation.
NP-KSD and NP-KSD_mean outperform the MMDAgg approach.
Usingthe mean statistics, NP-KSD_mean is having slightly higher power than KSD .
The mis-specified
NP-KSD_G has lower power, but is still competitive to MMDAgg .
The test power against the sample size N generated from the null model is shown in Figure 1(b).
Thegenerated samples are used as another sample set for the MMDAgg two-sample procedure, whileused for estimating the conditional score for NP-KSD-based methods.
As the generated sample sizeincreases, the power of MMDAgg increases more slowly than that of the NP-KSD-based methods,which achieve maximum test power in the presented setting.
The NP-KSD-based tests tend to havelower variability of the test power, indicating more reliable testing procedures than MMDAgg .
Mixture of Gaussian (MoG)
Next, we consider as a more difficult problem that the null model is atwo-component mixture of two independent Gaussians.
Both Gaussian components have identitycovariance matrix.
The alternative is set to perturb the covariance between adjacent coordinates.
The rejection rate against this perturbation of covariance terms are presented in Figure 1(c).
Theresults show consistent type I error.
The NP-KSD and NP-KSD_mean tests have better test powercompared to KSD and MMDAgg , although NP-KSD has slightly higher variance.
Among the
NP-KSD tests, the smallest variability is achieved by NP-KSD_mean .
For the test with m = 40 ,we also vary the re-sample size B .
As shown in Figure 1(d), while the variability of the average testpower also increased slightly.
From the result, we also see that for B = 20 = m/ 2 the test power isalready competive compared to B = 40 .
Additional experimental results including computationalruntime and training generative models for synthetic distributions are included in Appendix C.
4.3 Applications to deep generative models
For real-world applications, we assess models trained from well-studied generative modelling procedures, including a Generative Adversarial Network ( GAN )
[Goodfellow et al., 2014] with multilayerperceptron (MLP), a Deep Convolutional Generative Adversarial Network ( DCGAN )
[Radford et al.
,2015], and a Variational Autoencoder ( VAE )
[Kingma and Welling, 2013].
We also consider a Noise
Conditional Score Network ( NCSN )
[Song and Ermon, 2020], which is a score-based generativemodelling approach, where the score functions are learned [Song and Ermon, 2019] to performedannealed Langevin dynamics for sample generation.
We also denote Real as the scheme that generatessamples randomly from the training data, which essentially acts as a generator of the null distribution.
MNIST Dataset
This dataset contains 28 × 28 grey-scale images of handwritten digits [LeCun et al., 1998] 4 .
It consist of 60 , 000 training samples and 10 , 000 test samples.
Deep gen4 https erative models in Table 1 are trained using the training samples.
We assess the quality ofthese trained generative models by testing against the true observed MNIST samples (from thetest set).
Samples from both distributions are visually illustrated in Figure 3 in Appendix C.
[Table]
600 samples are generatedfrom the generative modelsand 100 samples are used forthe test; test level α = 0 .
From Table 1, we see that allthe deep generative modelshave high rejection rate, showing that the trained models are not good enough.
Testing with the Real scheme has controlled type-Ierror.
Thus, NP-KSD detects that the “real” data are a true sample set from the underlying dataset.
CIFAR
Dataset This dataset contains 32 × 32 RGB coloured images [Krizhevsky, 2009] 5 .
It consist of 50 , 000 training samples and 10 , 000 test samples.
Deep generative models in Table 2 are trained using the training samples and test samples are randomly drawn from the test set.
[Table]
Samples are illustrated in Figure 4 in
Appendix C. We also compare withthe CIFAR10.1 dataset[Recht et al.
,2018] 6 , which is created to differ from
CIFAR10 to investigate generalisationpower for training classifiers.
800 samples are generated from the generativemodels and 200 samples are used for the test; test level α = 0 .
Table 2 shows higher rejection ratesfor NP-KSD tests compared to MMDAgg, echoing the results for synthetic distributions.
The trained
DCGAN generates samples with lower rejection rate in the CIFAR10 dataset than in the CIFAR10.1dataset.
We also see that the score-based NCSN has higher rejection rate than the non-score-based
DCGAN, despite NP-KSD being a score-based test.
The distribution difference between CIFAR10and CIFAR10.1 can be well-distinguished from the tests.
Testing with the Real scheme again hascontrolled type-I error.
Conclusion and future directions
Synthetic data are in high demand, for example for training ML procedures; quality is important.
Synthetic data which miss important features in the data can lead to erroneous conclusions, whichin the case of medical applications could be fatal, and in the case of loan applications for examplecould be detrimental to personal or business development.
NP-KSD provides a method for assessingsynthetic data generators which comes with theoretical guarantees.
Our experiments on syntheticdata have shown that NP-KSD achieves good test power and controlled type-I error.
On real data,
NP-KSD detects samples from the true dataset.
That none of the classical deep learning methods usedin this paper has a satisfactory rejection rate indicates scope for further developments in syntheticdata generation.
Future research will assess alternatives to the computer-intensive Monte Carlo method for estimatingthe null distribution, for example adapting wild-bootstrap procedures.
It will explore alternativechoices of score estimation as well as of kernel functions.
Finally, some caution is advised.
The choice of summary statistic may have strong influence on theresults and a classification based on NP-KSD may still miss some features.
Erroneous decisionscould be reached when training classifiers.
Without scrutiny this could lead to severe consequencesfor example in health science applications.
Yet NP-KSD is an important step towards understandingblack-box data generating methods and thus understanding their potential shortcomings.