KSD Aggregated Goodness-of-ﬁt Test
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We investigate properties of goodness-of-ﬁt tests based on the Kernel Stein Discrepancy (KSD).
We introduce a strategy to construct a test, called KSDA GG ,which aggregates multiple tests with different kernels.
KSDA GG avoids splittingthe data to perform kernel selection (which leads to a loss in test power), andrather maximises the test power over a collection of kernels.
We provide theoretical guarantees on the power of KSDA GG : we show it achieves the smallestuniform separation rate of the collection, up to a logarithmic term.
KSDA GG canbe computed exactly in practice as it relies either on a parametric bootstrap or on awild bootstrap to estimate the quantiles and the level corrections.
In particular, forthe crucial choice of bandwidth of a ﬁxed kernel, it avoids resorting to arbitraryheuristics (such as median or standard deviation) or to data splitting.
We ﬁnd onboth synthetic and real-world data that KSDA GG outperforms other state-of-the-artadaptive KSD-based goodness-of-ﬁt testing procedures.
Introduction
Kernel selection remains a fundamental question in kernel-based nonparametric hypothesis testing,as it signiﬁcantly impacts the test power.
Kernel selection has attracted a signiﬁcant interest inthe literature and a number of methods have been proposed in different settings, such as in thetwo-sample, independence and goodness-of-ﬁt testing frameworks.
Those methods include usingheuristics (Gretton et al., 2012a), relying on data splitting (Gretton et al., 2012b; Sutherland et al.
,2017; Kübler et al., 2022), learning deep kernels (Grathwohl et al., 2020; Liu et al., 2020), workingin the post-selection inference framework (Yamada et al., 2019; Lim et al., 2019, 2020; Kübler et al.
,2020; Freidling et al., 2021), to name but a few.
In this work, we focus on aggregated tests, which have been investigated for the two-sample problemby Fromont et al.
(2013), Kim et al.
(2022) and Schrab et al.
(2021) using the Maximum Mean
Discrepancy (MMD, Gretton et al., 2012a), and for the independence problem by Albert et al.
(2022)and Kim et al.
using the Hilbert Schmidt Independence Criterion (HSIC, Gretton et al., 2005).
We extend the use of aggregated tests to the goodness-of-ﬁt setting, where we are given a modeland some samples, and we are interested in deciding whether the samples have been drawn fromthe model.
We employ the Kernel Stein Discrepancy (KSD, Chwialkowski et al., 2016;
Liu et al.,2016)
as our test statistic, which is an ideal measure of distance for this setting: it admits an estimatorwhich can be computed without requiring samples from the model, and does not require the model tobe normalised.
To the best of our knowledge, ours represents the ﬁrst aggregation procedure for the
KSD test in the literature.
Related work.
Fromont et al.
(2012, 2013) introduced non-asymptotic aggregated tests for thetwo-sample problem with the equal sample sizes following a Poisson process, using as test statistic anunscaled version of the MMD.
They also provided theoretical results in terms of uniform separationrates using a wild bootstrap.
Albert et al.
(2022) then proposed an aggregated test for the independence problem using the HSIC, with guarantees using a theoretical quantile, but relying on permutations toobtain the test threshold in practice.
Kim et al.
(2022) then extended those results to also hold forthe estimated quantile, and generalised the two-sample results to hold for the MMD estimator withdifferent sample sizes using a wild bootstrap.
All those aforementioned results were proved for the
Gaussian kernel only.
Schrab et al.
(2021) generalised the two-sample results to hold for a wide rangeof kernels using either a wild bootstrap or a permutation-based procedure, and provided optimalityresults which hold with fewer restrictions on the class of functions.
Our work builds and extends onthe above results: we consider the goodness-of-ﬁt framework, where we have samples from onlyone of the two densities.
The main challenges arise from working with the Stein kernel that deﬁnesthe KSD test statistic: for example, we lose the transition invariant property of the kernel which iscrucial to work in the Fourier domain.
We also point out the very relevant work of Balasubramanianet al.
(2021) who considered adaptive MMD-based goodness-of-ﬁt tests and studied their uniformseparation rates over Sobolev balls in the asymptotic regime.
More generally, Li and Yuan (2019)studied asymptotic adaptive kernel-based tests for the three testing frameworks.
Finally, Key et al.(2021) addressed the complementary task of Stein test design for a family of models, rather than for asingle model.
Contributions.
We propose a solution to the fundamental kernel selection problem for the widelyused KSD goodness-of-ﬁt tests: we construct an adaptive test KSDA GG which aggregates multipletests with different kernels.
Our contribution is in showing, both theoretically and experimentally, thatthe aggregation procedure works in this novel setting in which it has never been considered before.
We consider the kernel selection framework; this general setting has many applications includingthe one of kernel bandwidth selection.
Our aggregated test allows for two numerical methods forestimating the test thresholds: the wild bootstrap and the parametric bootstrap (a procedure uniqueto the goodness-of-ﬁt framework).
We conduct a theoretical analysis: we provide a lower boundon the uniform separation rate (Baraud, 2002) of KSDA GG , a condition which guarantees testpower.
We discuss the implementation of KSDA GG and experimentally validate our proposedapproach on benchmark problems, not only on datasets classically used in the literature but alsoon original data obtained using state-of-the-art generative models (i.e. Normalizing Flows).
Weobserve, both on synthetic and real-world data, that KSDA GG obtains higher power than other
KSD-based adaptive state-of-the-art tests.
Contributing to the real-world applications of thesegoodness-of-ﬁt tests, we provide publicly available code to allow practitioners to employ our method:https://anonymous.4open.science/r/ksdagg-DBF7/README.md .
Outline.
Section 2 presents our framework and our notation.
Section 3 introduces our algorithm
KSDA GG (in Algorithm 1) and contains our main theoretical results.
Section 4 presents numericalexperiments to support KSDA GG .
We close the paper with avenues for future research in Section 5.
Notation
We consider the goodness-of-ﬁt problem where given access to a known probability density p (model)and to some i.i.d.
d -dimensional samples X N :=
( X i )
Ni =1 drawn from an unknown density q , wewant to decide whether p 6 = q holds.
This can be expressed as a statistical hypothesis testing problemwith null hypothesis H 0 : p = q and alternative H a : p 6 = q .
As a measure of distance between p and q , we use the Kernel Stein Discrepancy (KSD) introduced by
Chwialkowski et al.
(2016) and Liu et al.
For a kernel k , the KSD is deﬁned as the Maximum
Mean Discrepancy (MMD, Gretton et al., 2012a) between p and q using the Stein kernel associatedto k , that is
[Equation]
where the Stein kernel h p,k : R d ⇥ R d ! R is deﬁned as
[Equation]
and satisﬁes the Stein identity E p [ h p,k ( X, · )]
= 0 .
A quadratic-time KSD estimator can be computedas the U
-statistic
(Hoeffding, 1992)
[Equation]
In this work, the model density p is always known, we do not always explicitly write the dependenceof p for all variables (as we do for KSD 2 p,k and h p,k ).
We assume that the kernel k is such that
[Equation]
We now address the requirements for consistency of the Stein test (Chwialkowski et al., 2016,
Theorem 2.2): we assume that the kernel k is C 0 -universal
(Carmeli et al., 2010, Deﬁnition 4.1)
andthat E q
[Equation]
We use the notations P p and P q to denote the probability under the model p and under q , respectively.
Given a kernel  : R d ⇥ R d ! R and a function f : R d ! R in L R d  , we consider the integraltransform T  deﬁned as
[Equation]
for y 2 R d .
When the kernel  is translation invariant, the integral transform corresponds to aconvolution, however, this is not true of the Stein kernel.
Construction of tests and bounds
We now introduce the single and aggregated KSD tests.
We show that these control the probability oftype I error as desired, and provide conditions for the control of the probability of type II error.
3.1 Single test
We ﬁrst construct a KSD test for a ﬁxed kernel k as proposed by Chwialkowski et al.
(2016) and Liuet al.
To estimate the test threshold, we can either use a wild bootstrap (Shao, 2010; Leuchtand Neumann, 2013; Fromont et al., 2012; Chwialkowski et al., 2014) or a parametric bootstrap(Stute et al., 1993).
Both methods work by simulating sampling values  [Equation] ¯
[Equation] k  from the(asymptotic) distribution of [ KSD 2 p,k under the null hypothesis and estimating the (1  ↵ ) -quantileusing a Monte Carlo approximation
[Equation]
where ¯ K • 1 k  · · ·  ¯ K • B 1 k are the sorted elements
¯ K 1 k , . .
, ¯ K B 1 k  .
The single test is then deﬁned as
[Equation]
For the parametric bootstrap , we directly draw new samples ( X 0 i ) N {+
+}
i =1 from the model p and computethe KSD
[Equation]
For the wild bootstrap , we ﬁrst generate n i.i.d.
Rademacher random variables ✏ 1 , . . . , ✏ n takingvalues in {  1 , 1 } n , and then compute
[Equation]
Both these processes are then repeated B 1 times.
Since it uses samples from the model p , the parametric bootstrap (Stute et al., 1993) results in atest with non-asymptotic level ↵ .
This comes at the cost of being computationally more expensiveand assuming that we are able to sample from p (which may be out of reach in some settings).
Conversely, the wild bootstrap has the advantage of not requiring to sample from p , which makes itcomputationally more efﬁcient as only one kernel matrix needs to be computed, but it only achievesthe desired level ↵ asymptotically (Shao, 2010; Leucht and Neumann, 2013; Chwialkowski et al.
Note that we cannot obtain a non-asymptotic level for the wild bootstrap by relying onthe result of Romano and Wolf (2005, Lemma 1) as done in the two-sample framework by Fromontet al.
(2013) and Schrab et al.
This is because in our case ¯ K k and [ KSD 2 p,k ( X N ) are notexchangeable variables under the null hypothesis, due to the asymmetry of the KSD statistic withrespect to p and q .
Having discussed control of the probability of type I error of the single test  k ↵ , we now providea condition on k p  q k 2 which ensures that the probability of type II error is controlled by some 122  2 (0 , 1) .
The smallest such value of k p  q k 2 , provided that p  q lies in some given class of 123 functions, is called the uniform separation rate (Baraud, 2002).
Theorem 3.1.
Let := p  q and assume that max( k p k 1 , k q k 1 )  M .
Consider C k as deﬁned in
Equation (2) , ↵ 2 (0 , e  1 ) ,  2 (0 , 1) and B 1 2 N satisfying B 1  3 ↵ [Equation] ↵ [Equation]
[Equation]  . There exists a positive constant C such that the condition
[Equation]
T h p,k  2 + C log ✓ ↵ ◆ p C k  N guarantees control over the probability of type II error, such that P q  [Equation] ( X [Equation]
=   .
Theorem 3.1, which is proved in Appendix A, provides a power guaranteeing condition consisting oftwo terms.
The ﬁrst term k  T h p,k k 22 indicates the size of the effect of the Stein operator on the 127 difference in densities :
= p  q , and is a measure of distance from the null (where this quantity iszero).
The second term log  1 / ↵  (  N )  1 p C k is obtained from upper bounding the variance of the
KSD U -statistic, it depends on the expectation of the squared Stein kernel C
k :=
E q,q [ h p,k ( X, Y ) 2 ] .
This second term also controls the quantile of the test.
3.2 Aggregated test
We can now introduce our aggregated test, which is motivated by the earlier works of Fromont et al.
Albert for different testing frameworks.
We compute e K 1 k , . .
, e K B 2 k further simulated KSD values from the null hypothesis obtained usingeither a parametric bootstrap or a wild bootstrap as in Equations (3) or (4), respectively.
Considera ﬁnite collection of kernels K satisfying the properties presented in Section 2.
We construct anaggregated test  K ↵ , called KSDA GG , which rejects the null hypothesis if one of the single tests  ku ↵ w k  k 2 K rejects it, that is
[Equation]
for some
[Equation]
.
The levels of the single tests are adjusted to ensure the aggregated test has the prescribed level ↵ .
This adjustment is performed by introducing positive weights ( w k ) k 2 K satisfying
P k 2 K w k  1 and
[Equation]
where
[Equation]
is a Monte Carlo approximation of the probability of type I error of our aggregated test with correction
[Equation]
Algorithm 1 KSDA Inputs: samples X N = ( x i ) Ni =1 , density p or score function r log p ( · ) , ﬁnite kernel collection K , weights ( w k ) k 2 K , level ↵ 2 (0 , e  1 ) , estimation parameters B 1 , B 2 , B 3 2 N , parametric or wild bootstrap
Output: 0 (fail to reject H 0 ) or 1 (reject H 0 )
Algorithm: for k 2 K do Compute  ¯ K
bk  1  b  B 1 as in Equations (3) or (4)
Sort in ascending order to obtain  ¯ K • bk  1  b  B 1 Compute  e
K bk {+ 
 +} b  B 2 as in Equations (3) or (4) u min = 0 , u max = min
k 2 K w  1 k for t = 1 , . . .
, B 3 do u = 12 ( u min + u max ) , b P u = 1 B
B X b =1 ✓ max k 2 K ⇣e
K bk  ¯ K • d B 1 (1  uw k ) e k ⌘ >◆ if b
P u  ↵ then u min =
u else u max = u u ↵ = u min if max k 2 K ⇣ [ KSD p,k ( X N )  ¯ K • d B 1 (1  u ↵ w k )
e k ⌘ > 0 then return 1 else return
Time complexity: O  [Equation]
To compute u ↵ , we estimate the supremum in Equation (5) by performing B 3 steps of the bisectionmethod.
Detailed pseudocode for KSDA GG is provided in Algorithm 1.
We verify in the next proposition that performing this correction indeed ensures that our aggregatedtest  K ↵ has the prescribed level ↵ .
Proposition 3.2.
For ↵ 2 (0 , 1) and a collection of kernels K , the aggregated test  K ↵ satisﬁes
[Equation]
non-asymptotically using a parametric bootstrap and asymptotically using a wild bootstrap.
The proof of Proposition 3.2 is presented in Appendix B. We now provide guarantees for the powerof our aggregated test KSDA GG in terms of its uniform separation rate.
Theorem 3.3.
Let :=
p  q denote the difference in densities and assume that k p k 1  M and k q k 1  M .
Consider the aggregated test  K ↵ with a collection of kernels K and associated positive weights ( w k ) k 2 K satisfying
[Equation]
k 2 K w k  1 , and with parameters ↵ 2 (0 , e  1 ) and B 1 , B 2 , B 3 2 N satisfying B 1  3 ↵
[Equation] and B 3  log [Equation] 1 k  .
Consider some  2 (0 , 1) and C k as deﬁned in Equation (2) .
There exists a positive constant C such that if
[Equation]
then the probability of type II error of  K ↵ is controlled by  , that is, P q   K ↵
[Equation]
X N [Equation] =   .
We prove Theorem 3.3 in Appendix C. We observe that the aggregation procedure allows to achievethe smallest uniform separation rate of the single tests
k ↵  k 2 K up to some logarithmic weightingterm log(1 /w k ) .
3.3 Bandwidth selection
A speciﬁc application of the setting we have considered is the problem of bandwidth selection for a ﬁxed kernel.
Given a kernel k : R d ⇥ R d ! R , the function
[Equation]
is also a kernel for any bandwidth  > 0 .
A common example is the Gaussian kernel, for whichwe have k ( x, y ) =
exp( k x  y k 22 ) and k  ( x, y ) = exp   k x  y k  .
As shown by Gorhamand Mackey (2017), a more appropriate kernel for goodness-of-ﬁt testing using the KSD is the
IMQ (inverse multiquadric) kernel, which is deﬁned with k ( x, y ) = ⇣ 1 + k x  y k⌘   k for a ﬁxedparameter  k 2 (0 , 1) as
[Equation]
which is the well-known form of the IMQ kernel with parameters  > 0 and  k 2 (0 , 1) .
Note that itis justiﬁed to consider the kernel up to a multiplicative constant because our single and aggregatedtests are invariant under this kernel transformation.
In practice, as suggested by Gretton et al.
(2012a)
, the bandwidth is often set to a heuristic such as themedian or the standard deviation of the L 2 -distances between the samples ( X i )
Ni
=1 , however, theseare arbitrary choices with no theoretical guarantees.
Another common approach proposed by Grettonet al.
(2012b) for the linear-time setting, and extended to the quadratic-time setting by Liu et al.(2020), is to resort to data splitting in order to select a bandwidth on held-out data, by maximising fora proxy for asymptotic power (see Section 4.1 for details).
Both methods were originally proposedfor the two-sample problem, but extend straightforwardly to the goodness-of-ﬁt setting.
By considering a kernel collection K ⇤ = { k  :  2 ⇤ } for a collection of bandwidths ⇤ , we can use our aggregated test KSDA GG to test multiple bandwidths using all the data and without resorting to arbitrary heuristics.
We now obtain an expression for the uniform separation rate of  K ⇤ ↵ in terms of the bandwidths  2 ⇤ .
Corollary 3.4.
Consider ↵ 2 (0 , e  1 ) ,  2 (0 , 1) and B 1 , B 2 , B 3 2 N satisfying the conditions of
Theorem 3.3 and assume that max( k p k 1 , k q k 1 )  M .
Given a collection ⇤ of positive bandwidths with associated positive weights ( w  )  2 ⇤ satisfying [Equation] 1 , we consider K ⇤ =
{ k  :  2 ⇤ } .
There exists a positive constant C such that the condition
[Equation]
ensures control over the probability of type II error of the aggregated test P q
[Equation]
Corollary 3.4 follows from applying Theorem 3.3 to the collection of kernels K ⇤ .
We do not imposeany restrictions on :=
p  q such as assuming it belongs to a speciﬁc regularity class.
For thisreason, our result holds more generally but the dependence on  in the terms k  T h p,k  k 22 andlog  [Equation] [Equation]
p C k  is not explicit. For a particular regularity class, one can obtain auniform separation rate N  r for some r > 0 by choosing appropriate collections of bandwidths andweights (depending on N ) such that the two terms have matching orders of N .
Implementation and experiments
We consider three different experiments based on a Gamma one-dimensional distribution, a GaussianBernoulli Restricted Boltzmann Machine, and a Normalizing Flow for the MNIST dataset.
Wecompare our proposed aggregated test KSDA GG against three alternatives: the KSD test which usesthe median bandwidth, a test which splits the data to select an ‘optimal’ bandwidth according to aproxy for asymptotic test power, and a test which uses extra data for bandwidth selection.
The ‘extradata’ test is designed simply to provide a best case oracle for the bandwidth selection procedurewhich maximises asymptotic test power, it cannot be used in practice (i.e. any extra samples from qwould normally be incorporated into the sample being tested).
In order to ensure that our tests alwayshave correct levels for all bandwidth values, dimensions and sample sizes, we use the parametricbootstrap in our experiments.
4.1 Alternative bandwidth selection approaches
Gretton et al.
(2012a) proposed to use the median heuristic as kernel bandwidth, it consists in the median of the L 2 -distances between the samples given by
[Equation]
Gretton et al.
(2012b)
ﬁrst proposed, for the two-sample problem using a linear-time MMD estimator,to split the data and to use half of it to select an ‘optimal’ bandwidth which maximises a proxy forasymptotic power.
This procedure was extended to quadratic-time estimators and to the goodnessof-ﬁt framework by Jitkrittum et al.
(2017), Sutherland et al.
(2017) and Liu et al.
Thesestrategies rely on the asymptotic normality of the test statistic under H a .
In our setting, the asymptoticpower proxy to maximise is the ratio [ KSD 2 p,k ( X N ) / b  H a where b  2 H a is a closed-form regularisedpositive estimator of the asymptotic variance of [ KSD 2 p,k under H a (Liu et al., 2020, Equation 5).
Inour experiments, we also consider a test which has access to N extra samples drawn from q to selectan ‘optimal’ bandwidth to run the KSD test on the original N samples X N .
This test is interesting tocompare to because it uses an ‘optimal’ bandwidth without being detrimental to power.
4.2 Experimental details
In our experiments, we use collections of bandwidths of the form ⇤ ( `  , ` + ) :
= [-(cid:8) 
-] 2 i  med : i =`  , . . .
, ` + for the median bandwidth  med and integers `  < ` + with uniform weights w  :=1 / ( ` +  `  + 1) .
For the tests which split the data, we select the bandwidth out of the collection⇤ ( `  , ` + ) which maximises the power proxy discussed in Section 4.1.
All our experiments are runwith level ↵ = 0 .
05 using the IMQ kernel deﬁned in Equation (6) with parameter  k = 0 .
We usea parametric bootstrap with B 1 = B 2 = 500 simulated KSD values to compute the adjusted testthresholds, and B 3 = 50 steps of bisection method to estimate the correction u ↵ in Equation (5).
To estimate the probability of rejecting the null hypothesis, we average the test outputs across 200repetitions.
All experiments have been run on an AMD Ryzen Threadripper 3960X 24 Cores 128Gb
RAM CPU at 3.8GHz, the runtime is of the order of a couple of hours (signiﬁcant speedup canbe obtained by using parallel computing).
We have used the implementation of Jitkrittum et al.(2017) to sample from a Gaussian-Bernoulli Restricted Boltzmann Machine, and Phillip Lippe’simplementation of MNIST Normalizing Flows, both under the MIT license.
[Figure]
4.3 Gamma distribution
For our ﬁrst experiment, we consider a one-dimensional Gamma distribution with shape parameter 5and scale parameter 5 as the model p .
We draw 500 samples from a Gamma distribution with the same scale parameter 5 and with a shifted shape parameter 5 + s for s 2 { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } .
Weconsider the collection of bandwidths ⇤ (0 , 10) .
The results we obtained are presented in Figure 1a.
We observe that all tests have the prescribedlevel 0.05 under the null hypothesis, which corresponds to the case s = 0 .
As the shift parameters increases, the two densities p and q become more different and rejection of the null becomes aneasier task, thus the test power increases.
Our aggregated test KSDA GG achieves the same power asthe ‘best case’ bound on the performance of the asymptotic power heuristic, yielded by the splittingtest with extra data.
The median test obtains only slightly lower power, this closeness in powercan be explained by the fact that this one-dimensional problem is a simple one.
We note that thenormal splitting test has signiﬁcantly lower power: this is because, even though it uses an ‘optimal’bandwidth, it is then run on only half the data, which results in a loss of power.
4.4 Gaussian-Bernoulli Restricted Boltzmann Machine
As ﬁrst considered by Liu et al.
(2016) for goodness-of-ﬁt testing using the KSD, we consider a
Gaussian-Bernoulli Restricted Boltzmann Machine.
It is a graphical model with a binary hidden variable h 2 {  1 , 1 } d h and a continuous observable variable x 2 R d .
Those variables have joint density
[Equation]
where Z is an unknown normalizing constant.
By marginalising over h , we obtain the density p of x
[Equation]
We can sample from it using a Gibbs sampler with 2000 burn-in iterations.
We use the dimensions d = 50 and d h = 40 as considered by Jitkrittum et al.
(2017) and Grathwohl et al.
Even though computing p is intractable for large dimension d h , the score function admits a convenient closed form
[Equation]
We draw the components of b and c from Gaussian standard distributions and sample Rademachervariables taking values in {  1 , 1 } for the elements of B for the model p .
We draw 1000 samplesfrom a distribution q which is constructed in a similar way as p but with the difference that
some
Gaussian noise N (0 ,  ) is injected into the elements of B .
We consider the standard deviations of theperturbations  2 { 0 , 0 . 01 , 0 .
We run our experiments with the collection of bandwidths⇤ (  20 , 0) and provide the results in Figure 1b.
Again, we observe that our aggregated test KSDA GG matches the power obtained by the test whichuses extra data to select an ‘optimal’ bandwidth.
This means that, in this experiment, KSDA GGobtains the same power as the ‘best’ single test.
The difference between KSDA GG and the medianheuristic test is signiﬁcant on this experiment, and the splitting test obtains lowest power of the fourtests.
Again, all tests have well-calibrated levels (  = 0 ) and increasing the noise level  results inmore power for all the tests.
[Figure]
4.5 MNIST Normalizing Flow
Finally, we consider a high-dimensional problem working with images in dimensions 28 2 = 784 .
Weconsider a multi-scale Normalizing Flow
(Dinh et al., 2017; Kingma and Dhariwal, 2018) which hasbeen trained on the MNIST dataset (LeCun et al., 1998, 2010), it is a generative model which has a probability density p .
As observed in Figure 2, some samples produced by the model can look exactlylike MNIST digits, while other do not resemble digits.
This Normalizing Flow has been trained to‘ideally’ produce samples from the MNIST dataset.
We are interested in whether or not we can detectthe difference in densities.
Given some images of digits, are we able to tell if those were sampledfrom the Normalizing Flow model?
We consider the case where the images from q are sampled fromthe model (level experiment, conﬁrming performance under H 0 ), and the case where the samplesfrom q are drawn from the true MNIST dataset (power experiment).
The experiments are run withthe collection of bandwidths ⇤ (  20 , 0) .
The results are displayed in Figure 3a and Figure 3b.
In Figure 3a, we observe that the four tests have correct levels (around 0.05) for the ﬁve differentsample sizes considered (the small ﬂuctuations about the designed test level can be explained bythe fact that we are averaging 200 test outputs to estimate these levels).
The well-calibrated levelsobtained in Figure 3a demonstrate the validity of the power results presented in Figure 3b.
In Figure 3b, we observe that only our aggregated test KSDA GG obtains high power, that is, is ableto detect that MNIST samples are not drawn from the Normalizing Flow.
The power of the othertests increases only marginally as the sample size increases.
We notice that the test which uses extradata to select an ‘optimal’ bandwidth performs poorly when compared to KSDA GG .
This could beexplained by the fact that this test selects the bandwidth using a proxy for the asymptotic power, andthat in this high-dimensional setting, the asymptotic regime is not attained with sample sizes below500.
[Figure]
Discussion
We have introduced KSDA GG , an aggregated goodness-of-ﬁt test based on the Kernel Stein Discrepancy.
We have investigated the theoretical properties of this adaptive test.
We have shown thatit achieves the desired level and have provided conditions to guarantee high power by exhibiting alower bound on its uniform separation rate.
We have observed in our experiments that KSDA GGoutperforms alternative state-of-the-art approaches to KSD kernel adaptation for goodness-of-ﬁttesting.
This work covers the problem of KSD adaptivity in the goodness-of-ﬁt framework without requiringdata splitting.
A potential future direction of interest could be to tackle the adaptivity problem ofthe KSD-based linear-time goodness-of-ﬁt test proposed by Jitkrittum et al.
In this setting,the data is split to select feature locations (and the kernel bandwidth), the KSD test is then run usingthose adaptive features.
A challenging problem would be to obtain those adaptive features using anaggregation procedure which avoids splitting the data.
[-Proof of Theorem 3.
-]
[-ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
-]
[-,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 ≤ (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
-]
[-ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
-]
[-ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
-]
[-which is the minimax rate over Sobolev balls (Balasubramanian et al., 
-]