KSD Aggregated Goodness-of-ﬁt Test
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We investigate properties of goodness-of-ﬁt tests based on the Kernel Stein Discrepancy (KSD).
We introduce a strategy to construct a test, called KSDA GG ,which aggregates multiple tests with different kernels.
KSDA GG avoids splittingthe data to perform kernel selection (which leads to a loss in test power), andrather maximises the test power over a collection of kernels.
We provide theoretical guarantees on the power of KSDA GG : we show it achieves the smallestuniform separation rate of the collection, up to a logarithmic term.
KSDA GG canbe computed exactly in practice as it relies either on a parametric bootstrap or on awild bootstrap to estimate the quantiles and the level corrections.
In particular, forthe crucial choice of bandwidth of a ﬁxed kernel, it avoids resorting to arbitraryheuristics (such as median or standard deviation) or to data splitting.
We ﬁnd onboth synthetic and real-world data that KSDA GG outperforms other state-of-the-artadaptive KSD-based goodness-of-ﬁt testing procedures.
Introduction
Kernel selection remains a fundamental question in kernel-based nonparametric hypothesis testing,as it signiﬁcantly impacts the test power.
Kernel selection has attracted a signiﬁcant interest inthe literature and a number of methods have been proposed in different settings, such as in thetwo-sample, independence and goodness-of-ﬁt testing frameworks.
Those methods include usingheuristics (Gretton et al., 2012a), relying on data splitting (Gretton et al., 2012b; Sutherland et al.
,2017; Kübler et al., 2022), learning deep kernels (Grathwohl et al., 2020; Liu et al., 2020), workingin the post-selection inference framework (Yamada et al., 2019; Lim et al., 2019, 2020; Kübler et al.
,2020; Freidling et al., 2021), to name but a few.
In this work, we focus on aggregated tests, which have been investigated for the two-sample problemby Fromont et al.
(2013), Kim et al.
(2022) and Schrab et al.
(2021) using the Maximum Mean
Discrepancy (MMD, Gretton et al., 2012a), and for the independence problem by Albert et al.
(2022)and Kim et al.
using the Hilbert Schmidt Independence Criterion (HSIC, Gretton et al., 2005).
We extend the use of aggregated tests to the goodness-of-ﬁt setting, where we are given a modeland some samples, and we are interested in deciding whether the samples have been drawn fromthe model.
We employ the Kernel Stein Discrepancy (KSD, Chwialkowski et al., 2016;
Liu et al.,2016)
as our test statistic, which is an ideal measure of distance for this setting: it admits an estimatorwhich can be computed without requiring samples from the model, and does not require the model tobe normalised.
To the best of our knowledge, ours represents the ﬁrst aggregation procedure for the
KSD test in the literature.
Related work.
Fromont et al.
(2012, 2013) introduced non-asymptotic aggregated tests for thetwo-sample problem with the equal sample sizes following a Poisson process, using as test statistic anunscaled version of the MMD.
They also provided theoretical results in terms of uniform separationrates using a wild bootstrap.
Albert et al.
(2022) then proposed an aggregated test for the independence problem using the HSIC, with guarantees using a theoretical quantile, but relying on permutations toobtain the test threshold in practice.
Kim et al.
(2022) then extended those results to also hold forthe estimated quantile, and generalised the two-sample results to hold for the MMD estimator withdifferent sample sizes using a wild bootstrap.
All those aforementioned results were proved for the
Gaussian kernel only.
Schrab et al.
(2021) generalised the two-sample results to hold for a wide rangeof kernels using either a wild bootstrap or a permutation-based procedure, and provided optimalityresults which hold with fewer restrictions on the class of functions.
Our work builds and extends onthe above results: we consider the goodness-of-ﬁt framework, where we have samples from onlyone of the two densities.
The main challenges arise from working with the Stein kernel that deﬁnesthe KSD test statistic: for example, we lose the transition invariant property of the kernel which iscrucial to work in the Fourier domain.
We also point out the very relevant work of Balasubramanianet al.
(2021) who considered adaptive MMD-based goodness-of-ﬁt tests and studied their uniformseparation rates over Sobolev balls in the asymptotic regime.
More generally, Li and Yuan (2019)studied asymptotic adaptive kernel-based tests for the three testing frameworks.
Finally, Key et al.(2021) addressed the complementary task of Stein test design for a family of models, rather than for asingle model.
Contributions.
We propose a solution to the fundamental kernel selection problem for the widelyused KSD goodness-of-ﬁt tests: we construct an adaptive test KSDA GG which aggregates multipletests with different kernels.
Our contribution is in showing, both theoretically and experimentally, thatthe aggregation procedure works in this novel setting in which it has never been considered before.
We consider the kernel selection framework; this general setting has many applications includingthe one of kernel bandwidth selection.
Our aggregated test allows for two numerical methods forestimating the test thresholds: the wild bootstrap and the parametric bootstrap (a procedure uniqueto the goodness-of-ﬁt framework).
We conduct a theoretical analysis: we provide a lower boundon the uniform separation rate (Baraud, 2002) of KSDA GG , a condition which guarantees testpower.
We discuss the implementation of KSDA GG and experimentally validate our proposedapproach on benchmark problems, not only on datasets classically used in the literature but alsoon original data obtained using state-of-the-art generative models (i.e. Normalizing Flows).
Weobserve, both on synthetic and real-world data, that KSDA GG obtains higher power than other
KSD-based adaptive state-of-the-art tests.
Contributing to the real-world applications of thesegoodness-of-ﬁt tests, we provide publicly available code to allow practitioners to employ our method:https://anonymous.4open.science/r/ksdagg-DBF7/README.md .
Outline.
Section 2 presents our framework and our notation.
Section 3 introduces our algorithm
KSDA GG (in Algorithm 1) and contains our main theoretical results.
Section 4 presents numericalexperiments to support KSDA GG .
We close the paper with avenues for future research in Section 5.
Notation
We consider the goodness-of-ﬁt problem where given access to a known probability density p (model)and to some i.i.d.
d -dimensional samples X N :=
( X i )
Ni =1 drawn from an unknown density q , wewant to decide whether p (cid:54) = q holds.
This can be expressed as a statistical hypothesis testing problemwith null hypothesis H 0 : p = q and alternative H a : p (cid:54) = q .
As a measure of distance between p and q , we use the Kernel Stein Discrepancy (KSD) introduced by
Chwialkowski et al.
(2016) and Liu et al.
For a kernel k , the KSD is deﬁned as the Maximum
Mean Discrepancy (MMD, Gretton et al., 2012a) between p and q using the Stein kernel associatedto k , that is
[Equation]
where the Stein kernel h p,k : R d × R d → R is deﬁned as
[Equation]
and satisﬁes the Stein identity E p [ h p,k ( X, · )]
= 0 .
A quadratic-time KSD estimator can be computedas the U
-statistic
(Hoeffding, 1992)
[Equation]
In this work, the model density p is always known, we do not always explicitly write the dependenceof p for all variables (as we do for KSD 2 p,k and h p,k ).
We assume that the kernel k is such that [Equation] 2 p
[Equation]
k
[Equation] and [Equation] We now address the requirements for consistency of the Stein test (Chwialkowski et al., 2016,
Theorem 2.2): we assume that the kernel k is C 0 -universal
(Carmeli et al., 2010, Deﬁnition 4.1)
andthat E q (cid:13)(cid:13)(cid:13)
[Equation]
(cid:16) [Equation] p ( X ) q ( X ) (cid:17)(cid:13)(cid:13)(cid:13) 2 2 < ∞ . We use the notations P p and P q to denote the probability under the model p and under q , respectively.
Given a kernel κ : R d × R d → R and a function f : R d → R in L 2 (cid:0) R d (cid:1) , we consider the integraltransform T κ deﬁned as
[Equation]
for y ∈ R d .
When the kernel κ is translation invariant, the integral transform corresponds to aconvolution, however, this is not true of the Stein kernel.
Construction of tests and bounds
We now introduce the single and aggregated KSD tests.
We show that these control the probability oftype I error as desired, and provide conditions for the control of the probability of type II error.
3.1 Single test
We ﬁrst construct a KSD test for a ﬁxed kernel k as proposed by Chwialkowski et al.
(2016) and Liuet al.
To estimate the test threshold, we can either use a wild bootstrap (Shao, 2010; Leuchtand Neumann, 2013; Fromont et al., 2012; Chwialkowski et al., 2014) or a parametric bootstrap(Stute et al., 1993).
Both methods work by simulating sampling values (cid:0) ¯ K 1 k , . . .
, ¯ K B 1 k (cid:1) from the(asymptotic) distribution of (cid:91) KSD 2 p,k under the null hypothesis and estimating the (1 − α ) -quantileusing a Monte Carlo approximation
[Equation]
where ¯ K • 1 k ≤ · · · ≤ ¯ K • B 1 k are the sorted elements (cid:0)
¯ K 1 k , . .
, ¯ K B 1 k (cid:1) .
The single test is then deﬁned as
[Equation]
For the parametric bootstrap , we directly draw new samples ( X (cid:48) i ) N [-
-](cid:48) i
=1 from the model p and computethe KSD
[Equation]
For the wild bootstrap , we ﬁrst generate n i.i.d.
Rademacher random variables (cid:15) 1 , . . . , (cid:15) n takingvalues in {− 1 , 1 } n , and then compute
[Equation]
Both these processes are then repeated B 1 times.
Since it uses samples from the model p , the parametric bootstrap (Stute et al., 1993) results in atest with non-asymptotic level α .
This comes at the cost of being computationally more expensiveand assuming that we are able to sample from p (which may be out of reach in some settings).
Conversely, the wild bootstrap has the advantage of not requiring to sample from p , which makes itcomputationally more efﬁcient as only one kernel matrix needs to be computed, but it only achievesthe desired level α asymptotically (Shao, 2010; Leucht and Neumann, 2013; Chwialkowski et al.
Note that we cannot obtain a non-asymptotic level for the wild bootstrap by relying onthe result of Romano and Wolf (2005, Lemma 1) as done in the two-sample framework by Fromontet al.
(2013) and Schrab et al.
This is because in our case ¯ K k and (cid:91) KSD 2 p,k ( X N ) are notexchangeable variables under the null hypothesis, due to the asymmetry of the KSD statistic withrespect to p and q .
Having discussed control of the probability of type I error of the single test ∆ kα , we now providea condition on (cid:107) p − q (cid:107) 2 which ensures that the probability of type II error is controlled by someβ ∈ (0 , 1) .
The smallest such value of (cid:107) p − q (cid:107) 2 , provided that p − q lies in some given class offunctions, is called the uniform separation rate (Baraud, 2002).
Theorem 3.1.
Let ψ := p − q and assume that max( (cid:107) p (cid:107) ∞ , (cid:107) q (cid:107) ∞ ) ≤ M .
Consider C k as deﬁned in
Equation (2) , α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and B 1 ∈ N satisfying B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) .
There exists a positive constant C such that the condition
[Equation]
guarantees control over the probability of type II error, such that P q (cid:0) ∆ kα ( X N )
= 0 (cid:1) ≤ β .
Theorem 3.1, which is proved in Appendix A, provides a power guaranteeing condition consisting oftwo terms.
The ﬁrst term (cid:107) ψ − T h p,k ψ (cid:107) 22 indicates the size of the effect of the Stein operator on thedifference in densities ψ :=
p − q , and is a measure of distance from the null (where this quantity iszero).
The second term log (cid:0) 1 /α (cid:1) ( βN ) − 1 √ C k is obtained from upper bounding the variance of the
KSD U -statistic, it depends on the expectation of the squared Stein kernel C
k :=
E q,q [ h p,k ( X, Y ) 2 ] .
This second term also controls the quantile of the test.
3.2 Aggregated test
We can now introduce our aggregated test, which is motivated by the earlier works of Fromont et al.(2012, 2013), Albert et al. (2022), and Schrab et al.
(2021) for different testing frameworks.
We compute (cid:101) K 1 k , . . .
, (cid:101) K B 2 k further simulated KSD values from the null hypothesis obtained usingeither a parametric bootstrap or a wild bootstrap as in Equations (3) or (4), respectively.
Considera ﬁnite collection of kernels K satisfying the properties presented in Section 2.
We construct anaggregated test ∆ K α , called KSDA GG , which rejects the null hypothesis if one of the single tests 138 (cid:0) ∆ ku α w k (cid:1) k ∈K rejects it, that is∆
[Equation]
X
[Equation]
for some k ∈ K (cid:1) .
The levels of the single tests are adjusted to ensure the aggregated test has the prescribed level α .
This adjustment is performed by introducing positive weights ( w k ) k ∈K satisfying (cid:80)
k ∈K w k ≤ 1 andsome correction
[Equation]
where
[Equation]
is a Monte Carlo approximation of the probability of type I error of our aggregated test with correction u
[Equation]
Inputs: samples X N = ( x i ) Ni =1 , density p or score function ∇ log p ( · ) , ﬁnite kernel collection K , weights ( w k ) k ∈K , level α ∈ (0 , e − 1 ) , estimation parameters B 1 , B 2 , B 3 ∈ N , parametric or wild bootstrap
Output: 0 (fail to reject H 0 ) or 1 (reject H 0 )
Algorithm: for k ∈ K do Compute (cid:0)
¯ K bk (cid:1) 1 ≤ b ≤ B 1 as in Equations (3) or (4)
Sort in ascending order to obtain (cid:0) ¯ K • bk (cid:1) 1 ≤ b ≤ B
Compute (cid:0) (cid:101) K bk [- 
 -](cid:1) 1 ≤ b ≤ B 2 as in Equations (3) or (4) u min = 0 , u max = min
k ∈K w − 1 k for t = 1 , . . .
, B 3 do u = 12 ( u min + u max ) , (cid:98) P u = 1 B
B 2 (cid:88) b = 1 (cid:18) max k ∈K (cid:16)(cid:101) K bk − ¯ K •(cid:100)
B 1 (1 − uw k ) (cid:101) k (cid:17) > 0 (cid:19) if (cid:98) P u ≤ α then u min =
u else u max = u
u α = u min if max k ∈K (cid:16) (cid:91) KSD 2 p,k ( X N ) − ¯ K •(cid:100)
B 1 (1 − u α w k ) (cid:101) k (cid:17) > 0 then return 1 else return
Time complexity: O (cid:0) N 2 ( B 1 + B 2 ) | Λ | (cid:1)
To compute u α , we estimate the supremum in Equation (5) by performing B 3 steps of the bisectionmethod.
Detailed pseudocode for KSDA GG is provided in Algorithm 1.
We verify in the next proposition that performing this correction indeed ensures that our aggregatedtest ∆ K α has the prescribed level α .
Proposition 3.2.
For α ∈ (0 , 1) and a collection of kernels K , the aggregated test ∆ K α satisﬁes
[Equation]
non-asymptotically using a parametric bootstrap and asymptotically using a wild bootstrap.
The proof of Proposition 3.2 is presented in Appendix B. We now provide guarantees for the powerof our aggregated test KSDA GG in terms of its uniform separation rate.
Theorem 3.3.
Let ψ :=
p − q denote the difference in densities and assume that (cid:107) p (cid:107) ∞ ≤ M and (cid:107) q (cid:107) ∞ ≤ M .
Consider the aggregated test ∆ K α with a collection of kernels K and associated positive weights ( w k ) k
∈K satisfying (cid:80)
k ∈K w k ≤ 1 , and with parameters α ∈ (0 , e − 1 ) and B 1 , B 2 , B 3 ∈ N satisfying B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1)
+ α (1 − α ) (cid:1) , B 2 ≥ 8 α 2 log (cid:0) 2 β (cid:1) and B 3 ≥ log 2 (cid:0) 4 α min k ∈K w − 1 k (cid:1) .
Consider some β ∈ (0 , 1) and C k as deﬁned in Equation (2) .
There exists a positive constant C such that if
[Equation]
then the probability of type II error of ∆ K α is controlled by β , that is, P q (cid:0) ∆ K α
[Equation]
X N ) = 0 (cid:1) ≤ β .
We prove Theorem 3.3 in Appendix C. We observe that the aggregation procedure allows to achievethe smallest uniform separation rate of the single tests (cid:0) ∆ kα (cid:1)
k ∈K up to some logarithmic weightingterm log(1 /w k ) .
3.3 Bandwidth selection
A speciﬁc application of the setting we have considered is the problem of bandwidth selection for a ﬁxed kernel.
Given a kernel k : R d × R d → R , the function
[Equation]
is also a kernel for any bandwidth λ > 0 . A common example is the Gaussian kernel, for whichwe have k ( x, y ) = exp( −(cid:107) x − y (cid:107) 22 ) and k λ ( x, y ) = exp (cid:0) − (cid:107) x − y (cid:107) 22 (cid:14) λ 2 (cid:1) .
As shown by Gorhamand Mackey (2017), a more appropriate kernel for goodness-of-ﬁt testing using the KSD is the
IMQ (inverse multiquadric) kernel, which is deﬁned with k ( x, y ) = (cid:16) 1 + (cid:107) x − y (cid:107) 22 (cid:17) − β k for a ﬁxedparameter β k ∈ (0 , 1) as
[Equation] [Equation]
which is the well-known form of the IMQ kernel with parameters λ > 0 and β k ∈ (0 , 1) .
Note that itis justiﬁed to consider the kernel up to a multiplicative constant because our single and aggregatedtests are invariant under this kernel transformation.
In practice, as suggested by Gretton et al.
(2012a) , the bandwidth is often set to a heuristic such as themedian or the standard deviation of the L 2 -distances between the samples ( X i ) Ni =1 , however, theseare arbitrary choices with no theoretical guarantees.
Another common approach proposed by Grettonet al.
(2012b) for the linear-time setting, and extended to the quadratic-time setting by Liu et al.(2020), is to resort to data splitting in order to select a bandwidth on held-out data, by maximising fora proxy for asymptotic power (see Section 4.1 for details).
Both methods were originally proposedfor the two-sample problem, but extend straightforwardly to the goodness-of-ﬁt setting.
By considering a kernel collection K Λ = { k λ : λ ∈ Λ } for a collection of bandwidths Λ , we can use our aggregated test KSDA GG to test multiple bandwidths using all the data and without resorting to arbitrary heuristics.
We now obtain an expression for the uniform separation rate of ∆ K Λ α in terms of the bandwidths λ ∈ Λ .
Corollary 3.4.
Consider α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and B 1 , B 2 , B 3 ∈ N satisfying the conditions of
Theorem 3.3 and assume that max( (cid:107) p (cid:107) ∞ , (cid:107) q (cid:107) ∞ ) ≤ M .
Given a collection Λ of positive bandwidths with associated positive weights ( w λ ) λ ∈ Λ satisfying (cid:80) λ ∈ Λ w λ ≤ 1 , we consider K Λ =
{ k λ : λ ∈ Λ } .
There exists a positive constant C such that the condition
[Equation]
ensures control over the probability of type II error of the aggregated test P q (cid:0) ∆ K Λ α ( X N ) = 0 (cid:1) ≤ β . Corollary 3.4 follows from applying Theorem 3.3 to the collection of kernels K Λ . We do not imposeany restrictions on ψ :=
p − q such as assuming it belongs to a speciﬁc regularity class. For thisreason, our result holds more generally but the dependence on λ in the terms (cid:107) ψ − T h p,kλ ψ (cid:107) 22 andlog (cid:0) 1 / ( αw λ ) (cid:1) ( βN ) − 1 (cid:112) C k λ is not explicit. For a particular regularity class, one can obtain auniform separation rate N − r for some r > 0 by choosing appropriate collections of bandwidths andweights (depending on N ) such that the two terms have matching orders of N .  Implementation and experiments
We consider three different experiments based on a Gamma one-dimensional distribution, a GaussianBernoulli Restricted Boltzmann Machine, and a Normalizing Flow for the MNIST dataset.
Wecompare our proposed aggregated test KSDA GG against three alternatives: the KSD test which usesthe median bandwidth, a test which splits the data to select an ‘optimal’ bandwidth according to aproxy for asymptotic test power, and a test which uses extra data for bandwidth selection.
The ‘extradata’ test is designed simply to provide a best case oracle for the bandwidth selection procedurewhich maximises asymptotic test power, it cannot be used in practice (i.e. any extra samples from qwould normally be incorporated into the sample being tested).
In order to ensure that our tests alwayshave correct levels for all bandwidth values, dimensions and sample sizes, we use the parametricbootstrap in our experiments.
4.1 Alternative bandwidth selection approaches
Gretton et al. (2012a) proposed to use the median heuristic as kernel bandwidth, it consists in the median of the L 2 -distances between the samples given by [Equation]
Gretton et al.
(2012b)
ﬁrst proposed, for the two-sample problem using a linear-time MMD estimator,to split the data and to use half of it to select an ‘optimal’ bandwidth which maximises a proxy forasymptotic power.
This procedure was extended to quadratic-time estimators and to the goodnessof-ﬁt framework by Jitkrittum et al.
(2017), Sutherland et al.
(2017) and Liu et al.
Thesestrategies rely on the asymptotic normality of the test statistic under H a .
In our setting, the asymptoticpower proxy to maximise is the ratio (cid:91) KSD 2 p,k ( X N ) / (cid:98) σ H a where (cid:98) σ 2 H a is a closed-form regularisedpositive estimator of the asymptotic variance of (cid:91) KSD 2 p,k under H a (Liu et al., 2020, Equation 5).
Inour experiments, we also consider a test which has access to N extra samples drawn from q to selectan ‘optimal’ bandwidth to run the KSD test on the original N samples X N .
This test is interesting tocompare to because it uses an ‘optimal’ bandwidth without being detrimental to power.
4.2 Experimental details
In our experiments, we use collections of bandwidths of the form Λ( (cid:96) − , (cid:96) + ) :
{+= (cid:8) 
+}
i λ med : i =(cid:96) − , . . .
, (cid:96) + (cid:9) for the median bandwidth λ med and integers (cid:96) − < (cid:96) + with uniform weights w λ :
=1 / ( (cid:96) + − (cid:96) − + 1) .
For the tests which split the data, we select the bandwidth out of the collectionΛ( (cid:96) − , (cid:96) + ) which maximises the power proxy discussed in Section 4.1.
All our experiments are runwith level α = 0 . 05 using the IMQ kernel deﬁned in Equation (6) with parameter β k = 0 . We usea parametric bootstrap with B 1 = B 2 = 500 simulated KSD values to compute the adjusted testthresholds, and B 3 = 50 steps of bisection method to estimate the correction u α in Equation (5). To estimate the probability of rejecting the null hypothesis, we average the test outputs across 200repetitions.
All experiments have been run on an AMD Ryzen Threadripper 3960X 24 Cores 128Gb RAM CPU at 3.8GHz, the runtime is of the order of a couple of hours (signiﬁcant speedup canbe obtained by using parallel computing). We have used the implementation of Jitkrittum et al.(2017) to sample from a Gaussian-Bernoulli Restricted Boltzmann Machine, and Phillip Lippe’simplementation of MNIST Normalizing Flows, both under the MIT license. [Figure]
For our ﬁrst experiment, we consider a one-dimensional Gamma distribution with shape parameter 5and scale parameter 5 as the model p . We draw 500 samples from a Gamma distribution with the same scale parameter 5 and with a shifted shape parameter 5 + s for s ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } . Weconsider the collection of bandwidths Λ(0 , 10) .
The results we obtained are presented in Figure 1a. We observe that all tests have the prescribedlevel 0.05 under the null hypothesis, which corresponds to the case s = 0 . As the shift parameters increases, the two densities p and q become more different and rejection of the null becomes aneasier task, thus the test power increases. Our aggregated test KSDA GG achieves the same power asthe ‘best case’ bound on the performance of the asymptotic power heuristic, yielded by the splittingtest with extra data.
The median test obtains only slightly lower power, this closeness in powercan be explained by the fact that this one-dimensional problem is a simple one. We note that thenormal splitting test has signiﬁcantly lower power: this is because, even though it uses an ‘optimal’bandwidth, it is then run on only half the data, which results in a loss of power. 4.4 Gaussian-Bernoulli Restricted Boltzmann Machine As ﬁrst considered by Liu et al.
(2016) for goodness-of-ﬁt testing using the KSD, we consider a Gaussian-Bernoulli Restricted Boltzmann Machine. It is a graphical model with a binary hidden variable h ∈ {− 1 , 1 } d h and a continuous observable variable x ∈ R d . Those variables have joint density
[Equation] where Z is an unknown normalizing constant. By marginalising over h , we obtain the density p of x [Equation]
x [Equation] = [Equation]
p
[Equation] We can sample from it using a Gibbs sampler with 2000 burn-in iterations. We use the dimensions d = 50 and d h = 40 as considered by Jitkrittum et al. (2017) and Grathwohl et al.
Even though computing p is intractable for large dimension d h , the score function admits a convenient closed form
[Equation]
We draw the components of b and c from Gaussian standard distributions and sample Rademachervariables taking values in {− 1 , 1 } for the elements of B for the model p .
We draw 1000 samplesfrom a distribution q which is constructed in a similar way as p but with the difference that
some
Gaussian noise N (0 , σ ) is injected into the elements of B .
We consider the standard deviations of theperturbations σ ∈ { 0 , 0 . 01 , 0 .
We run our experiments with the collection of bandwidthsΛ( − 20 , 0) and provide the results in Figure 1b.
Again, we observe that our aggregated test KSDA GG matches the power obtained by the test whichuses extra data to select an ‘optimal’ bandwidth.
This means that, in this experiment, KSDA GGobtains the same power as the ‘best’ single test.
The difference between KSDA GG and the medianheuristic test is signiﬁcant on this experiment, and the splitting test obtains lowest power of the fourtests.
Again, all tests have well-calibrated levels ( σ = 0 ) and increasing the noise level σ results inmore power for all the tests.
[Figure]
4.5 MNIST Normalizing Flow
Finally, we consider a high-dimensional problem working with images in dimensions 28 2 = 784 .
Weconsider a multi-scale Normalizing Flow
(Dinh et al., 2017; Kingma and Dhariwal, 2018) which hasbeen trained on the MNIST dataset (LeCun et al., 1998, 2010), it is a generative model which has a probability density p .
As observed in Figure 2, some samples produced by the model can look exactlylike MNIST digits, while other do not resemble digits.
This Normalizing Flow has been trained to‘ideally’ produce samples from the MNIST dataset.
We are interested in whether or not we can detectthe difference in densities.
Given some images of digits, are we able to tell if those were sampledfrom the Normalizing Flow model?
We consider the case where the images from q are sampled fromthe model (level experiment, conﬁrming performance under H 0 ), and the case where the samplesfrom q are drawn from the true MNIST dataset (power experiment).
The experiments are run withthe collection of bandwidths Λ( − 20 , 0) .
The results are displayed in Figure 3a and Figure 3b.
In Figure 3a, we observe that the four tests have correct levels (around 0.05) for the ﬁve differentsample sizes considered (the small ﬂuctuations about the designed test level can be explained bythe fact that we are averaging 200 test outputs to estimate these levels).
The well-calibrated levelsobtained in Figure 3a demonstrate the validity of the power results presented in Figure 3b.
In Figure 3b, we observe that only our aggregated test KSDA GG obtains high power, that is, is ableto detect that MNIST samples are not drawn from the Normalizing Flow.
The power of the othertests increases only marginally as the sample size increases.
We notice that the test which uses extradata to select an ‘optimal’ bandwidth performs poorly when compared to KSDA GG .
This could beexplained by the fact that this test selects the bandwidth using a proxy for the asymptotic power, andthat in this high-dimensional setting, the asymptotic regime is not attained with sample sizes below500.
[Figure]
Discussion
We have introduced KSDA GG , an aggregated goodness-of-ﬁt test based on the Kernel Stein Discrepancy.
We have investigated the theoretical properties of this adaptive test.
We have shown thatit achieves the desired level and have provided conditions to guarantee high power by exhibiting alower bound on its uniform separation rate.
We have observed in our experiments that KSDA GGoutperforms alternative state-of-the-art approaches to KSD kernel adaptation for goodness-of-ﬁttesting.
This work covers the problem of KSD adaptivity in the goodness-of-ﬁt framework without requiringdata splitting.
A potential future direction of interest could be to tackle the adaptivity problem ofthe KSD-based linear-time goodness-of-ﬁt test proposed by Jitkrittum et al.
In this setting,the data is split to select feature locations (and the kernel bandwidth), the KSD test is then run usingthose adaptive features.
A challenging problem would be to obtain those adaptive features using anaggregation procedure which avoids splitting the data.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendices.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] See link in
Section 1: https://anonymous.4open.science/r/ksdagg-DBF7/README.md .
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] Since the test outputs are binary (0 or 1), there is no need
to include error bars since these are deterministic given the average which is plotted.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.2.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.2.
(b) Did you mention the license of the assets?
[Yes] See Section 4.2.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See link in Section 1: https://anonymous.4open.science/r/ksdagg-DBF7/README.md .
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A
{+Proof of Theorem 3.
 Note that [Equation] where+}
[Equation] since (cid:91) KSD 2 p,k is an unbiased estimator. By Chebyshev’s inequality (Chebyshev, 1899), we know that [Equation]
  . We deduce that P q (cid:0) ∆ kα ( X N ) = 0 (cid:1) ≤ β if [Equation]
The condition in Equation (7), which controls the probability of type I error of ∆ kα , needs to holdwith high probability since (cid:98) q k 1 − α depends on the randomness of the samples X N drawn from q andon the randomness of the Rademacher random variables in Equation (4).
We want to derive a condition in terms of (cid:107) p − q (cid:107) 22 rather than in terms of KSD 2 p,k ( q ) as in Equation (7).
For this, using Stein’s identity, we obtain
[Equation]
which gives
[Equation]
To guarantee P q (cid:0) ∆ kα ( X N ) = 0 (cid:1) ≤ β , an equivalent condition to the one presented in Equation (7) isthen
[Equation]
which needs to hold with high probability over the randomness in (cid:98) q k 1 − α .
We now upper bound thequantile and variance terms in Equation (8) to obtain a new condition ensuring control of the type IIerror.
To bound the quantile term (cid:98) q k 1 − α using B 1 wild bootstrapped statistics, using the Dvoretzky–Kiefer–
Wolfowitz inequality (Dvoretzky et al., 1956; Massart, 1990) as done by Schrab et al.
(2021, Appendix
E.4), it is sufﬁcient to bound the quantile (cid:98) q k, ∞ 1 − α using wild bootstrapped statistics without ﬁnite approximation provided that B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) .
Using the concentration bound for i.i.d.
Rademacher chaos of de la Peña and Giné (1999, Corollary 3.2.6) as presented by Kim et al.
Equation 39)
, there exists some constant C > 0 such that
[Equation]
 
[Equation]
N
[Equation]
(cid:88)
[Equation]

 ≤ 2 exp   −
CtN ( N − 1) (cid:115) (cid:88) 1 ≤ i (cid:54)
[Equation]
N h p
[Equation]
k ( X i
[Equation]
X
[Equation]
  which gives
[Equation]
for some different constant C > 0 . Using Markov’s inequality, for any δ ∈ (0 , 1) , with probability at least 1 − δ we have [Equation] which gives
[Equation]
N since α ∈ (0 , e − 1 ) , where in the last inequality C > 0 is a different constant depending on δ .
Using
Dvoretzky–Kiefer–Wolfowitz inequality (Dvoretzky et al., 1956) as explained above, we deduce that [Equation] with arbitrarily high probability, for some constant C > 0 . We now bound the variance term in Equation (7).
Using the result of Albert et al. (2022, Equation 6), there exists C > 0 such that [Equation] where
[Equation]
and [Equation] since [Equation]
by Stein’s identity.
We deduce that
[Equation]
for some different constant C > 0 depending on M . Using the classical inequalities √ x + y ≤ √ x + √ y and 2 √ xy ≤ x + y for x [Equation] y >
[Equation]
which are alsoconsidered in the works of Fromont et al. (2013), Albert et al.
(2022) and Schrab et al. (2021), weobtain
[Equation]
since α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and C k ≥ 1 by assumption, 2 where the constant C > 0 is different on the last line. By considering two times the largest of the two constants in the quantile and variance bounds of Equations (9) and (10), we obtain [Equation]
By combining this bound with the condition in Equation (8), we get that P q (cid:0) ∆ kα ( X N ) = 0 (cid:1) ≤ β if [Equation]
which concludes the proof. B Proof of Proposition 3.
Recall that the correction term in Equation (5) is deﬁned as [Equation] (cid:27) where (cid:98) q k
[Equation] = [Equation] k as deﬁned in Section 3.1.
Hence, we have
[Equation]
≤ α. Recall that our estimator is (cid:91) KSD 2 p,k ( X N ) where X N := ( X i ) Ni =1 are drawn from q .
Recall also from
Equation (3), that for the parametric bootstrap the element (cid:101) K bk is computed by drawing new samples( X (cid:48)
i ) N (cid:48)
i =1 from the model p and computing (cid:91) KSD 2 p,k (cid:16) ( X (cid:48) i ) N (cid:48) i =1 (cid:17) . Hence, at every sample size, underthe null hypothesis H 0 : p = q the quantities (cid:101) K bk and (cid:91) KSD 2 p,k ( X N ) are identically distributed.
Forthe wild bootstrap, as deﬁned in Equation (4), Chwialkowski et al. (2014, 2016) show that, under H 0 ,(cid:101) K bk and (cid:91) KSD 2 p,k ( X N ) have the same asymptotic distribution. Therefore, by taking the expectation in Equation (11), we obtain
[Equation]
which holds non-asymptotically using the parametric bootstrap, and asymptotically using the wildbootstrap
Note that this is different from the two-sample case where using a wild bootstrap yieldswell-calibrated non-asymptotic levels (Schrab et al., 2021, Proposition 8).
The same reasoning holds by replacing u α by any value u ∈ (0 , u α ) . In particular, it holds for thelower bound (cid:98) u α of the interval obtained by performing the bisection method to approximate thesupremum in the deﬁnition of u α . This lower bound (cid:98) u
α is the one we use in practice, as shownin Algorithm 1 with the step ‘ u α = u min ’.
We have proved that the test with correction term u αapproximated with a bisection method also achieves the desired level α .
C Proof of Theorem 3.3
As explained in Appendix B and utilised in Algorithm 1, we use the lower bound (cid:98) u
α of the intervalobtained by performing the bisection method to approximate the supremum in the deﬁnition of u α in
Equation (5).
The assumptions B 2 ≥ 8 α 2 log (cid:0) 2 β (cid:1) and B 3 ≥ log 2 (cid:0) 4 α min k ∈K w − 1 k (cid:1) ensure that (cid:98) u
α ≥ α/ 2as shown by Schrab et al.
(2021, Appendix E.9).
The probability of type II error of ∆ K α is
[Equation]
In order to guarantee P q (cid:0) ∆ K α ( X N ) = 0 (cid:1) ≤ β it is sufﬁcient to guarantee P q (cid:16) ∆ k ∗ αw k ∗ / 2 ( X N ) = 0 (cid:17) ≤β for some k ∗ ∈ K to be speciﬁed shortly in Equation (13).
By assumption, 3 we have [Equation] In order to apply Theorem 3.1 to ∆ kα k with α k := αw k / 2 for k ∈ K , we need to ensure that the condition on B 1 of Theorem 3.1 is satisﬁed for all k ∈ K , that is
[Equation] for k ∈ K . Since 0 < α k < α < e − 1 , we have α (1 − α ) ≥ α k (1 − α k ) for k ∈ K . We also have
[Equation] for k ∈ K . We deduce that [Equation]
for k ∈ K , and so, applying Theorem 3.1 to ∆ kα k for k ∈ K is justiﬁed. We obtain that P q
[Equation]
if
[Equation]
or, with a different constant C > 0 , if
[Equation] since log (cid:16) 2 αw k (cid:17) ≤ (log(2) + 1) log (cid:16) 1 αw k (cid:17) as α ∈ (0 , e − 1 ) and w k ∈ (0 , 1) .
Now, let
[Equation]
argmin [Equation]
[Equation] Finally, we have P q (cid:0) ∆ K α
[Equation]
N
[Equation]
[Equation]
β , that is, if
[Equation] as desired.
In this section, we prove minimax optimality of KSDA GG over Sobolev balls when the followingassumptions hold. • The model density p is strictly positive on its compact support S ⊆ R d .
• The score function ∇ log p ( x ) is continuous on S .• The support of density q is a subset of S .•
The kernel used is a scaled Gaussian kernel (deﬁned below).
In particular, any strictly positive twice-differentiable density truncated to some d -dimensional intervalwill satisfy the assumptions for the model p . Any density truncated to the same d -dimensional intervalwill satisfy the assumption for the density q . As an example, one can consider truncated normaldensities.
We introduce some notation.
For some c ∈ R , we write c := ( c, . . . , c ) ∈ R d . We write( a 1 , . . .
, a d ) ≤ ( b 1 , . . . , b d ) when a i ≤ b i for i = 1 , . . .
, d .
We use C, C (cid:48) to denote some genericconstants which may change on different lines.
Note that, by properties of compactness on R d , there exists some a > 0 such that S ⊆ (cid:2) − a 2 , a 2 (cid:3) d .
Since the score function ∇ log p ( x ) is continuous on S which is compact, there exists some c 1 >such that
[Equation]
for i = 1 , . . .
, d and for all x ∈ S.
We work with a scaled Gaussian kernel with bandwidth λ ≤
[Equation]
which satisﬁes
[Equation]
The Stein kernel associated to k λ satisﬁes
[Equation]
(cid:33)
where the constant C 0 :
= 2 π d/ 2 ( dc 21 + 2 dc 1 a + d ) .
The Stein kernel can be upper bounded by a scaled Gaussian kernel on S .
Writing
[Equation]
which is of the form considered by Schrab et al.
(2021, Section 3.1) with equal bandwidths for eachdimension, we have shown that
[Equation]
all x
[Equation] [Equation] Recall from Theorem 3.1 that a sufﬁcient condition for control of the probability of type II error by β is [Equation]
Using the upper bound
[Equation]
where the last inequality holds as in Schrab et al.
(2021, Equation (22)), we obtain the sufﬁcientcondition
[Equation]
where we have absorbed the term C 0 √ M in the constant C .
We would like to upper bound the term (cid:13)(cid:13)(cid:13) ψ − T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 by (cid:13)(cid:13)
{+ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
+}
but this is not possibleusing Equation (14).
Instead, we can use the triangle inequality to get
[Equation]
{+,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 ≤ (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
+}
+ (cid:13)(cid:13)(cid:13) T ¯ k λ ψ − T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2≤ (cid:13)(cid:13)
{+ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
+}
+ (cid:13)(cid:13)(cid:13) T ¯ k λ ψ + T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2≤ (cid:13)(cid:13)
{+ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
+}
+ ( C 0 + 1) 2 (cid:13)(cid:13)
T ¯ k λ ψ (cid:13)(cid:13) 2since (cid:0) T h p,kλ ψ (cid:1) (
[Equation]
) = (cid:90)
S h p,k λ ( x, y ) ψ ( x ) d x ≤ (cid:90)
S ¯ k λ ( x, y ) ψ ( x ) d x = (cid:0) T h p,kλ ψ (cid:1) ( y ) for all y ∈ S .
For the term (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 , we assume that ψ ∈ S sd ( R ) where
[Equation]
is the Sobolev ball of radius R > 0 with smoothness parameter s > 0 , where (cid:98) f denotes the Fourier transform of f .
The term (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 can then be upper bounded exactly as done by Schrab et al.
(2021, Appendix E.6) with the difference that we choose t > 0 such that S 1 < 0 .
5 rather than S 1 <( S 1 is deﬁned in Schrab et al.
(2021, Appendix E.6)).
Following their reasoning, assuming that ψ ∈ S sd ( R ) , we obtain that there exists some S 1 ∈ (0 , 0 . 5) and some constant C > 0 (depending on d, s, R ) such that
[Equation]
To upper bound the term (cid:13)(cid:13)
T ¯ k λ ψ (cid:13)(cid:13) 2 2 , we modify the reasoning of Schrab et al.
(2021, Appendix E.6) used for the term (cid:13)(cid:13)
ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 .
First, note that since ¯ k λ is translation-invariant, T ¯ k λ is a convolution as
[Equation]
where ϕ
[Equation]
i = λ − 1 K (cid:16) u
[Equation]
λ (cid:17) , and
K ( u ) :
= π − 1 / 2 exp( − u 2 ) . By properties of Fourier transforms, we have (cid:99) ϕ λ ( ξ ) = (cid:81) di =1 (cid:98) K ( λξ i ) for ξ ∈ R d .
Note that
[Equation]
Choose t > 0 such that
[Equation]
For s > 0 , deﬁne
[Equation]
Then, using Plancherel’s Theorem as in Schrab et al.
(2021, Appendix E.6), we obtain
[Equation]
[Equation]
ξ
[Equation]
(cid:89)
i
=
(cid:98)
[Equation]
( λξ i ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:98) ψ ( ξ ) (cid:12)(cid:12)(cid:12) 2 d ξ = (cid:90) (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:89)
i
=
(cid:98)
[Equation]
( λξ
[Equation]
) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:98)
[Equation]
( ξ
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
ξ
[Equation]
(cid:90) (cid:107) ξ (cid:107)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d (cid:89) i
= (cid:98) K
( λξ i ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:98) ψ ( ξ ) (cid:12)(cid:12)(cid:12)
[Equation]
d
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
Combining those results, the upper bound on (cid:13)(cid:13)(cid:13) ψ − T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 becomes
[Equation]
Equation (15) then becomes
[Equation]
which gives [Equation] Having proved the power guaranteeing condition in Equation (16), we can then derive rates for the KSD and KSDAgg tests as similarly done by Schrab et al.
Set
λ :
=
N − 2 / (4 s + d ) to get thecondition
[Equation]
{+which is the minimax rate over Sobolev balls (Balasubramanian et al., 
+}
Albert et al., 2022;
Schrab et al., 2021).
In practice, the smoothness parameter of the Sobolev ball is not known, so we cannot set λ :=
N − 2 / (4 s + d ) (i.e. it cannot be implemented). Instead, we can use our aggregated test KSDAgg. Adapting the proof of Theorem 3.3 with the single condition Equation (16), we obtain the aggregatedcondition [Equation]
Similarly to Schrab et al. (2021, Corollary 10), consider [Equation] Let
[Equation] The bandwidth λ ∗ := 2 − (cid:96) ∗ ∈ Λ satisﬁes [Equation] ≤ C ln(
[Equation]
C ln(ln( N )) as w λ ∗ := 6 π − 2 ( (cid:96) ∗ ) − 2 , and [Equation] N [Equation]
N
[Equation]
N
[Equation]
N
[Equation]
We get [Equation] N [Equation]
+
[Equation]
) and log
[Equation]
The power condition Equation (17) becomes
[Equation]
We have proved that, in the compactly supported setting, KSDAgg is minimax optimal and adaptiveover the Sobolev balls {S sd ( R ) : s > 0 , R > 0 } , up to an iterated logarithmic term.