KSD Aggregated Goodness-of-ﬁt Test
Abstract
We investigate properties of goodness-of-ﬁt tests based on the Kernel Stein Discrepancy (KSD).
We introduce a strategy to construct a test, called KSDA GG , which aggregates multiple tests with different kernels.
KSDA GG avoids splitting the data to perform kernel selection (which leads to a loss in test power), and rather maximises the test power over a collection of kernels.
We provide theoretical guarantees on the power of KSDA GG : we show it achieves the smallest uniform separation rate of the collection, up to a logarithmic term.
KSDA GG can be computed exactly in practice as it relies either on a parametric bootstrap or on a wild bootstrap to estimate the quantiles and the level corrections.
In particular, for the crucial choice of bandwidth of a ﬁxed kernel, it avoids resorting to arbitrary heuristics (such as median or standard deviation) or to data splitting.
We ﬁnd on both synthetic and real-world data that KSDA GG outperforms other state-of-the-art adaptive KSD-based goodness-of-ﬁt testing procedures.
Introduction
Kernel selection remains a fundamental question in kernel-based nonparametric hypothesis testing, as it signiﬁcantly impacts the test power.
Kernel selection has attracted a signiﬁcant interest in the literature and a number of methods have been proposed in different settings, such as in the two-sample, independence and goodness-of-ﬁt testing frameworks.
Those methods include using heuristics (Gretton et al., 2012a), relying on data splitting (Gretton et al., 2012b;
Sutherland et al., 2017; Kübler et al., 2022), learning deep kernels (Grathwohl et al., 2020; Liu et al., 2020), working in the post-selection inference framework (Yamada et al., 2019; Lim et al., 2019, 2020; Kübler et al., 2020;
Freidling et al., 2021), to name but a few.
In this work, we focus on aggregated tests, which have been investigated for the two-sample problem by Fromont et al.
(2013), Kim et al.
(2022) and Schrab et al.
(2021) using the Maximum Mean
Discrepancy (MMD, Gretton et al., 2012a), and for the independence problem by Albert et al. (2022) and Kim et al.
using the Hilbert Schmidt Independence Criterion (HSIC, Gretton et al., 2005).
We extend the use of aggregated tests to the goodness-of-ﬁt setting, where we are given a model and some samples, and we are interested in deciding whether the samples have been drawn from the model.
We employ the Kernel Stein Discrepancy (KSD, Chwialkowski et al., 2016;
Liu et al., 2016) as our test statistic, which is an ideal measure of distance for this setting: it admits an estimator which can be computed without requiring samples from the model, and does not require the model to be normalised.
To the best of our knowledge, ours represents the ﬁrst aggregation procedure for the
KSD test in the literature.
Related work.
Fromont et al.
(2012, 2013) introduced non-asymptotic aggregated tests for the two-sample problem with the equal sample sizes following a Poisson process, using as test statistic an unscaled version of the MMD.
They also provided theoretical results in terms of uniform separation rates using a wild bootstrap.
Albert et al.
(2022) then proposed an aggregated test for the independence problem using the HSIC, with guarantees using a theoretical quantile, but relying on permutations to obtain the test threshold in practice.
Kim et al.
(2022) then extended those results to also hold for the estimated quantile, and generalised the two-sample results to hold for the MMD estimator with different sample sizes using a wild bootstrap.
All those aforementioned results were proved for the
Gaussian kernel only.
Schrab et al.
(2021) generalised the two-sample results to hold for a wide range of kernels using either a wild bootstrap or a permutation-based procedure, and provided optimality results which hold with fewer restrictions on the class of functions.
Our work builds and extends on the above results: we consider the goodness-of-ﬁt framework, where we have samples from only one of the two densities.
The main challenges arise from working with the Stein kernel that deﬁnes the KSD test statistic: for example, we lose the transition invariant property of the kernel which is crucial to work in the Fourier domain.
We also point out the very relevant work of Balasubramanian et al.
(2021) who considered adaptive MMD-based goodness-of-ﬁt tests and studied their uniform separation rates over Sobolev balls in the asymptotic regime.
More generally, Li and Yuan (2019) studied asymptotic adaptive kernel-based tests for the three testing frameworks.
Finally, Key et al. (2021) addressed the complementary task of Stein test design for a family of models, rather than for a single model.
Contributions.
We propose a solution to the fundamental kernel selection problem for the widelyused KSD goodness-of-ﬁt tests: we construct an adaptive test KSDA [Equation] which aggregates multiple tests with different kernels.
Our contribution is in showing, both theoretically and experimentally, that the aggregation procedure works in this novel setting in which it has never been considered before.
We consider the kernel selection framework; this general setting has many applications including the one of kernel bandwidth selection.
Our aggregated test allows for two numerical methods for estimating the test thresholds: the wild bootstrap and the parametric bootstrap (a procedure unique to the goodness-of-ﬁt framework).
We conduct a theoretical analysis: we provide a lower bound on the uniform separation rate (Baraud, 2002) of KSDA GG , a condition which guarantees test power.
We discuss the implementation of KSDA GG and experimentally validate our proposed approach on benchmark problems, not only on datasets classically used in the literature but also on original data obtained using state-of-the-art generative models (i.e. Normalizing Flows).
We observe, both on synthetic and real-world data, that KSDA GG obtains higher power than other
KSD-based adaptive state-of-the-art tests.
Contributing to the real-world applications of these goodness-of-ﬁt tests, we provide publicly available code to allow practitioners to employ our method: https://github.com/antoninschrab/ksdagg-paper .
Outline.
Section 2 presents our framework and our notation.
Section 3 introduces our algorithm
KSDA GG (in Algorithm 1) and contains our main theoretical results.
Section 4 presents numerical experiments to support KSDA GG .
Notation
We consider the goodness-of-ﬁt problem where given access to a known model probability density p (which can be unnormalised since we only need its score function ∇ log p ( · ) ) and to some i.i.d.
d -dimensional samples X N :=
( X i )
Ni =1 drawn from an unknown density q , we want to decide whether p (cid:54) =
q holds.
This can be expressed as a statistical hypothesis testing problem with null hypothesis H 0 : p = q and alternative H a : p (cid:54) = q .
As a measure of distance between p and q , we use the Kernel Stein Discrepancy (KSD) introduced by
Chwialkowski et al.
(2016) and Liu et al.
For a kernel k , the KSD is deﬁned as the Maximum
Mean Discrepancy (MMD, Gretton et al., 2012a) between p and q using the Stein kernel associated to k , that is
[Equation]
where the Stein kernel h p,k : R d × R d → R is deﬁned as
[Equation]
and satisﬁes the Stein identity E p [ h p,k ( X, · )]
= 0 .
A quadratic-time KSD estimator can be computed as the U -statistic
(Hoeffding, 1992)
[Equation]
In this work [Equation] the model density p is always known, we do not always explicitly write the dependence of p for all variables (as we do for KSD 2 p,k and h p,k ).
We assume that the kernel k is such that
[Equation]
and
[Equation]
We now address the requirements for consistency of the Stein test (Chwialkowski et al., 2016,
Theorem 2.2): we assume that the kernel k is C 0 -universal
(Carmeli et al., 2010, Deﬁnition 4.1) and that E q (cid:13)(cid:13)(cid:13) ∇ (cid:16) log p ( X ) q ( X ) (cid:17)(cid:13)(cid:13)(cid:13) 2 2 < ∞ .
We use the notations P p and P q to denote the probability under the model p and under q , respectively.
Given a kernel κ : R d × R d → R and a function f : R d → R in L 2 (cid:0) R d (cid:1) , we consider the integral transform T κ deﬁned as
[Equation]
for y ∈ R d .
When the kernel κ is translation invariant, the integral transform corresponds to a convolution, however, this is not true of the Stein kernel.
Construction of tests and bounds
We now introduce the single and aggregated KSD tests.
We show that these control the probability of type I error as desired, and provide conditions for the control of the probability of type II error.
3.1 Single test
We ﬁrst construct a KSD test for a ﬁxed kernel k as proposed by Chwialkowski et al.
(2016) and Liu et al.
To estimate the test threshold, we can either use a wild bootstrap (Shao, 2010; Leucht and Neumann, 2013; Fromont et al., 2012; Chwialkowski et al., 2014) or a parametric bootstrap (Stute et al., 1993).
Both methods work by simulating sampling values (cid:0) ¯ K 1 k , . . .
, ¯ K B 1 k (cid:1) from the (asymptotic) distribution of (cid:91) KSD 2 p,k under the null hypothesis and estimating the (1 − α ) -quantile using a Monte Carlo approximation[-
-]
[Equation]
where ¯ K • 1 k ≤ · · · ≤ ¯ K • B 1 k are the sorted elements (cid:0)
¯ K 1 k , . .
, ¯ K B 1 k (cid:1) .
The single test is then deﬁned as
[Equation]
For the parametric bootstrap , we directly draw new samples ( X (cid:48) i ) N (cid:48) i
=1 from the model p (it might not always be possible to so) and compute the KSD
[Equation]
For the wild bootstrap , we ﬁrst generate n i.i.d.
Rademacher random variables (cid:15) 1 , . . .
, (cid:15) n taking values in {− 1 , 1 } n , and then compute
[Equation]
Both these processes are then repeated B 1 times.
Since it uses samples from the model p , the parametric bootstrap (Stute et al., 1993) results in a test with non-asymptotic level α .
This comes at the cost of being computationally more expensive and assuming that we are able to sample from p (which may be out of reach in some settings).
Conversely, the wild bootstrap has the advantage of not requiring to sample from p , which makes it computationally more efﬁcient as only one kernel matrix needs to be computed, but it only achieves the desired level α asymptotically (Shao, 2010; Leucht and Neumann, 2013; Leucht, 2012; Chwialkowski et al., 2014, 2016) assuming Lipschitz continuity of h p,k
(see Appendix A for details). Note that we cannot obtain a non-asymptotic level for the wild bootstrap by relying on the result of Romano and Wolf (2005, Lemma 1) as done in the two-sample framework by Fromont et al.
(2013) and Schrab et al.
This is because in our case ¯ K k and (cid:91) KSD 2 p,k ( X N ) are not exchangeable variables under the null hypothesis, due to the asymmetry of the KSD statistic with respect to p and q .
Having discussed control of the probability of type I error of the single test ∆ kα , we now provide a condition on (cid:107) p − q (cid:107) 2 which ensures that the probability of type II error is controlled by some β ∈ (0 , 1) .
The smallest such value of (cid:107) p − q (cid:107) 2 , provided that p − q lies in some given class of functions, is called the uniform separation rate (Baraud, 2002).
Theorem 3.1.
Let ψ :=
p − q and assume that max( (cid:107) p (cid:107) ∞ , (cid:107) q (cid:107) ∞ ) ≤ M .
Consider C k as deﬁned in
Equation (2) , α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and B 1 ∈ N satisfying B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) .
There exists a positive constant C (depending on M and d ) such that the condition
[Equation]
guarantees control over the probability of type II error, such that P q (cid:0) ∆ kα ( X N )
= 0 (cid:1) ≤ β .
Theorem 3.1, which is proved in Appendix E.1, provides a power guaranteeing condition consisting of two terms.
The ﬁrst term (cid:107) ψ − T h p,k ψ (cid:107) 22 indicates the size of the effect of the Stein operator on the difference in densities ψ :=
p − q , and is a measure of distance from the null (where this quantity is zero).
The second term log (cid:0) 1 /α (cid:1) ( βN ) − 1 √ C k is obtained from upper bounding the variance of the
KSD U -statistic, it depends on the expectation of the squared Stein kernel C
k :=
E q,q [ h p,k ( X, Y ) 2 ] .
This second term also controls the quantile of the test.
3.2 Aggregated test
We can now introduce our aggregated test, which is motivated by the earlier works of Fromont et al. (2012, 2013), Albert et al.
(2022), and Schrab et al.
(2021) for different testing frameworks.
We compute (cid:101) K 1 k , . . .
, (cid:101) K B 2 k further simulated KSD values from the null hypothesis obtained using either a parametric bootstrap or a wild bootstrap as in Equations (3) or (4), respectively.
Consider a ﬁnite collection of kernels K satisfying the properties presented in Section 2.
We construct an aggregated test ∆ K α , called KSDA GG , which rejects the null hypothesis if one of the single tests (cid:0) ∆ ku α w k (cid:1) k ∈K rejects it, that is
[Equation]
The levels of the single tests are adjusted to ensure the aggregated test has the prescribed level α .
This adjustment is performed by introducing positive weights ( w k )
k
∈K satisfying (cid:80)
k ∈K w k ≤ 1 and some correction
[Equation]
Inputs: samples X N = ( x i ) Ni =1 , density p or score function ∇ log p ( · ) , ﬁnite kernel collection K , weights ( w k ) k ∈K , level α ∈
(0 , e − 1 ) , estimation parameters B 1 , B 2 , B 3 ∈ N , parametric or wild bootstrap
Output: 0 (fail to reject H 0 ) or 1 (reject H 0 )
Algorithm: for k ∈ K do
Compute (cid:0)
¯ K bk (cid:1) 1 ≤ b ≤ B 1 as in Equations (3) or (4)
Sort in ascending order to obtain (cid:0) ¯ K • bk (cid:1) 1 ≤ b ≤ B
Compute (cid:0) (cid:101) K bk (cid:1) 1 ≤ b ≤ B 2 as in Equations (3) or (4) u min = 0 , u max = min
k ∈K w − 1 k for t = 1 , . . .
, B 3 do u = 12 ( u [Equation] + u [Equation]
, (cid:98) P [Equation] = [Equation]
B
[Equation]
(cid:88)
[Equation]
1 (cid:18) max k ∈K (cid:16)(cid:101) [Equation] •(cid:100) B 1 (1 − uw k ) (cid:101) k (cid:17) > 0 (cid:19) if (cid:98) P u ≤ α then u min =
u else u max = u
u α = u min if max k ∈K (cid:16) (cid:91)
KSD 2 p,k ( X N ) − ¯ K •(cid:100)
B 1 (1 − u α w k ) (cid:101) k (cid:17) > 0 then return 1 else return
Time complexity: O (cid:0) N 2 ( B 1 + B 2 ) |K| (cid:1) where [Equation]
is a Monte Carlo approximation of the probability of type I error of our aggregated test with correction u
[Equation] To compute u α , we estimate the supremum in Equation (5) by performing B 3 steps of the bisection method.
Detailed pseudocode for KSDA GG is provided in Algorithm 1.
We verify in the next proposition that performing this correction indeed ensures that our aggregated test ∆ K α has the prescribed level α .
Proposition 3.2.
For α ∈ (0 , 1) and a collection of kernels K , the aggregated test ∆ K α satisﬁes
[Equation]
non-asymptotically using a parametric bootstrap and asymptotically using a wild bootstrap (with
Lipschitz continuity of h p,k required).
The proof of Proposition 3.2 is presented in Appendix E.2. Details about the wild bootstrap case are presented in Appendix A. We now provide guarantees for the power of our aggregated test KSDA GG in terms of its uniform separation rate.
Theorem 3.3.
Let ψ :=
p − q denote the difference in densities and assume that (cid:107) p (cid:107) ∞ ≤ M and (cid:107) q (cid:107) ∞ ≤ M .
Consider the aggregated test ∆ K α with a collection of kernels K and associated positive weights ( w k ) k
∈K satisfying (cid:80)
k ∈K w k ≤ 1 , and with parameters α ∈ (0 , e − 1 ) and B 1 , B 2 , B
3 ∈ N satisfying B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1)
+ α (1 − α ) (cid:1) , B 2 ≥ 8 α 2 log (cid:0) 2 β (cid:1) and B 3 ≥ log 2 (cid:0) 4 α min k ∈K w − 1 k (cid:1) .
Consider some β ∈ (0 , 1) and C k as deﬁned in Equation (2) .
There exists a positive constant C (depending on M and d ) such that if
[Equation]
then the probability of type II error of ∆ K α is controlled by β , that is, P q (cid:0) ∆ K α ( X N ) = 0 (cid:1) ≤ β .
The proof Theorem 3.3 can be found in Appendix E.3, it relies on upper bounding a probability of intersections of events by the minimum of the probabilities of each event, on checking that the assumptions of Theorem 3.1 are satisﬁed for the adjusted levels, and on applying Theorem 3.1.
We observe that the aggregation procedure allows to achieve the smallest uniform separation rate of the single tests (cid:0) ∆ kα (cid:1) k ∈K up to some logarithmic weighting term log(1 /w k ) .
3.3 Bandwidth selection
A speciﬁc application of the setting we have considered is the problem of bandwidth selection for a ﬁxed kernel.
Given a kernel k : R d × R d → R , the function
[Equation]
is also a kernel for any bandwidth λ > 0 .
A common example is the Gaussian kernel, for which we have k ( x, y ) =
exp( −(cid:107) x − y (cid:107) 22 ) and k λ ( x, y ) =
exp (cid:0) − (cid:107) x − y (cid:107) 22 (cid:14) λ 2 (cid:1) .
As shown by Gorham and Mackey (2017), a more appropriate kernel for goodness-of-ﬁt testing using the KSD is the
IMQ (inverse multiquadric) kernel, which is deﬁned with k ( x, y ) = (cid:16) 1 + (cid:107) x − y (cid:107) 22 (cid:17) − β k for a ﬁxed parameter β k ∈ (0 , 1) as
[Equation]
which is the well-known form of the IMQ kernel with parameters λ > 0 and β k ∈ (0 , 1) .
Note that it is justiﬁed to consider the kernel up to a multiplicative constant because our single and aggregated tests are invariant under this kernel transformation.
In practice, as suggested by Gretton et al.
(2012a)
, the bandwidth is often set to a heuristic such as the median or the standard deviation of the L 2 -distances between the samples ( X i )
Ni =1 , however, these are arbitrary choices with no theoretical guarantees.
Another common approach proposed by Gretton et al.
(2012b) for the linear-time setting, and extended to the quadratic-time setting by Liu et al. (2020), is to resort to data splitting in order to select a bandwidth on held-out data, by maximising for a proxy for asymptotic power (see Section 4.1 for details).
Both methods were originally proposed for the two-sample problem, but extend straightforwardly to the goodness-of-ﬁt setting.
By considering a kernel collection K Λ = { k λ : λ ∈ Λ } for a collection of bandwidths Λ , we can use our aggregated test KSDA GG to test multiple bandwidths using all the data and without resorting to arbitrary heuristics.
We now obtain an expression for the uniform separation rate of ∆ K Λ α in terms of the bandwidths λ ∈ Λ .
Corollary 3.4.
Consider α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and B 1 , B 2 , B 3 ∈ N satisfying the conditions of
Theorem 3.3 and assume that max( (cid:107) p (cid:107) ∞ , (cid:107) q (cid:107) ∞ ) ≤ M .
Given a collection Λ of positive bandwidths with associated positive weights ( w λ ) λ ∈ Λ satisfying (cid:80) λ ∈ Λ w λ ≤ 1 , we consider K Λ =
{ k λ : λ ∈ Λ } .
There exists a positive constant C (depending on M and d ) such that the condition
[Equation]
ensures control over the probability of type II error of the aggregated test P q (cid:0) ∆ K Λ α ( X N )
= 0 (cid:1) ≤ β .
Corollary 3.4 follows from applying Theorem 3.3 to the collection of kernels K Λ .
Our results hold with great generality as we have not imposed any restrictions on ψ :=
p − q such as assuming it belongs to a speciﬁc regularity class.
For this reason, the dependence on λ in the terms (cid:107) ψ − T h p,kλ ψ (cid:107) 22 and log (cid:0) 1 / ( αw λ ) (cid:1) ( βN ) − 1 (cid:112) C k λ is not explicit. In Section 3.4, we characterise this dependence with regularity assumptions on ψ := p − q , and derive uniform seperation rates in terms of the sample size.
3.4 Uniform separation rates over restricted Sobolev balls
In this section, we derive uniform separation rates with stronger assumptions on p and q . This provides settings in which the power guaranteeing conditions of Theorems 3.1 and 3.3 are satisﬁed, and illustrates the interactions between the two terms in those conditions. We make the following assumptions. • The model density p is strictly positive on its connected and compact support S ⊆ R d .
The score function ∇ log p ( x ) is continuous and bounded on S .
• The support of the density q is a connected and compact subset of S .
The kernel used is a scaled Gaussian kernel k λ ( x, y ) :
= λ 2 − d exp (cid:16) −(cid:107) x − y (cid:107) 22 / λ 2 (cid:17) .
In particular, any strictly positive twice-differentiable density truncated to some d -dimensional interval will satisfy the assumptions for the model p .
Any density truncated to the same d -dimensional interval will satisfy the assumption for the density q .
As an example, one can consider truncated normal densities.
For smoothness parameter s > 0 , radius R > 0 and dimension d ∈ N \ { 0 } , the Sobolev ball S sd ( R ) is deﬁned as the function space
[Equation]
where (cid:98) f denotes the Fourier transform of f .
For s, t, d, R, L all strictly positive, we deﬁne the restricted Sobolev ball S s,t d ( R, L ) as containing all functions f ∈ S s d ( R ) satisfying
[Equation]
With the Sobolev assumption p − q ∈ S sd ( R ) , the densities p and q can differ at any frequencies.
The restricted Sobolev assumption p − q ∈ S s,td
( R, L ) does not include the case in which the densities p and q differ only at low frequencies due to the additional restriction in Equation (7).
Theorem 3.5.
Suppose that the four assumptions on p , q and k λ listed above hold and assume that max( (cid:107) p (cid:107) ∞ , (cid:107) q (cid:107) ∞ ) for some M > 0 . We show that there exists some L > 0 such that if p − q ∈ S s,td ( R, L ) then statements (i) & (ii) hold. The constants are independent of the sample size. (i) Under the assumptions of Theorem 3.1, for the KSD test with bandwidth λ :=
N − 2 / (4 s + d ) , the condition
[Equation]
guarantees control over the probability of type II error, so that P q (cid:0) ∆ λα
[Equation]
X N ) = 0 (cid:1) ≤ β.
(ii) Under the assumptions of Theorem 3.3, for KSDAgg with the collection
[Equation]
N
[Equation]
(cid:17)(cid:109)(cid:111)(cid:111) and weights w λ := 6 /π 2 (cid:96) 2 , we have P q (cid:0) ∆ Λ α ( X N ) = 0 (cid:1) ≤ β provided that (cid:107) ψ (cid:107)
[Equation]
(cid:18)
N ln(ln( N ))
(cid:19)
[Equation]
4 s/ (4 s + d ) .
The proof of Theorem 3.5 is presented in Appendix E.4.
The uniform separation rate N − 4 s/ (4 s + d ) in
Theorem 3.5 is known to be optimal in the minimax sense over (unrestricted)
Sobolev balls S sd ( R ) (Li and Yuan, 2019; Balasubramanian et al., 2021;
Albert et al., 2022; Schrab et al., 2021).
However, this rate is for the KSD test with bandwidth depending on the smoothness parameter s > 0 which is unknown in practice.
Note that even methods which split the data to perform kernel selection are not able to select this bandwidth depending on s and cannot achieve the minimax rate.
The aggregation procedure is adaptive to this unknown parameter s : the collection of bandwidths Λ does not depend on s (it can be implemented in practice).
The price to pay for this adaptivity is only an iterated logarithmic factor in the minimax rate over Sobolev balls S sd ( R ) .
The uniform separation rates presented in Theorem 3.5 are over the restricted Sobolev balls S s,td ( R, L ) .
Whether those rates can also be derived under less restrictive assumptions and in the more general setting of (unrestricted)
Sobolev balls S sd ( R ) is a challenging problem, which is left for future work.
We consider three different experiments based on a Gamma one-dimensional distribution, a GaussianBernoulli Restricted Boltzmann Machine, and a Normalizing Flow for the MNIST dataset.
We compare our proposed aggregated test KSDA GG against three alternatives: the KSD test which uses the median bandwidth, a test which splits the data to select an ‘optimal’ bandwidth according to a proxy for asymptotic test power, and a test which uses extra data for bandwidth selection.
The ‘extra data’ test is designed simply to provide a best case oracle for the bandwidth selection procedure which maximises asymptotic test power, it cannot be used in practice (i.e. any extra samples from q would normally be incorporated into the sample being tested).
In order to ensure that our tests always have correct levels for all bandwidth values, dimensions and sample sizes, we use the parametric bootstrap in our experiments.
4.1 Alternative bandwidth selection approaches
Gretton et al.
(2012a) proposed to use the median heuristic as kernel bandwidth, it consists in the median of the L 2 -distances between the samples given by
[Equation]
Gretton et al.
(2012b)
ﬁrst proposed, for the two-sample problem using a linear-time MMD estimator, to split the data and to use half of it to select an ‘optimal’ bandwidth which maximises a proxy for asymptotic power.
This procedure was extended to quadratic-time estimators and to the goodnessof-ﬁt framework by Jitkrittum et al.
(2017), Sutherland et al.
(2017) and Liu et al.
These strategies rely on the asymptotic normality of the test statistic under H a .
In our setting, the asymptotic power proxy to maximise is the ratio (cid:91) KSD 2 p,k ( X N ) / (cid:98) σ H a where (cid:98) σ 2 H a is a closed-form regularised positive estimator of the asymptotic variance of (cid:91) KSD 2 p,k under H a (Liu et al., 2020, Equation 5).
In our experiments, we also consider a test which has access to N extra samples drawn from q to select an ‘optimal’ bandwidth to run the KSD test on the original N samples X N .
This test is interesting to compare to because it uses an ‘optimal’ bandwidth without being detrimental to power.
4.2 Experimental details
In our experiments, we use collections of bandwidths of the form Λ( (cid:96) − , (cid:96) + ) :
= (cid:8)
i λ med : i = (cid:96) − , . . .
, (cid:96) + (cid:9) for the median bandwidth λ med and integers (cid:96) − < (cid:96) + with uniform weights w λ :
= 1 / ( (cid:96) + − (cid:96) − + 1) .
For the tests which split the data, we select the bandwidth out of the collection Λ( (cid:96) − , (cid:96) + ) which maximises the power proxy discussed in Section 4.1.
All our experiments are run with level α = 0 .
05 using the IMQ kernel deﬁned in Equation (6) with parameter β k = 0 .
We use a parametric bootstrap with B 1 = B 2 = 500 simulated KSD values to compute the adjusted test thresholds, and B 3 = 50 steps of bisection method to estimate the correction u α in Equation (5).
To estimate the probability of rejecting the null hypothesis, we average the test outputs acrossrepetitions.
All experiments have been run on an AMD Ryzen Threadripper 3960X 24 Cores 128Gb
RAM CPU at 3.8GHz, the runtime is of the order of a couple of hours (signiﬁcant speedup can be obtained by using parallel computing).
We have used the implementation of Jitkrittum et al. (2017) to sample from a Gaussian-Bernoulli Restricted Boltzmann Machine, and Phillip Lippe’s implementation of MNIST Normalizing Flows, both under the MIT license.
4.3 Gamma distribution
For our ﬁrst experiment, we consider a one-dimensional Gamma distribution with shape parameterand scale parameter 5 as the model p .
We draw 500 samples from a Gamma distribution with the same scale parameter 5 and with a shifted shape parameter 5 + s for s ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } .
We consider the collection of bandwidths Λ(0 , 10) .
The results we obtained are presented in Figure 1a.
We observe that all tests have the prescribed level 0.05 under the null hypothesis, which corresponds to the case s = 0 .
As the shift parameter s increases, the two densities p and q become more different and rejection of the null becomes an easier task, thus the test power increases.
Our aggregated test KSDA GG achieves the same power as the ‘best case’ bound on the performance of the asymptotic power heuristic, yielded by the splitting [Figure] test with extra data.
The median test obtains only slightly lower power, this closeness in power can be explained by the fact that this one-dimensional problem is a simple one.
We note that the normal splitting test has signiﬁcantly lower power: this is because, even though it uses an ‘optimal’ bandwidth, it is then run on only half the data, which results in a loss of power.
4.4 Gaussian-Bernoulli Restricted Boltzmann Machine
As ﬁrst considered by Liu et al.
(2016) for goodness-of-ﬁt testing using the KSD, we consider a
Gaussian-Bernoulli Restricted Boltzmann Machine.
It is a graphical model with a binary hidden variable h ∈ {− 1 , 1 } d h and a continuous observable variable x ∈ R d .
Those variables have joint density
[Equation]
where Z is an unknown normalizing constant.
By marginalising over h , we obtain the density p of x
[Equation]
We can sample from it using a Gibbs sampler with 2000 burn-in iterations.
We use the dimensions d = 50 and d h = 40 as considered by Jitkrittum et al.
(2017) and Grathwohl et al.
Even though computing p is intractable for large dimension d h , the score function admits a convenient closed form
[Equation]
We draw the components of b and c from Gaussian standard distributions and sample Rademacher variables taking values in {− 1 , 1 } for the elements of B for the model p .
We draw 1000 samples from a distribution q which is constructed in a similar way as p but with the difference that
some
Gaussian noise N (0 , σ ) is injected into the elements of B .
We consider the standard deviations of the perturbations σ ∈ { 0 , 0 . 01 , 0 .
We run our experiments with the collection of bandwidths Λ( − 20 , 0) and provide the results in Figure 1b.
Again, we observe that our aggregated test KSDA GG matches the power obtained by the test which uses extra data to select an ‘optimal’ bandwidth.
This means that, in this experiment, KSDA GG obtains the same power as the ‘best’ single test.
The difference between KSDA GG and the median heuristic test is signiﬁcant on this experiment, and the splitting test obtains lowest power of the four tests.
Again, all tests have well-calibrated levels ( σ = 0 ) and increasing the noise level σ results in more power for all the tests.
[Figure]
4.5 MNIST Normalizing Flow
Finally, we consider a high-dimensional problem working with images in dimensions 28 2 = 784 .
We consider a multi-scale Normalizing Flow (Dinh et al., 2017; Kingma and Dhariwal, 2018) which has been trained on the MNIST dataset (LeCun et al., 1998, 2010), it is a generative model which has a probability density p .
As observed in Figure 2, some samples produced by the model can look exactly like MNIST digits, while other do not resemble digits.
This Normalizing Flow has been trained to ‘ideally’ produce samples from the MNIST dataset.
We are interested in whether or not we can detect the difference in densities.
Given some images of digits, are we able to tell if those were sampled from the Normalizing Flow model?
We consider the case where the images from q are sampled from the model (level experiment, conﬁrming performance under H 0 ), and the case where the samples from q are drawn from the true MNIST dataset (power experiment).
The experiments are run with the collection of bandwidths Λ( − 20 , 0) .
The results are displayed in Figure 3a and Figure 3b.
In Figure 3a, we observe that the four tests have correct levels (around 0.05) for the ﬁve different sample sizes considered (the small ﬂuctuations about the designed test level can be explained by the fact that we are averaging 200 test outputs to estimate these levels).
The well-calibrated levels obtained in Figure 3a demonstrate the validity of the power results presented in Figure 3b.
In Figure 3b, we observe that only our aggregated test KSDA GG obtains high power, that is, is able to detect that MNIST samples are not drawn from the Normalizing Flow.
The power of the other tests increases only marginally as the sample size increases.
We notice that the test which uses extra data to select an ‘optimal’ bandwidth performs poorly when compared to KSDA GG .
This could be explained by the fact that this test selects the bandwidth using a proxy for the asymptotic power, and that in this high-dimensional setting, the asymptotic regime is not attained with sample sizes below 500.
[Figure]
A Asymptotic level for wild bootstrap
The proof of asymptotic level using a wild bootstrap was proved by Chwialkowski et al.
Proposition 3.2) relying on the result of Leucht (2012, Theorem 2.1) and of and Chwialkowski et al.
(2014, Lemma 5). Their results for V -statistics can be extended to the use of U -statistics.
They proved this result for the more general non-i.i.d. case while we need only the result for the simpler i.i.d. case. They proved that the difference between true quantiles and the wild bootstrap quantiles converges to zero in probability under the null hypothesis with the following dependence on N :
[Equation]
converges to 0 in probability, where K N is the KSD estimator of Equation (1), and B N is the wild bootstrap KSD of Equation (4).
Lipschitz continuity of h p,k is required for the result to hold.
The time complexity of KSDAgg is O (cid:0) |K| ( B 1 + B 2 ) N 2 (cid:1) as provided in Algorithm 1.
It grows linearly with the number of kernels |K| , quadratically with the sample size N , and linearly with the number of bootstrap samples B 1 + B 2 .
[Table]
We report in Table 1 the runtimes of KSDAgg, KSD median, KSD split, and KSD split extra data, all using a wild bootstrap to estimate the quantiles in the MNIST Normalizing Flow experimental setting considered in Section 4.5.
Since we consider a collection of bandwidths Λ , the time complexity of KSDAgg is O ( | Λ | ( B 1 +
B 2 ) N 2 ) and the one for median KSD is O ( B 1 N 2 ) where | Λ | = 21 and B 1 = B 2 = 500 .
While
KSDAgg takes roughly 10 times longer to run than KSD median, we could have expected a larger difference looking at the time complexities.
This can be explained by the fact that there are two major time-consuming steps: (i) computing the kernel matrices, and (ii) computing the wild bootstrap samples.
While (i) has complexity O ( N 2 ) and (ii) complexity O ( BN 2 + NB 2 ) , the constant for step (i) is much larger than the one for step (ii) since (i) requires computing the matrix of pairwise distances while (ii) only consists in computing some matrix multiplications.
Note that to compute the | Λ | kernel matrices for KSDAgg, we need to compute the matrix of pairwise distances only once.
When splitting the data, the computationally expensive step is to select the bandwidth.
All the | Λ | kernel matrices need to be computed, as for KSDAgg, which is the expensive step (i).
The KSD split test runs only slightly faster than the KSD split extra test; it runs faster than KSDAgg but their runtimes are of the same order of magnitude.
More powerful than Bonferroni correction.
The Bonferroni correction for multiple testing corresponds to using the adjusted level α/ |K| for each of the |K| tests. With the correction used for
KSDAgg with uniform weights, the adjusted level for each test is u α / |K| with u α deﬁned in Equation (5). It can be shown that u α ≥ α (Albert et al., 2022, Lemma 4), which means that KSDAgg will always reject the null when the test with Bonferroni correction would reject it. When using uniform weights, the multiple testing correction we use is guaranteed to always lead to a test as least as powerful as the one using the Bonferroni correction.
The Bonferroni correction uses a loose union bound argument in order to guarantee control of the probability of type I error, the method we use aims to tighten this loose upper bound. We present edge-case examples to illustrate the strengths of our multiple testing correction. First, assume that the |K| events are all disjoints, then the union bound is tight and both methods yield the adjusted levels α/ |K| . Second, assume that all events are the same ( e.g. same kernels), then the Bonferroni correction still yields adjusted levels α/ |K| , while our multiple testing strategy can detect that there is nothing to correct for and provide ‘adjusted’ levels α .
We note that, given some ﬁxed weights and level, there does not exist a single u α associated to them, it will depend on the events. Choice of weights. Without any prior knowledge (which is often the case in practice), we recommend using uniform weights since we do not expect particular bandwidths to be better-suited than others. If the user has some prior knowledge of which bandwidths would be better for the task considered, then higher weights on those bandwidths can be used.
Allowing for weights whose sum is strictly smaller than 1 is only for convenience of being able to add a new bandwidth with a new weight without changing the previous weights (for examples with weights 6 /π 2 (cid:96) 2 for (cid:96) ∈ N \ { 0 } ). Multiplying all the weights by a constant simply results in dividing the correction u α deﬁned in Equation (5) by the same constant. This means that the product u α w λ remains the same, and hence the deﬁnition of the aggregated test is not affected by this sample. For simplicity, in practice, we use weights whose sum is equal to 1.
We will add a discussion of the choice of weights in the ﬁnal version. Interpretability. When KSDAgg rejects the null, we can check which speciﬁc kernels rejected the adjusted tests: this provides the kernels/bandwidths which are well-adapted to the problem, i.e. the "best" selection of bandwidths is naturally returned as a side-effect of the test (without requiring data splitting). This also contributes to interpretability of the resulting test, for instance if different kernels prioritise different features.
Extension to an uncountable collection of kernels. Extension to the case of a continuous collection of kernels (indexed by the bandwidth parameter on the positive real line) is a direction for future work, however, this extension is far from trivial. To the best of our knowledge, this has currently never been done without data splitting. Our tests retain high power even with large collections of kernels, and our method also allows to aggregate multiple kernels (Gaussian, Laplace, IMQ, Mattérn, etc.) with different parameters, but the extension to a continuous parametrization remains a challenge.
Median bandwidth test.
In settings (possibly adversarially constructed) where the median bandwidth is the ‘best’ bandwidth, the KSD test with median bandwidth could in theory be more competitive than KSDAgg since by considering a large collection of bandwidths we are not only considering the ‘best’ median bandwidth but also ‘worse’ bandwidths. However, in practice, we cannot know in advance which bandwidth would perform well, and KSDAgg retains power even for large collections of bandwidths (21 bandwidths considered in MNIST Normalizing Flow experiment in Section 4.5).
Data splitting test. Another setting (also possibly adversarially constructed) could be one in which the ‘best’ bandwidth lies in between two bandwidths of our collection and those two are ‘worse’ bandwidths. In that case, a test which uses data splitting to select an ‘optimal’ bandwidth would be able to select it, however one must bear in mind the loss of power due to data splitting. In such a setting, the power of KSDAgg could then be improved by considering a ﬁner collection of bandwidths.
However, in our experiments, the aggregation approach outperformed competing approaches.
E Proofs E.1 Proof of Theorem [-3.
-]3.
Note that
[Equation]
where
[Equation]
since (cid:91) KSD 2 p,k is an unbiased estimator.
By Chebyshev’s inequality (Chebyshev, 1899), we know that
[Equation]
We deduce that P q (cid:0) ∆ kα [Equation] X N ) = [Equation]
≤ β if
[Equation]
−
[Equation] The condition in Equation (8), which controls the probability of type I error of ∆ kα [Equation] needs to hold with high probability since (cid:98) q k 1 − α depends on the randomness of the samples X N drawn from q and on the randomness of the Rademacher random variables in Equation (4).
We want to derive a condition in terms of (cid:107) p − q (cid:107) 22 rather than in terms of KSD 2 p,k ( q ) as in Equation (8).
For this, using Stein’s identity, we obtain
[Equation]
which gives
[Equation]
[Equation] To guarantee P q (cid:0) ∆ kα ( X N ) = 0 (cid:1) ≤ β , an equivalent condition to the one presented in Equation (8) is then
[Equation]
which needs to hold with high probability over the randomness in (cid:98) q k 1 − α .
We now upper bound the quantile and variance terms in Equation (9) to obtain a new condition ensuring control of the type II error.
To bound the quantile term (cid:98) q k 1 − α using B 1 wild bootstrapped statistics, using the Dvoretzky–Kiefer–
Wolfowitz inequality (Dvoretzky et al., 1956; Massart, 1990) as done by Schrab et al.
(2021, Appendix
E.4), it is sufﬁcient to bound the quantile (cid:98) q k, ∞ 1 − α using wild bootstrapped statistics without ﬁnite approximation provided that B 1 ≥ 3 α 2 (cid:0) log (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) .
Using the concentration bound for i.i.d.
Rademacher chaos of de la Peña and Giné (1999, Corollary 3.2.6) as presented by Kim et al.
Equation 39)
, there exists some constant C > 0 such that
[Equation]
[Equation]
 which gives
[Equation]
for some different constant C > 0 .
Using Markov’s inequality, for any δ ∈ (0 , 1) , with probability at least 1 − δ we have
[Equation]
which gives
[Equation]
since α ∈ (0 , e − 1 ) , where in the last inequality C > 0 is a different constant depending on δ .
Using
Dvoretzky–Kiefer–Wolfowitz inequality (Dvoretzky et al., 1956) as explained above, we deduce that
[Equation]
with arbitrarily high probability, for some constant C > 0 .
We now bound the variance term in Equation (8).
Using the result of Albert et al.
(2022, Equation 6), there exists C > 0 such that
[Equation]
where
[Equation]
and
[Equation]
since
[Equation]
by Stein’s identity.
We deduce that
[Equation]
for some different constant C > 0 depending on M .
Using the classical inequalities √ x + y ≤ √ x + √ y and 2 √ xy ≤ x + y for x, y > 0 , which are also considered in the works of Fromont et al.
(2013), Albert et al.
(2022) and Schrab et al.
(2021), we obtain
[Equation]
since α ∈ (0 , e − 1 ) , β ∈ (0 , 1) and C k ≥ 1 by assumption, 2 where the constant C > 0 is different on the last line.
By considering two times the largest of the two constants in the quantile and variance bounds of Equations (10) and (11) [Equation] we obtain
[Equation]
By combining this bound with the condition in Equation (9) [Equation] we get that P q (cid:0) ∆ k α ( X N ) = 0 (cid:1) ≤ β if
[Equation]
which concludes the proof
[Equation]
E.2 Proof of Proposition 3.
Recall that the correction term in Equation (5) is deﬁned as
[Equation]
where (cid:98) q
[Equation]
as deﬁned in Section 3.1.
Hence, we have
[Equation]
Recall that our estimator is (cid:91) KSD 2 p,k ( X N ) where X N :=
( X i )
Ni =1 are drawn from q .
Recall also from
Equation (3), that for the parametric bootstrap the element (cid:101) K bk is computed by drawing new samples ( X (cid:48)
i ) N (cid:48)
i =1 from the model p and computing (cid:91)
KSD 2 p,k (cid:16) ( X (cid:48) i ) N (cid:48) i =1 (cid:17) .
Hence, at every sample size, under the null hypothesis H 0 : p = q the quantities (cid:101) K
bk and (cid:91) KSD 2 p,k ( X N ) are identically distributed.
For the wild bootstrap, as deﬁned in Equation (4), Chwialkowski et al.
(2014, 2016) show that, under H 0 , (cid:101) K bk and (cid:91)
KSD 2 p,k ( X N ) have the same asymptotic distribution.
Therefore, by taking the expectation in Equation (12), we obtain [Equation]
k for some k ∈ K
[Equation]
which holds non-asymptotically using the parametric bootstrap, and asymptotically using the wild bootstrap
Note that this is different from the two-sample case where using a wild bootstrap yields well-calibrated non-asymptotic levels (Schrab et al., 2021, Proposition 8).
The same reasoning holds by replacing u α by any value u ∈ (0 , u α ) .
In particular, it holds for the lower bound (cid:98)
u α of the interval obtained by performing the bisection method to approximate the supremum in the deﬁnition of u α .
This lower bound (cid:98) u
α is the one we use in practice, as shown in Algorithm 1 with the step ‘ u α = u min ’.
We have proved that the test with correction term u α approximated with a bisection method also achieves the desired level α .
E.3 Proof of Theorem 3.
As explained in Appendix E.2 and utilised in Algorithm 1, we use the lower bound (cid:98) u
α of the interval obtained by performing the bisection method to approximate the supremum in the deﬁnition of u α in
Equation (5).
The assumptions B 2 ≥ 8 α 2 log (cid:0) 2 β (cid:1) and B 3 ≥ log 2 (cid:0) 4 α min k ∈K w − 1 k (cid:1) ensure that (cid:98) u
α ≥ α/as shown by Schrab et al.
(2021, Appendix E.9).
The probability of type II error of ∆ K α is
[Equation]
for some k ∈ K (cid:17) . In order to guarantee P q (cid:0) ∆ K α ( X N ) = 0 (cid:1) ≤ β it is sufﬁcient to guarantee P q (cid:16) ∆ k ∗ αw k ∗ / 2 ( X N ) = 0 (cid:17) ≤ β for some k ∗ ∈ K to be speciﬁed shortly in Equation (14).
By assumption, 3 we have
[Equation]
In order to apply Theorem 3.1 to ∆ k α k with α k :=
αw k / 2 for k ∈ K , we need to ensure that the condition on B 1 of Theorem 3.1 is satisﬁed for all k ∈ K , that is
[Equation]
for k ∈ K .
Since 0 < α k < α < e − 1 , we have α (1 − α ) ≥ α k (1 − α k ) for k ∈ K .
We also have
[Equation]
for k ∈ K .
We deduce that
[Equation]
for k ∈ K , and so, applying Theorem 3.1 to ∆ kα k for k ∈ K is justiﬁed.
We obtain that
P q (cid:16) ∆ kαw k / 2 ( X N )
= 0 (cid:17) ≤ β if
[Equation]
or, with a different constant C > 0 , if
[Equation]
since log (cid:16) 2 αw k (cid:17) ≤ (log(2) + 1) log (cid:16) 1 αw k (cid:17) as α ∈ (0 , e − 1 ) and w k ∈ (0 , 1) .
Now, let k ∗ := argmin k ∈K (cid:18)(cid:13)(cid:13)
[Equation]
Finally, we have P q (cid:0) ∆ K α
[Equation]
X N
[Equation]
=
[Equation]
≤ β if P q (cid:16) ∆ k ∗ αw k ∗
[Equation]
2 ( X N ) = 0 (cid:17) ≤ β , that is, if
[Equation]
as desired.
E.4 Proof of Theorem 3.
Recall that we suppose that the following assumptions hold.
• The model density p is strictly positive on its connected and compact support S ⊆ R d .
The score function ∇ log p ( x ) is continuous and bounded on S .
• The support of the density q is a connected and compact subset of S .
The kernel used is a scaled Gaussian kernel k λ ( x, y ) :
= λ 2 − d exp (cid:16) −(cid:107) x − y (cid:107) 22 / λ 2 (cid:17) .
We introduce some notation.
For some c ∈ R , we write c :=
( c, . .
. , c ) ∈ R d .
We write ( a 1 , . . .
, a d ) ≤ ( b 1 , . . .
, b d ) when a i ≤ b i for i = 1 , . . .
, d .
We use C, C (cid:48) to denote some generic constants which may change on different lines.
Note that, by properties of compactness on R d , there exists some a > 0 such that S ⊆ (cid:2) − a 2 , a 2 (cid:3) d .
Since the score function ∇ log p ( x ) is continuous and bounded on S which is compact, there exists some c 1 > 0 such that
[Equation]
We work with a scaled Gaussian kernel with bandwidth λ ≤
[Equation]
which satisﬁes
[Equation]
[Table] [Table]
[Equation]
where the constant C 0 :
= 2 π d/ 2 ( dc 21 + 2 dc 1 a + d ) .
The Stein kernel can be upper bounded by a scaled Gaussian kernel on S .
Writing
[Equation]
which is of the form considered by Schrab et al.
(2021, Section 3.1) with equal bandwidths for each dimension
[Equation]
we have shown that
[Equation]
Writing ψ := p − q , recall from Theorem 3.1 that a sufﬁcient condition for control of the probability of type II error by β is
[Equation]
Using the upper bound
[Equation]
where the last inequality holds as in Schrab et al.
[Equation] Equation (22)) [Equation] we obtain the sufﬁcient condition
[Equation]
(16) where we have absorbed the term C 0 √ M in the constant C .
We would like to upper bound the term (cid:13)(cid:13)(cid:13) ψ − T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 by (cid:13)(cid:13)
ψ − T ¯ k λ ψ (cid:13)(cid:13) 2
but this is not possible using Equation (15).
Instead, we can use the triangle inequality to get
[Equation]
[-,kλ ψ (cid:13)(cid:13)(cid:13) 2 2 ≤ (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
-]
[-ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
 + (cid:13)(cid:13)(cid:13) T ¯ k λ ψ + T h p,kλ ψ (cid:13)(cid:13)(cid:13) 2≤ (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 
 + ( C 0 + 1) 2 (cid:13)(cid:13)-]since
[Equation]
for all y ∈ S . Recall that we assume that ψ ∈ S s,td ( R, L ) for some L to be determined, in particular ψ ∈ S sd ( R ) .
For the term (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 , we use the fact that ψ ∈ S sd ( R ) .
The term (cid:13)(cid:13) ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 can then be upper bounded exactly as done by Schrab et al.
(2021, Appendix E.6) with the difference that we choose (cid:101) t > 0 such that S 1 < 0 .
5 rather than S 1 < 1 ( S 1 is deﬁned in Schrab et al.
Appendix E.6)).
Following their reasoning, assuming that ψ ∈ S sd ( R ) , we obtain that there exists some S 1 ∈ (0 , 0 . 5) and some constant C > 0 (depending on d, s, R ) such that
[Equation]
To upper bound the term (cid:13)(cid:13)
T ¯ k λ ψ (cid:13)(cid:13) 2 2 , we modify the reasoning of Schrab et al.
(2021, Appendix E.6) used for the term (cid:13)(cid:13)
ψ − T ¯ k λ ψ (cid:13)(cid:13) 2 2 .
First, note that since ¯ k λ is translation-invariant, T ¯ k λ is a convolution as
[Equation]
where ϕ λ ( u ) : = d
[Equation]
i = λ −
[Equation]
(cid:16) u i λ (cid:17) , and
K ( u ) :
= π − 1 / 2 exp( − u 2 ) .
By properties of Fourier transforms, we have (cid:99) ϕ λ ( ξ )
=
(cid:81) di =1 (cid:98) K ( λξ i ) for ξ ∈ R d .
Note that (cid:12) (cid:12)(cid:12) (cid:12)(cid:12) d (cid:89)
i = (cid:98) K ( ξ i ) (cid:12) (cid:12)(cid:12) (cid:12)(cid:12) ≤
d (cid:89) i = (cid:90)
R (cid:12)(cid:12) K
( x ) e − ixξ i (cid:12) (cid:12) d x = d (cid:89) i = (cid:90)
R | K ( x ) | d x = 1 .
We now let L := 8( C 20 + 1) .
By assumption, we have ψ ∈ S s,td ( R, L ) , so (cid:90) (cid:107) ξ (cid:107) 2 ≤ t (cid:12)(cid:12)(cid:12) (cid:98) ψ ( ξ ) (cid:12)(cid:12)(cid:12) 2 d ξ ≤ 1 L (cid:90) R d
cid:12)(cid:12)(cid:12) (cid:98) ψ ( ξ ) (cid:12)(cid:12)(cid:12) 2 d ξ =
{+(cid:13)(cid:13) (cid:98) ψ (cid:13)(cid:13) 
+}
L .
For s > 0 , deﬁne T s := sup (cid:107) ξ (cid:107) 2 >t (cid:12)(cid:12)(cid:12)(cid:81) di =1 (cid:98) K ( ξ i ) (cid:12)(cid:12)(cid:12) (cid:107) ξ (cid:107) s≤t s < ∞ .
Then, using Plancherel’s Theorem as in Schrab et al. (2021, Appendix E.6), we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Table]
[Equation]
Equation (16) then becomes
[Equation]
which gives
[Equation]
Having proved the power guaranteeing condition in Equation (17), we can then derive rates for the
KSD and KSDAgg tests as similarly done by Schrab et al.
Set
λ :
=
N − 2 / (4 s + d ) to get the condition
[Equation]
which is the minimax rate over (unrestricted) Sobolev balls S sd ( R ) (Li and Yuan, 2019; Balasubramanian et al., [-
-]2021;
Albert et al., 2022; Schrab et al., 2021).
In practice, the smoothness parameter of the Sobolev ball is not known, so we cannot set λ :=
N − 2 / (4 s + d ) (i.e. it cannot be implemented).
Instead, we can use our aggregated test KSDAgg.
Adapting the proof of Theorem 3.3 with the single condition Equation (17), we obtain the aggregated condition
[Equation]
Similarly to Schrab et al.
(2021, Corollary 10), consider
[Equation]
Let
[Equation]
The bandwidth λ ∗ := 2 − (cid:96) ∗ ∈ Λ satisﬁes ln (cid:18)w
[Equation]
∗ (cid:19) ≤ C ln( (cid:96) ∗ ) ≤ C ln(ln( N )) as w λ ∗ := 6 π − 2 ( (cid:96) ∗ ) − 2 , and  (cid:18)
N ln(ln( N ))
(cid:19) − 2 / (4 s + d ) ≤ λ ∗ ≤ (cid:18)
N ln(ln( N ))
(cid:19) − 2 / (4 s + d ) . We get λ 2 s ∗ ≤ (cid:18)
N ln(ln( N ))
(cid:19) − 4 s/ (4 s + d ) and log (cid:18) αw λ ∗ (cid:19) βNλ d/ 2 ∗ ≤ C (cid:18) N ln(ln( N )) (cid:19) − 1 (cid:18)
N ln(ln( N ))
(cid:19) d/ (4 s
+
[Equation]
The power condition Equation (18) becomes
[Equation]
which concludes the proof
[Equation]