Computationally Efﬁcient Aggregated Kernel Tests using Incomplete U -statistics
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a series of computationally efﬁcient, nonparametric tests for thetwo-sample, independence and goodness-of-ﬁt problems, using the Maximum
Mean Discrepancy (MMD), Hilbert Schmidt Independence Criterion (HSIC), and
Kernel Stein Discrepancy (KSD), respectively.
Our test statistics are incomplete
U -statistics, with a computational cost that interpolates between linear time in thenumber of samples, and quadratic time, as associated with classical U -statistictests.
The three proposed tests aggregate over several kernel bandwidths to detectdepartures from the null on various scales: we call the resulting tests MMDAggInc,
HSICAggInc and KSDAggInc.
For the test thresholds, we derive a quantile boundfor wild bootstrapped incomplete U -statistics, which is of independent interest.
Wederive uniform separation rates for MMDAggInc and HSICAggInc, and quantifyexactly the trade-off between computational efﬁciency and the attainable rates: thisresult is novel for tests based on incomplete U -statistics, to our knowledge.
Wefurther show that in the quadratic-time case, the wild bootstrap incurs no penalty totest power over more widespread permutation-based approaches, since both attainthe same minimax optimal rates (which in turn match the rates that use oraclequantiles).
We support our claims with numerical experiments on the trade-offbetween computational efﬁciency and test power.
Introduction
Nonparametric hypothesis testing is a fundamental ﬁeld of statistics, and is widely used by the machinelearning community and practitioners in numerous other ﬁelds, due to the increasing availability ofhuge amounts of data.
When dealing with large-scale datasets, computational cost can quickly emergeas a major issue which might prevent from using expensive tests in practice; constructing efﬁcienttests is therefore crucial for their real-world applications.
In this paper, we construct kernel-basedaggregated tests using incomplete U -statistics (Blom, 1976) for the two-sample , independence andgoodness-of-ﬁt problems (which we detail in Section 2).
The quadratic-time aggregation procedure isknown to lead to state-of-the-art powerful tests (Fromont et al., 2012, 2013; Albert et al., 2022; Schrabet al., 2021, 2022), and we propose efﬁcient variants of these well-studied tests, with computationalcost interpolating from the classical quadratic-time regime to the linear-time one.
Related work: aggregated tests.
Kernel selection (or kernel bandwidth selection) is a fundamentalproblem in nonparametric hypothesis testing because it has a major inﬂuence on test power.
Motivated by this problem, non-asymptotic aggregated tests, which combine tests with different kernelbandwidths, have been proposed for the two-sample (Fromont et al., 2012, 2013;
Kim et al., 2022;
Schrab et al., 2021), independence (Albert et al., 2022;
Kim et al., 2022), and goodness-of-ﬁt (Schrabet al., 2022) testing frameworks.
Li and Yuan (2019) and Balasubramanian et al.
(2021) constructsimilar aggregated tests for these three problems, with the difference that they work in the asymptotic regime.
All the mentioned works study aggregated tests in terms of uniform separation rates (Baraud,2002).
Those rates depend on the sample size and satisfy the following property: if the L 2 -normdifference between the densities is greater than the uniform separation rate, then the test is guaranteedto have high power.
All aggregated kernel-based tests in the existing literature have been studiedusing estimators which are U -statistics (Hoeffding, 1992) with tests running in quadratic time.
Related work: linear-time kernel tests.
Several linear-time kernel tests have been proposed forthose three testing frameworks.
Those include tests using classical linear-time estimators with medianbandwidth (Gretton et al., 2012a; Liu et al., 2016) or selecting an optimal bandwidth on held-outdata to maximize power (Gretton et al., 2012b), tests using eigenspectrum approximation (Grettonet al., 2009), tests using post-selection inference for adaptive kernel selection, also using incomplete
U -statistics (Yamada et al., 2018, 2019; Lim et al., 2019, 2020; Kübler et al., 2020;
Freidling et al.,2021), tests which use a Nyström approximation of the asymptotic null distribution (Zhang et al.
,2018; Cherfaoui et al., 2022), random Fourier features tests (Zhang et al., 2018;
Zhao and Meng, 2015;
Chwialkowski et al., 2015), the current state-of-the-art adaptive tests which use features selectedon held-out data to maximize power (Jitkrittum et al., 2016, 2017a,b), as well as tests using neuralnetworks to learn a discrepancy (Grathwohl et al., 2020).
We also point out the very relevant worksof Kübler et al.
and Huggins and Mackey (2018) on quadratic-time tests, and of Ho and Shieh(2006), Zaremba et al.
and Zhang et al.
(2018) on the use of block U -statistics which havecomplexity O ( N 1 . 5 ) for block size √ N where N is the sample size.
Contributions and outline.
In Section 2, we present the three testing problems with their associatedwell-known quadratic-time kernel-based estimators (MMD, HSIC, KSD) which are U -statistics.
Weintroduce three associated incomplete U -statistics estimators, which can be computed in linear time,in Section 3.
We then provide quantile and variance bounds for generic incomplete U -statistics usinga wild bootstrap, in Section 4.
We study the level and power guarantees of linear-time tests usingincomplete U -statistics for a ﬁxed kernel bandwidth, in Section 5.
In particular, we obtain uniformseparation rates for the two-sample and independence tests over a Sobolev ball, and show that theserates are minimax optimal up to the cost incurred for efﬁciency of the test.
In Section 6, we proposeour efﬁcient aggregated tests which combine tests with multiple kernel bandwidths.
We prove that theproposed tests are adaptive over Sobolev balls and achieve the same uniform separation rate (up to aniterated logarithmic term) as the tests with optimal bandwidths.
As a result of our analysis, we haveshown minimax optimality over Sobolev balls of the quadratic-time tests using quantiles estimatedwith a wild bootstrap.
Whether this optimality result also holds for tests using the more generalpermutation-based procedure to approximate HSIC quantiles, was an open problem formulated by
Kim et al.
(2022), we prove that it indeed holds in Section 7.
We close the paper with numericalexperiments in Section 8, where we observe that MMDAggInc, HSICAggInc and KSDAggInc retainhigh power and outperform other state-of-the-art linear-time kernel tests.
Our implementation ofthe tests and code for reproducibility of the experiments are available online under the MIT License:https://anonymous.4open.science/r/agginc-10EF/README.md .
Background
Here we brieﬂy describe our main problems of interest, comprising the two-sample, independenceand goodness-of-ﬁt problems.
We approach these problems from a nonparametric point of viewusing the kernel-based statistics: MMD, HSIC, and KSD.
We brieﬂy introduce original forms ofthese statistics, which can be computed in quadratic time, and also discuss ways of calibrating testsproposed in the literature.
Two-sample testing.
In this problem, we are given independent samples X m :=
( X i ) 1 ≤ i ≤ m and 81 Y n =
( Y j ) 1 ≤ j ≤ n , consisting of i.i.d.
random variables with respective probability density functions 1p and q on R d .
We assume we work with balanced sample sizes so that there exists a constant 2 C > 0such that max( m, n ) ≤ C min( m, n ) .
We are interested in testing the null hypothesis H 0 : p = qagainst
the alternative H 1 : p (cid:54) =
q ; that is, we want to know if the samples come from the samedistribution.
Gretton et al.
(2012a) propose a non-parametric kernel test based on the Maximum Mean
Discrepancy (MMD), a measure between probability distributions which uses a characteristic kernelk (Fukumizu et al., 2008; Sriperumbudur et al., 2011).
It can be estimated using a quadratic-time estimator (Gretton et al., 2012a, Lemma 6) which, as noted by Kim et al.
(2022), can be expressed asa two-sample U -statistic
(both of second order) (Hoeffding, 1992),
[Equation]
where i ba denotes the set of all a -tuples drawn without replacement from { 1 , .
, b } so that (cid:12)(cid:12) i ba (cid:12)(cid:12) =
b · · · ( b − a + 1) , and where, for x 1 , x 2 , y 1 , y 2 ∈ R d , we leth MMD k (
[Equation]
1 , x
[Equation]
; y 1 , y
[Equation]
=
k
[Equation] [Equation]
Independence testing.
In this problem, we have access to i.i.d.
pairs of samples Z N := 93 (cid:0)
Z i (cid:1) 1 ≤ i ≤ N = (cid:0) ( X i , Y i ) (cid:1) 1 ≤ i ≤ N with joint probability density p xy on R d x × R d y and marginals p x on
R d x and p y on R d y .
We are interested in testing H 0 : p xy =
p x ⊗ p y against H 1 : p xy (cid:54) =
p x ⊗ p y ; thatis, we want to know if two components of the pairs of samples are independent or dependent.
Grettonet al.
(2005, 2008) propose a non-parametric kernel test based on the Hilbert Schmidt Independence
Criterion (HSIC).
It can be estimated using the quadratic-time estimator proposed by Song et al.(2012, Equation 5) which is a fourth-order one-sample U -statistic
[Equation]
for characteristic kernels k on R d x and (cid:96) on R d y (Gretton, 2015), and where for z a = ( x a , y a ) ∈
R d x × R d y , a = 1 , . .
, 4 , we let
[Equation]
Goodness-of-ﬁt testing.
For this problem, we are given a model density p on R d and i.i.d.
samples 102 Z
N := ( Z i ) 1 ≤ i ≤ N drawn from a density q on R d .
The aim is again to test H 0 : p = q against
H 1 : p (cid:54) =
q ; that is, we want to know if the samples have been drawn from the model.
Chwialkowskiet al.
(2016) and Liu et al.
both construct a non-parametric goodness-of-ﬁt test using the Kernel
Stein Discrepancy (KSD).
A quadratic-time KSD estimator can be computed as the second-orderone-sample U -statistic,
[Equation]
where the Stein kernel h p,k : R d × R d → R is deﬁned as
[Equation]
In order to guarantee consistency of the Stein goodness-of-ﬁt test (Chwialkowski et al., 2016,
Theorem 2.2)
, we assume that the kernel k is C 0 -universal
(Carmeli et al., 2010, Deﬁnition 4.1)
andthat E q (cid:104)(cid:13)(cid:13) ∇ log p ( Z ) q ( Z ) (cid:13)(cid:13) 2 2 (cid:105) <
∞ .
Quantile estimation.
Multiple strategies have been proposed to estimate the quantiles of test statisticsunder the null for these three tests.
We primarily focus on the wild bootstrap approach (Chwialkowskiet al., 2014), though our results also hold using a parametric bootstrap for the goodness-of-ﬁt setting(Schrab et al., 2022).
In Section 7, we show that the same uniform separation rates can be derived for
HSIC quadratic-time tests using permutations instead of a wild bootstrap.
More details on MMD, HSIC, KSD, and on quantile estimation are provided in Appendix A.
Incomplete U -statistics for MMD, HSIC and KSD
As presented above, the quadratic-time statistics for the two-sample (MMD), independence (HSIC)and goodness-of-ﬁt (KSD) problems can be rewritten as U -statistics with kernels h MMD k , h HSIC k,(cid:96) and h KSD k , respectively.
The computational cost of tests based on these U -statistics grows quadraticallywith the sample size.
When working with very large sample sizes, as it is often the case in real-worlduses of those tests, this quadratic cost can become very problematic, and faster alternative tests arebetter adapted to this ‘big data’ setting.
Multiple linear-time kernel tests have been proposed inthe three testing frameworks (see Section 1 for details).
We construct linear-time variants of theaggregated kernel tests proposed by Fromont et al.
(2013), Albert et al.
(2022), Kim et al.
(2022),and Schrab et al.
(2021, 2022) for the three settings, with the aim of retaining the signiﬁcant poweradvantages of the aggregation procedure observed for quadratic-time tests.
To this end, we proposeto replace the quadratic-time U -statistics presented in Equations (1), (3) and (5) with second orderincomplete U -statistics
(Blom, 1976; Janson, 1984; Lee, 1990),
[Equation]
where for the two-sample problem we let N :=
min( m, n ) , and where the design D b is a subset of i b 2(the set of all 2 -tuples drawn without replacement from { 1 , . . .
, b } ).
Note that D (cid:98) N/ 2 (cid:99) ⊆
i N/
2⊂ i N 2 .
The design can be deterministic.
For example, for the two-sample problem with equal even samplesizes m = n = N , the deterministic design D N = { (2 a − 1 , 2 a ) : a = 1 , . .
, N/ 2 } correspondsto the MMD linear-time estimator proposed by Gretton et al.
(2012a, Lemma 14).
For ﬁxed designsize, the elements of the design can also be chosen at random without replacement, in which case theestimators in Equations (7) to (9) become random quantities given the data.
The results presented inthis paper hold for both deterministic and random (without replacement) design choices.
By ﬁxingthe design sizes in Equations (7) to (9) to be
[Equation]
for some small constant c ∈ N \ { 0 } , we obtain incomplete U -statistics which can be computed inlinear time.
Note that by pairing the samples
Z i :=
( X i , Y i ) , i = 1 , . .
, N for the MMD case and(cid:101)
Z i := (cid:0) Z i , Z i + (cid:98) N/ 2 (cid:99) (cid:1) , i = 1 , . . .
, (cid:98) N/ 2 (cid:99) for the HSIC case, we observe that all three incomplete
U -statistics of second order have the same form, with only the kernel functions and the designdiffering.
The motivation for deﬁning the estimators in Equations (7) to (9) as incomplete U -statisticsof order 2 (rather than of higher order) derives from the reasoning of Kim et al.
(2022, Section 6)
Quantile and variance bounds for incomplete U -statistics
Here we derive upper quantile and variance bounds for a second order incomplete degenerate Ustatistic with a generic degenerate kernel h , for some design D ⊆
i N 2 , deﬁned as
[Equation]
We will use these results to bound the quantiles and variances of our three test statistics for ourhypothesis tests in Section 5.
The derived bounds are of independent interest.
In the following lemma, building on the results of Lee (1990), we directly derive an upper bound onthe variance of the incomplete U -statistic
in terms of the sample size N and of the design size |D| .
Lemma 1.
The variance of the incomplete U -statistic can be upper bounded in terms of the quantities σ 21 := var (cid:0) E (cid:2)
h
( Z, Z (cid:48) )
(cid:12)(cid:12) Z (cid:48)
(cid:3)(cid:1) and σ 22 :=
var( h ( Z, Z (cid:48) )) with different bounds depending on the design choice.
For deterministic design D d , and for random design D r , we have var (cid:0) U (cid:1) ≤ C (cid:18)
N |D d |
[Equation]
1 |D d | σ(cid:19) and var (cid:0)
[Equation]
The proof of Lemma 1 is deferred to Appendix D.
We emphasize the fact that this variance boundalso holds for random design with replacement, as considered by Blom (1976) and Lee (1990).
Forrandom design, we observe that if |D| (cid:16) N 2 then the bound is σ 21 /N + σ 22 /N 2 which is the variancebound of the complete U -statistic
(Albert et al., 2022, Lemma 10).
If N ≤ |D| ≤ N 2 , the variancebound is σ 21 /N + σ 22 / |D| , and if |D| ≤ N it is σ 22 / |D| since σ 21 ≤ σ 22 / 2 (Blom, 1976, Equation 2.1).
Kim et al.
(2022) develop exponential concentration bounds for permuted complete U -statistics, and
Clémençon et al.
(2013) study the uniform approximation of U -statistics by incomplete U -statistics.
To the best of our knowledge, no quantile bounds have yet been obtained for incomplete U -statisticsin the literature.
While permutations are well-suited for complete U -statistics (Kim et al., 2022),using them with incomplete U -statistics results in having to compute new kernel values, and thiscomes at an extra computational cost we would like to avoid.
Restricting the set of permutations tothose for which the kernel values have already been computed for the original incomplete U -statisticcorresponds exactly to using a wild bootstrap (Schrab et al., 2021, Appendix B).
Hence, we considerthe wild bootstrapped second order
incomplete U -statistic
[Equation]
for i.i.d.
Rademacher random variables (cid:15) 1 , . . .
, (cid:15) N with values in {− 1 , 1 } , for which we derive anexponential concentration bound (quantile bound).
We note the in-depth work of Chwialkowski et al.(2014) on the wild bootstrap procedure for kernel tests with applications to quadratic-time MMD and
HSIC tests.
We now provide exponential tail bounds for wild bootstrapped incomplete U -statistics.
Lemma 2.
There exists some constant C > 0 such that, for every t ≥ 0 , we have
[Equation]
where A 2inc := |D| − 2 (cid:80) ( i,j ) ∈D h
( Z i , Z j ) 2 and A 2 := |D| − 2 (cid:80) ( i,
j ) ∈
i N 2 h ( Z i , Z j ) 2 .
Lemma 2 is proved in Appendix E.
While the second bound in Lemma 2 is less tight, it has the beneﬁtof not depending on the choice of design D but only on the design size |D| which is usually ﬁxed.
Efﬁcient kernel tests using incomplete U -statistics
We now formally deﬁne the hypothesis tests obtained using the incomplete U -statistics with a wildbootstrap.
This is done for ﬁxed kernel bandwidths λ ∈ (0 , ∞ )
d x , µ ∈ (0 , ∞ )
d y , for the kernels 3k λ
( x, y ) :
= d x (cid:89) i =
[Equation]
λ
[Equation]
K i (cid:18) x i − y i λ
[Equation]
(cid:19) , (cid:96) µ ( x, y ) :
= d y (cid:89) i =  µ
[Equation]
L i (cid:18) x i − y i µ
[Equation]
(cid:19) , (12) for characteristic kernels ( x, y ) (cid:55)→ K i ( x − y ) , ( x, y )
(cid:55)→ L i ( x − y ) on R × R for functions
K i , L i ∈
L 1 ( R ) ∩ L 2 ( R ) integrating to 1.
We unify the notation for the three testing frameworks.
For the twosample and goodness-of-ﬁt problems, we work only with k λ and have d = d x .
For the independence problem, we work with the two kernels k λ and (cid:96) µ , and for ease of notation we let d := d x + d y and λ d x + i := µ i for i = 1 , . . .
, d y .
We also simply write p :=
p xy and q :=
p x ⊗ p y .
We let U λ and h λ denote either MMD 2 k λ and h MMD k λ , or HSIC k
[Equation]
,(cid:96)
[Equation]
and h HSIC k λ ,(cid:96)
[Equation]
, or KSD 2 p,k λ and h KSD k λ ,p , respectively.
We denote the design size of the incomplete U -statistics in Equations (7) to (9) by
[Equation]
For the three testing frameworks, we estimate the quantiles of the test statistics by simulating thenull hypothesis using a wild bootstrap, as done in the case of complete U -statistics by Fromontet al.
(2012), Schrab et al.
(2021) for the two-sample problem, and by Schrab et al.
(2022) for thegoodness-of-ﬁt problem.
This is done by considering the original test statistic U B 1 +1 λ :=
U λ together with B 1 wild bootstrapped incomplete
U -statistics U 1 λ , . .
, U B 1 λ computed as in Equation (11), andestimating the (1 − α ) -quantile with a Monte Carlo approximation
[Equation]
where U • 1 λ ≤ · · · ≤ U • B 1 +1 λ are the sorted elements U 1 λ , . .
, U B 1 +1 λ .
The test ∆ λα is deﬁned as rejecting the null if the original test statistic U λ is greater than the estimated (1 − α )
-quantile, that is,
[Equation]
We show in Proposition 1 that the test ∆ λα has well-calibrated asymptotic level for goodness-of-ﬁttesting, and well-calibrated non-asymptotic level for two-sample and independence testing.
The proofof the latter non-asymptotic guarantee is based on the exchangeability of U 1 λ , . . .
, U B 1 +1 λ under thenull hypothesis along with the result of Romano and Wolf (2005, Lemma 1).
A similar proof strategycan be found in Fromont et al.
(2012, Proposition 2), Albert et al.
(2022, Proposition 1), and Schrabet al.
(2021, Proposition 1).
The exchangeability of wild bootstrapped incomplete U -statistics forindependence testing does not follow directly from the mentioned works.
We show this through anintriguing connection between the MMD kernel and the HSIC kernel (proof deferred to Appendix C).
Proposition 1.
The test ∆ λα has level α ∈ (0 , 1) , i.e. , P H 0 (cid:0)
∆ λα ( Z N ) = 1 (cid:1) ≤ α .
This holds nonasymptotically for the two-sample and independence cases, and asymptotically for goodness-of-ﬁt.
Having established the validity of the test ∆ λα , we now study power guarantees for it in terms ofthe L 2 -norm of the difference in densities (cid:107) p − q (cid:107) 2 .
In Theorem 1, we show for the three tests that,if (cid:107) p − q (cid:107) 2 exceeds some threshold, we can guarantee high test power.
For the two-sample andindependence problems, we derive a uniform separation rate (Baraud, 2002) over Sobolev balls
[Equation]
(14) with radius R > 0 and smoothness parameter s > 0 .
This uniform separation rate is the smallestvalue of t such that for any alternative with (cid:107) p − q (cid:107) 2 > t and p − q ∈ S sd ( R ) the probability of type
II error of ∆ λα can be controlled by β ∈ (0 , 1) .
Before presenting Theorem 1, we need to introducemore notation uniﬁed over the three testing frameworks; we deﬁne the integral transform T λ as
[Equation]
(15) for f ∈ L 2 ( R d ) , x ∈ R d , where K λ := k λ for the two-sample problem, K λ :=
k λ ⊗ (cid:96) µ for theindependence problem, and K λ :=
h KSD k λ ,p for the goodness-of-ﬁt problem.
Note that, for the twosample and independence testing frameworks, since K λ is translation-invariant, the integral transformcorresponds to a convolution.
However, this is not true for the goodness-of-ﬁt framework as h KSD k λ ,p isnot translation-invariant.
We are now in a position to present our main contribution in Theorem 1:we derive a power guarantee condition for our tests using incomplete U -statistics, and a uniformseparation rate over Sobolev balls for the two-sample and independence settings.
Theorem 1.
(i) Let σ 22 ,λ := E (cid:2)
h λ ( Z, Z (cid:48) )
2 (cid:3) .
Assume (cid:107) p (cid:107) ∞ ≤ M and (cid:107) q (cid:107) ∞ ≤ M for some M > 0 .
For λ ∈ (0 , ∞ ) d with λ 1 · · · λ d < 1 , α ∈ (0 , e − 1 ) , β ∈ (0 , 1) , B 1 ≥ 2 α 2 (cid:0) ln (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) , if (cid:107) p − q (cid:107) 22 ≥ (cid:107) ( p − q ) − T λ ( p − q ) (cid:107) 22 + C N L ln(1 /α ) β σ 2 ,λ for some constant C > 0 , then P H 1 (cid:0) ∆ λα ( Z N )
= 0 (cid:1)
≤ β (type II error), where σ 2 ,λ ≤ C/ √ λ 1 · · · λ d for MMD and HSIC.(ii)
Fix R > 0 and s > 0 , and consider the bandwidths λ ∗ i :=
( N/L ) 2 / (4 s + d ) for i
= 1 , . .
, d .
For
MMD and HSIC, the uniform separation rate of ∆ λ ∗ α over the Sobolev ball S sd ( R ) is (up to a constant) ( N/L ) 2 s/ (4 s + d ) .
4 Level is non-asymptotic for the goodness-of-ﬁt case when using a
The proof of Theorem 1 relies on the variance and quantile bounds presented in Lemmas 1 and 2, andalso uses results of Albert et al.
(2022) and Schrab et al.
(2021, 2022) on complete U -statistics.
Thedetails can be found in Appendix F.
The power condition in Theorem 1 corresponds to a variance-biasdecomposition; for large bandwidths the bias term (ﬁrst term) dominates, while for small bandwidthsthe variance term (second term which also controls the quantile) dominates.
We recall that theminimax (i.e. optimal) rate over the Sobolev ball S sd ( R ) is (1 /N )
2 s/ (4 s + d ) for the two-sample (Liand Yuan, 2019, Theorem 5 (ii)) and independence (Albert et al., 2022, Theorem 4) problems.
Wehighlight that the rate for our incomplete U -statistic test has the same dependence in the exponent asthe minimax rate; that is ( N/L ) 2 s/ (4 s + d ) = (1 /N ) 2 s/ (4 s + d ) (cid:0) N 2 /L (cid:1) 2 s/ (4 s + d ) where we recall that
L ≤ N 2 is the design size and N is the sample size.
We reach the following conclusions
• If L (cid:16) N 2 then the test runs in quadratic time and we recover exactly the minimax rate.
• If N < L < N 2 then the rate still converges to
but we incur the cost ( N 2 /L )
2 s/ (4 s + d ) .
in the minimax rate (trade-off between computational efﬁciency and rate of convergence).
• If L ≤ N then there is no guarantee that the rate converges to 0.
Efﬁcient aggregated kernel tests using incomplete U -statistics
We now introduce our aggregated tests that combine single tests with different bandwidths.
Our aggregation scheme is similar to those in Fromont et al.
(2013), Albert et al.
(2022) and Schrab et al.
(2021, 2022), and can yield an adaptive test to the unknown smoothness parameter s of the
Sobolev ball S sd ( R ) , with relatively low price.
Let Λ be a ﬁnite collection of bandwidths, ( w λ ) λ ∈ Λ be associated weights satisfying (cid:80) λ
∈
Λ w λ ≤ 1 and u α be some correction term deﬁned shortly in
Equation (16).
Then, using the incomplete U -statistic
U
λ , we deﬁne our aggregated test ∆ Λ α as
[Equation]
for
[Equation]
The levels of the single tests are weighted and adjusted with a correction term
[Equation]
where the wild bootstrapped incomplete U -statistics (cid:101) U 1 λ , .
, (cid:101) U B 2 λ computed as in Equation (11)are used to perform a Monte Carlo approximation of the probability under the null, and where thesupremum is estimated using B 3 steps of bisection method.
Proposition 1, along with the reasoningof Schrab et al.
(2021, Proposition 8), ensures that ∆ Λ α has non-asymptotic level α for the twosample and independence cases, and asymptotic level α for the goodness-of-ﬁt case.
We refer to thethree aggregated test constructed using incomplete U -statistics as MMDAggInc, HSICAggInc and
KSDAggInc.
The computational complexity of those tests is O ( | Λ | ( B 1 + B 2 ) L ) , which means thatif L (cid:16) N as in Equation (10), the tests run efﬁciently in linear time in the sample size.
We formally record error guarantees of ∆ Λ α and derive uniform separation rates over Sobolev balls.
Theorem 2.
(i) Let σ 22 ,λ := E (cid:2)
h λ ( Z, Z (cid:48) )
2 (cid:3) .
Assume (cid:107) p (cid:107) ∞ ≤ M and (cid:107) q (cid:107) ∞ ≤ M for some
M > 0 .
Consider a collection Λ such that λ 1 · · · λ d < 1 for all λ ∈ Λ .
For α ∈ (0 , e − 1 ) ,
B 1 ≥
α 2 (cid:0) ln (cid:0) 8 β (cid:1) + α (1 − α ) (cid:1) , B 2 ≥ 8 α 2 ln (cid:0) 2 β (cid:1) , B 3 ≥ log 2 (cid:0) 4 α min λ ∈ Λ w − 1 λ (cid:1) , if (cid:107) p − q (cid:107) 22 ≥ min λ ∈ Λ (cid:18) (cid:107) ( p − q ) − T λ ( p − q ) (cid:107) 22 + C N L ln(1 / ( αw λ ))
β σ 2 ,λ (cid:19) for some constant C > 0 , then P H 1 (cid:0) ∆ Λ α ( Z N ) = 0 (cid:1) ≤ β (type II error), where σ 2 ,λ ≤ C/ √ λ 1 · · · λ d for MMD and HSIC.(ii) Consider the collections of bandwidths and weights (independent of R and s )
[Equation]
(cid:108)
[Equation]
L/N ln(ln(
[Equation]
/N ))
(cid:17)(cid:109)(cid:111)(cid:111) ,
[Equation]
For two-sample and independence problems, the uniform separation rate of ∆ Λ α over the Sobolev balls (cid:8) S sd ( R ) : R > 0 , s > 0 (cid:9) is (up to a constant) (cid:18)
ln(ln( L/N ))
L/N (cid:19) 2 s/ (4 s + d ) .
The extension from Theorem 1 to Theorem 2 has been proved for complete U -statistics in thetwo-sample (Fromont et al., 2013; Schrab et al., 2021), independence (Albert et al., 2022)
andgoodness-of-ﬁt (Schrab et al., 2022) testing frameworks.
The proof of Theorem 2 follows with thesame reasoning by simply replacing N with L/N as we work with incomplete U -statistics; this‘replacement’ is theoretically justiﬁed by Theorem 1.
From Theorem 2, the aggregated test ∆ Λ α isadaptive over Sobolev balls (cid:8) S sd ( R ) : R > 0 , s > 0 (cid:9) : the test ∆ Λ α does not depend on the unknownsmoothness parameter s (unlike ∆ λ ∗ α in Theorem 1) and achieves the minimax rate up to an iteratedlogarithmic factor and up to the cost incurred for efﬁciency of the test (i.e. L/N instead of N ).
Minimax optimal permuted quadratic-time aggregated independence test
Considering Theorem 2 with our incomplete U -statistic with full design D =
i N 2 for which L (cid:16) N 2 ,we have proved that the quadratic-time two-sample and independence aggregated tests using a wildbootstrap achieve the rate (ln(ln( N )) /N )
2 s/ (4 s + d ) over the Sobolev balls (cid:8) S sd ( R ) :
R > 0 , s > 0 (cid:9) .
This is the minimax rate (Li and Yuan, 2019; Albert et al., 2022), up to some iterated logarithmicterm.
For the two-sample problem, Kim et al.
(2022) and Schrab et al.
(2021) show that this is alsotrue using complete U -statistics with either a wild bootstrap or permutations.
Whether the equivalentstatement for independence test with permutations holds is unknown; the rate can be proved usingtheoretical (unknown) quantiles with a Gaussian kernel (Albert et al., 2022), but has not yet beenproved using permutations.
Kim et al.
(2022, Proposition 8.7) consider this problem, again using a
Gaussian kernel, but they do not obtain the correct dependence on α (i.e. ln(1 /α ) is replaced withα − 1 / 2 ), hence they cannot recover the desired rate.
As pointed out by Kim et al.
(2022, Section 8):‘It remains an open question as to whether [the power guarantee] continues to hold when α − 1 / 2 isreplaced by ln(1 /α ) ’.
We now prove that we can improve the α -dependence to ln(1 /α ) 3 / 2 for anybounded kernel of the form presented in Equation (12), and that this allows us to obtain the desiredrate over Sobolev balls (cid:8) S sd ( R ) : R > 0 , s > d/ 4 (cid:9) .
The assumption s > d/ 4 imposes a strongersmoothness restriction on p − q ∈ S sd ( R ) , which is similarly also considered by Li and Yuan (2019).
Theorem 3.
Consider the quadratic-time independence test using the complete U -statistic HSICestimator with a quantile estimated using permutations as done by Kim et al.
(2022, Proposition 8.7),with kernels as in (12) for bounded functions
K i and L j for i = 1 , . . .
, d x , j = 1 , . .
d y .(i)
Consider the assumptions of Theorem 1.
For ﬁxed R > 0 and s > d/ 4 , with the bandwidths λ ∗ i := N − 2 / (4 s + d ) for i = 1 , . . .
, d , the probability of type II error of the test is controlled by β when
[Equation]
for some constant C > 0 .
The uniform separation rate over the Sobolev ball S sd ( R
) is, up to a constant, (1 /N ) 2 s/ (4 s + d ) .(ii)
Consider the assumptions of Theorem 2, the uniform separation rate over the Sobolev balls (cid:8) S
s d ( R ) : R > 0 , s > d/ 4 (cid:9) is (cid:0) ln(ln( N )) /N
(cid:1) 2 s/ (4 s + d ) , up to a constant, with the collections Λ := (cid:110)(cid:0) 2 − (cid:96) , . . .
, 2 − (cid:96) (cid:1) ∈ (0 , ∞ )
d : (cid:96) ∈ (cid:110) 1 , . . .
, (cid:108) 2 d
[Equation]
2 (cid:16)
[Equation]
The proof of Theorem 3, in Appendix G, uses the exponential concentration bound of Kim et al.(2022, Theorem 6.3) for permuted complete U -statistics.
As discussed by Kim et al.
(2022, Section8.3), their proposed sample-splitting method can also be used to obtain the correct dependency on α .
Experiments
For the two-sample problem, we consider testing samples drawn from a uniform density on [0 , 1] dagainst samples drawn from a perturbed uniform density.
For the independence problem, the jointdensity is a perturbed uniform density on [0 , 1] d x + d y , the marginals are then simply uniform densities.
Those perturbed uniform densities can be shown to lie in Sobolev balls (Li and Yuan, 2019; Albertet al., 2022), to which our tests are adaptive.
For the goodness-of-ﬁt problem, we use a GaussianBernoulli Restricted Boltzmann Machine as ﬁrst considered by Liu et al.
(2016) in this testingframework.
Details on the experiments (e.g. model/test parameters) are presented in Appendix B.
We consider our incomplete aggregated tests MMDAggInc, HSICAggInc and KSDAggInc, withparameter R ∈ { 1 , . . .
, N − 1 } which ﬁxes the deterministic design to consist of the ﬁrst R subdiagonals of the N × N matrix, that is, D :=
{ ( i, i + r ) : i = 1 , . .
. , N − r for r = 1 , . .
, R } withsize |D| =
RN − R
( R − 1) / 2 .
We run our incomplete tests with R ∈ { 1 , 100 , 200 } and also considerthe complete test which uses the full design D = i N 2 .
We compare their performances with currentlinear-time state-of-the art tests: OST PSI (Kübler et al., 2020) which performs kernel selection usingpost selection inference, ME, SCF, FSIC and FSSD (Jitkrittum et al., 2016, 2017a,b) which evaluatethe witness functions at a ﬁnite set of locations chosen to maximize the power, and LSD (Grathwohlet al., 2020) which uses a neural network to learn the Stein discrepancy (see Appendix B for details).
Similar trends are observed across all our experiments in Figure 1, in the three testing frameworks,when varying the sample size, the dimension, and the difﬁculty of the problem (scale of perturbationsor noise level).
The linear-time tests AggInc R = 200 almost match the power obtained by thequadratic-time tests AggCom in all settings (except in Figure 1(i) where the difference is larger)while being computationally much more efﬁcient as can be seen in Figure 1(d,h,l).
The incompletetests with R = 100 has power only slightly below the one using R = 200 , and runs roughly twiceas fast (Figure 1(d,h,l)).
In all experiments, those three tests (AggInc R = 100 , 200 and AggCom)have signiﬁcantly higher power than the linear-time tests which optimize test locations (ME, SCF,
FSIC and FSSD); in the two-sample case the aggregated tests run faster for small sample size butslower for large sample size, in the independence case the aggregated tests run much faster, and inthe goodness-of-ﬁt case the tests optimizing test locations run faster.
While both types of tests arelinear, we note that the run times of the tests of Jitkrittum et al.
(2016, 2017a,b) increase slowerwith the sample size than our aggregated tests with R = 100 , 200 , but a ﬁxed computational cost isincurred for the optimization step, even for small sample sizes.
In the goodness-of-ﬁt framework, LSDmatches the power of KSDAggInc R = 100 when varying the noise level in Figure 1(k) (KSDAggInc
R = 200 has higher power), and matches the power of KSDAggInc R = 200 when varying thehidden dimension in Figure 1(j) where d x = 100 .
When varying the sample size in Figure 1(i), both
KSDAggInc tests with R = 100 , 200 achieve much higher power than LSD.
Unsurprisingly, AggInc
R = 1 , which runs much faster than all the aforementioned tests, has low power in every experiment.
For the two-sample problem, it obtains slightly higher power than OST PSI which runs even faster.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See framed bullet points at the
end of Section 5 (limitation for the case L < N ).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See main text for
Theorem 2 and appendices for all other proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] URL inSection 1: https://anonymous.4open.science/r/agginc-10EF/README.md .
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix B.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] For the time plots in Figure 1 we report the error bars
which represent the standard deviation (they are really small and cannot always been
seen).
For the power plots in Figure 1, since the test outputs are binary (0 or 1), there is
no need to include error bars since these are deterministic given the average which is
plotted.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix B.
(b) Did you mention the license of the assets?
[Yes] See Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] URL in Section 1: https://anonymous.4open.science/r/agginc-10EF/README.md .
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Details on MMD, HSIC and KSD
In this section, we present more details than those presented in Section 2 on the Maximum Mean
Discrepancy, on the Hilbert Schmidt Independence Criterion, and on the Kernel Stein Discrepancy.
Maximum Mean Discrepancy.
Gretton et al.
(2012a) introduce the Maximum Mean Discrepancy(MMD) which is a measure between probability densities p and q on R d .
It is deﬁned as theintegral probability metric (IPM; Müller, 1997) over a reproducing kernel Hilbert space H k (RKHS;
Aronszajn, 1950) with associated kernel k .
Gretton et al.
(2012a, Lemma 4) show that the MMD isequal to the H k -norm of the difference between the mean embeddings µ p ( u ) := E X ∼ p
[ k ( X, u )] andµ q ( u ) :=
E Y ∼ q [ k ( Y, u )] for u ∈ R d .
The square of the MMD is equal to
[Equation]
where X and X (cid:48) (respectively Y and Y (cid:48) ) are independent.
Using a characteristic kernel (Fukumizuet al., 2008; Sriperumbudur et al., 2011) guarantees that MMD 2 k ( p, q ) =
if and only if p = q ,a crucial property for using the MMD to construct a two-sample test.
With i.i.d.
samples X m :=( X i ) 1 ≤ i ≤ m from p and i.i.d.
samples Y n =
( Y j ) 1 ≤ j ≤ n from q , Gretton et al.
(2012a, Lemma 6)propose to use the unbiased quadratic-time MMD estimator (cid:92) MMD 2 k
( X m , Y n ) deﬁned as
[Equation]
where ˜ K XX and ˜ K YY are the kernel matrices K XX := (cid:0)
k
( X i , X j ) (cid:1) 1 ≤ i,j ≤ m and K YY :
= 493 (cid:0)
k
( Y i , Y j ) (cid:1) 1 ≤ i,j ≤ n with diagonal entries set to 0, where K XY :
= (cid:0) k
( X i , Y j ) (cid:1) 1 ≤ i ≤ m, 1 ≤ j ≤ n , andwhere 1 is a one-dimensional vector with all entries equal to 1 of variable length determined by thecontext 5 .
As noted by Kim et al.
(2022), this MMD estimator can be rewritten as a two-sample
U -statistic
(both of second order) (Hoeffding, 1992)(cid:92) MMD
[Equation]
X m
[Equation]
Y n
[Equation]
=
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation] [Equation]
where i ba denotes the set of all a -tuples drawn without replacement from { 1 , .
, b } so that (cid:12)(cid:12) i ba (cid:12)(cid:12) =
b · · · ( b − a + 1) , for example (cid:12)(cid:12) i m 2 (cid:12)(cid:12)
= m (
m − 1) , and where, for x 1 , x 2 , y 1 , y 2 ∈ R d , we leth MMD k ( x 1 , x 2 ; y 1 , y
[Equation]
y
[Equation]
y
[Equation]
This kernel can easily be symmetrized (Kim et al., 2022) using a symmetrization trick (Duembgen, 1998), this corresponds to working with ¯ h
[Equation]
k (
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
( x
[Equation]
, x
[Equation]
y
[Equation]
, y 2 ) and the MMD expression as a U -statistic still holds when replacing h MMD k with its symmetrizedvariant ¯ h MMD k .
Hilbert Schmidt Independence Criterion.
For a joint probability density p xy on R d x × R d y withmarginals p x on R d x and p y on R d y , Gretton et al.
(2005) introduce the Hilbert Schmidt Independence
Criterion (HSIC) which is deﬁned as
[Equation]
with kernels k on R d x and (cid:96) on R d y giving the product kernel κ (cid:0) ( x, y ) , ( x
[Table]
, y
[Table]
) (cid:1) :
= k ( x, x
[Table]
) (cid:96) ( y, y
[Table]
)on R d x × R d y .
With i.i.d.
pairs of samples Z N :=
(cid:0) Z i (cid:1) 1 ≤ i ≤ N = (cid:0) ( X i , Y i ) (cid:1) 1 ≤ i ≤ N drawn from p xy ,a natural unbiased HSIC estimator (Gretton et al., 2008; Song et al., 2012) is then
[Table]
which is a fourth-order one-sample U -statistic.
For z a = ( x a , y a ) ∈ R d x × R d y , a = 1 , . . .
, 4 , we let
[Equation]
We stress the fact that this HSIC estimator can actually be computed in quadratic time as shown by
Song et al.
(2012, Equation 5) who provide the following closed-form expression
[Equation]
where ˜ K and ˜ L are the kernel matrices K := (cid:0)
k
( X i , X j ) (cid:1) 1 ≤ i,j ≤ N and L := (cid:0) (cid:96) ( Y i , Y j ) (cid:1) 1 ≤ i,j ≤ N with diagonal entries set to 0.
Again, this kernel can be symmetrized (Song et al., 2012;
Kim et al., 2022) using a symmetrization trick (Duembgen, 1998), and the HSIC expression as a U -statistic still holds when replacing h HSIC k with its symmetrized variant
[Equation]
Kernel Stein Discrepancy.
For probability densities p and q on R d , Chwialkowski et al.
(2016) and
Liu et al.
(2016) introduce the Kernel Stein Discrepancy (KSD) deﬁned as
[Equation]
where the Stein kernel h p,k : R d × R d → R is deﬁned as
[Equation]
The Stein kernel satisﬁes the Stein identity E p [ h KSD k,p ( X, · )]
= 0 .
The KSD is particularly useful
forthe goodness-of-ﬁt setting with a model density p and i.i.d.
samples Z N :=
( Z i ) 1 ≤ i ≤ N drawn froma density q because it admits an estimator which does not require samples from the model p .
Thequadratic-time KSD estimator can be computed as the second-order one-sample U -statistic
[Equation]
, Z j ) = 1 (cid:62)
[Equation]
N ( N − 1) where ˜ H is the kernel matrix H := (cid:0) h
KSD k,p ( Z i , Z j ) (cid:1) 1 ≤ i,j ≤ N with diagonal entries set to 0.
The Steinkernel h KSD k is already symmetric, we can write ¯ h KSD k ( x, y ) :
=
h KSD k ( x, y all x,
consistency
Quantile estimation.
There exist many approaches to estimating the quantiles of the test statisticsunder the null hypothesis in the three frameworks: using the quantile of a known distributionfree asymptotic null distribution (Gretton et al., 2008, 2012a), sampling from an asymptotic nulldistribution with eigenspectrum approximation (Gretton et al., 2009), using permutations (Grettonet al., 2008; Albert et al., 2022; Kim et al., 2022; Schrab et al., 2021), using a wild bootstrap (Fromontet al., 2012; Chwialkowski et al., 2014, 2016; Schrab et al., 2021, 2022), using parametric bootstrap(Key et al., 2021; Schrab et al., 2022), using other bootstrap methods (Liu et al., 2016), to name buta few.
Permutation-based tests have been shown to correctly control the non-asymptotic level forthe two-sample (Schrab et al., 2021;
Kim et al., 2022) and independence (Albert et al., 2022; Kimet al., 2022) problems.
For the two-sample test, using a wild bootstrap also guarantees well-calibratednon-asymptotic level (Fromont et al., 2012; Schrab et al., 2021).
For the goodness-of-ﬁt setting, whilea wild bootstrap guarantees only control of the asymptotic level (Chwialkowski et al., 2016), usinga parametric bootstrap results in well-calibrated non-asymptotic level (Schrab et al., 2022).
In thiswork, we focus on the wild bootstrap approach, though we point out that our results also hold using aparametric bootstrap for the goodness-of-ﬁt setting as done by Schrab et al.
B Detailed experimental protocol
In this section, we present details on the experiments and on the tests considered.
We also showempirically that all the tests have well-calibrated levels.
Implementation and computational resources.
All experiments have been run on an AMD Ryzen
Threadripper 3960X 24 Cores 128Gb RAM CPU at 3.8GHz, except the LSD test (Grathwohl et al.
,2020) for which a neural network has been trained using a NVIDIA RTX A5000 24Gb Graphics Card.
The overall runtime of all the experiments is of the order of a couple of hours (signiﬁcant speedupcan be obtained by using parallel computing).
For the ME, SCF, FSIC and FSSD tests of Jitkrittumet al.
(2016, 2017a,b), and for the LSD test of Grathwohl et al.
(2020), we use the implementations ofthe respective authors, under the MIT license.
The implementation of our computationally efﬁcientaggregated tests, as well as the code for reproducibility of the experiments, are available here underthe MIT license.
Kernels.
For the two-sample and independence experiments, we use the Gaussian kernel 6 with equal bandwidths λ 1 = · · · = λ d = ˜ λ , which is deﬁned as
[Equation]
and similarly for the kernel (cid:96) µ .
As shown by Gorham and Mackey (2017), a more appropriate kernel for goodness-of-ﬁt testing is the IMQ (inverse multiquadric) kernel
[Equation]
x − y (cid:107)˜ λ(cid:33) − β k ∝ (cid:16) ˜ λ
2 + (cid:107) x − y (cid:107) 22 (cid:17) − β k for some β k ∈ (0 , 1) .
In our goodness-of-ﬁt experiments, we ﬁx the parameter β k = 0 .
Two-sample and independence experiments.
In our experiments, we consider perturbed uniformdensities, those can be shown to lie in Sobolev balls and are used to derive the minimax rates over
Sobolev balls for the two-sample and independence problems (Albert et al., 2022; Li and Yuan, 2019).
For the two-sample problem, we consider testing samples drawn from a uniform density againstsamples drawn from a perturbed uniform density, as considered by Schrab et al.
(2021, see Equation17 for formal deﬁnition and Figure 2 for illustrations).
We scale the perturbations so that the perturbeddensity takes value in the whole interval [0 , 2] , we then consider some inverse scaling parameter S ≥ 1such that it takes value in the interval [1 − 1 /S, 1 + 1 /S ] .
Intuitively, as S increases, the perturbationis shrunk.
In Figure 1(a,d), we consider 2 perturbations with inverse scaling parameter S = 2 indimension d = 1 and vary the sample size N ∈ { 200 , 400 , 600 , 800 , 1000 } .
In Figure 1(b), we varythe dimension d ∈ { 1 , 2 , 3 , 4 } for 1 perturbation with S = 1 and N = 1000 .
In Figure 1(c), we use 1perturbation with d = 1 and N = 1000 , we vary the inverse scaling parameter S ∈ { 1 , 2 , 3 , 4 , 5 } .
For the independence problem, we draw samples from the joint perturbed uniform density in dimensiond x + d y , the marginals are simply uniform densities in dimensions d x and d y , respectively.
Weﬁx d x = 1 and vary d y exactly as in the two-sample setting.
The parameters for the independenceexperiments in Figure 1(e–h) are the same as those of the two-sample experiments in Figure 1(a–d)detailed above (with the only difference that for Figure 1(f) we consider d y ∈ { 1 , 2 , 3 } ).
Goodness-of-ﬁt experiments.
In Figure 1(i–l), we use a Gaussian-Bernoulli Restricted Boltzmann
Machine (GBRBM) with the same setting considered by Liu et al. (2016), Grathwohl et al.
(2020) and
Schrab et al.
This is a hidden variable model with a continuous observable variable in R d x anda hidden binary variable in {− 1 , 1 } d h , the joint density is intractable but the score function admits aclosed form.
The GBRBM has parameters b ∈ R d x and c ∈ R d h , which are drawn from Gaussianstandard distributions, and a parameter B ∈ R d x × d h .
For the model p , the elements of B are sampleduniformly from {− 1 , 1 } (i.i.d.
Rademacher variables).
The samples come from a GBRBM q withthe same parameters as the model p but where some Gaussian noise N (0 , σ ) is injected into theelements of B .
In Figure 1(i,l), we consider dimensions d x = 50 and d h = 40 with noise standarddeviation σ = 0 .
and we vary the sample size N ∈ { 200 , 400 , 600 , 800 , 1000 } .
In Figure 1(j),we ﬁx d x = 100 , N = 1000 , σ = 0 .
and we vary the hidden dimension d h ∈ { 20 , 40 , 60 , 80 } .
For ﬁxed observed dimension d x , as the hidden dimension d h increases the size of B ∈ R d x × d hbecomes larger, so there is more evidence of the noise being injected, which makes the problemeasier.
Hence, the test power increases as d h increases for ﬁxed d x .
In Figure 1(k), we considerdimensions d x = 50 and d h = 40 with sample size N = 1000 , we vary the noise standard deviationsσ ∈ { 0 , 0 . 01 , 0 .
AggInc tests.
For MMDAggInc and KSDAggInc, we use the collection of bandwidths
[Equation]
where
[Equation]
where 1 d is a d -dimensional vector with all entries equal to 1.
For HSICAggInc, we work with the collection of pairs of bandwidths
[Equation]
for the kernels k λ and (cid:96) µ deﬁned in Equation (12), where
[Equation]
and
[Equation]
All aggregated tests are run with uniform weights deﬁned as w λ := 1 / | Λ | for all λ ∈ Λ .
The designchoice consists of R sub-diagonals of the kernel matrix for R ∈ { 1 , 100 , 200 } , it is formally deﬁnedin Section 8.
We use B 1 = 500 and B 2 = 500 wild bootstrapped statistics to estimate the quantilesand the probability under the null for the correction in Equation (16), respectively.
For that correctionterm, we use B 3 = 50 steps of bisection method to approximate the supremum.
ME, SCF, FSIC and FSSD tests.
Jitkrittum et al.
(2016) uses the two-sample tests ME and SCFproposed by Chwialkowski et al.
(2015) with features which are chosen to maximise a lower boundon the test power.
The ME test is based on analytic Mean Embeddings while the SCF test usesthe difference in Smooth Characteristic Functions.
For the independence problem, Jitkrittum et al.(2017a) constructs a FSIC test which uses their proposed normalised Finite Set Independence
Criterion.
Jitkrittum et al.
(2017b) proposes a goodness-of-ﬁt test based on the Finite Set Stein
Discrepancy (FSSD).
All those tests utilise test statistics which evaluate the witness function of either
MMD, HSIC, or KSD, at some test locations (i.e. features) chosen on held-out data to maximise testpower.
For the two-sample SCF test, the test locations are in the frequency domain rather than inthe spatial domain.
All tests are used with 10 test locations which are chosen on half of the data, asdone in the experiments of Jitkrittum et al.
(2016, 2017a,b).
The ME and SCF tests use the quantilesof the known chi-square asymptotic null distributions.
The FSIC test uses 500 permutations tosimulate the null hypothesis and compute the test threshold to ensure well-calibrated non-asymptoticlevel.
The FSSD test simulates 2000 samples from the asymptotic null distribution (weighted sumof chi-squares) with the eigenvalues being computed from the covariance matrix with respect to theobserved sample.
For the two-sample and independence tests, the bandwidths of the Gaussian kernelsare selected during the optimization procedure.
For the goodness-of-ﬁt test, the bandwidth of the
LSD test.
The Kernelised Stein Discrepancy (KSD) is a Stein Discrepancy (Gorham and Mackey,2017) where the class of functions is taken to be the unit ball of a reproducing kernel Hilbert space(RKHS).
Grathwohl et al.
(2020) propose to instead consider some more expressive class of functionsconsisting of neural networks, resulting in the Learned Stein Discrepancy (LSD).
For goodness-of-ﬁttesting, they propose to split the data into training (80%), validation (10%) and testing (10%) sets.
They construct a test statistic which is asymptotically normal under both H 0 and H 1 .
Using thetraining set, they train the parametrised neural network to maximise the test power by optimizinga proxy for it which is derived following the reasonings of Gretton et al.
(2012b), Sutherland et al.(2017) and Jitkrittum et al.
(2017b).
They perform model selection on the validation set.
Finally,they run the test on the testing set using the quantile of the asymptotic normal distribution under thenull.
As in the experiments of Grathwohl et al.
(2020), a 2-layer MLP with 300 units per layer andwith Swish nonlinearity (Ramachandran et al., 2017) is used.
Their model is trained using the Adamoptimizer (Kingma and Ba, 2014) for 1000 iterations, with dropout (Srivastava et al., 2014), withweight decay of strength 0 . 0005 , with learning rate 10 − 3 , and with L 2 regularising strength 0 .
Well-calibrated levels.
All tests are run with level α = 0 .
05 , it is veriﬁed in Tables 1 to 6 that all testshave well-calibrated levels for the three testing frameworks, when varying either the sample size orthe dimension.
The levels plotted are averages obtained across 200 repetitions, this explains the smallﬂuctuations observed from the desired test level α = 0 .
The settings of those six experimentscorrespond to the settings of the experiments presented in Figure 1(a,b,e,f,i,j) detailed above, withthe difference that we are working under the null hypothesis (i.e. perturbed uniform densities arereplaced with uniform densities, and the noise standard deviation for the GBRBB is set to σ = 0 ).
[Table] [Table]
uniform densities varying the sample size
[Figure]
[Table] [Table]
[Table]
C Proof of Proposition 1
The asymptotic level of the goodness-of-ﬁt test using a wild bootstrap follows from the results of
Shao (2010), Leucht and Neumann (2013), Chwialkowski et al.
As pointed out by
Schrab et al.
for the complete U -statistics, the KSD test statistic and the wild bootstrapped
KSD statistics are not exchangeable under the null, and hence non-asymptotic level cannot be provedusing the result of Romano and Wolf (2005, Lemma 1).
The non-asymptotic level for the two-sample test follows exactly from the reasoning of Schrab et al.(2021, Propositon 1).
The fact that we work with incomplete U -statistics rather than with theircomplete counterparts does not affect the proof of exchangeability of U 1 λ , . .
, U B 1 +1 λ .
For the independence problem, Albert et al.
(2022, Proposition 1) prove that the quadratic-time HSICestimator and the permuted test statistics are exchangeable under the null hypothesis, it remainsto be shown that this also holds in our incomplete setting using a wild bootstrap.
Assuming thatexchangeability under the null holds, the desired non-asymptotic level α can then be guaranteed usingthe result of Romano and Wolf (2005, Lemma 1), exactly as done by Albert et al.
(2022, Proposition1).
We now prove that U 1 λ , . . .
, U B 1 +1 λ for the independence problem are exchangeable under the null.
Since U 1 λ , . . . , U B 1 λ are i.i.d.
given the data, they are exchangeable under the null.
So, we need toprove that
[Equation]
is, under the null, distributed like
[Table]
(18) where (cid:15) 1 , . . .
, (cid:15) N are i.i.d.
Rademacher random variables.
Using the result of Schrab et al.
Appendix B, Proposition 11), considering the identity s 1 ( i ) = i for i = 1 , . .
, 2 (cid:98) N/ 2 (cid:99) , and the swapfunction s − 1 ( i ) =
i + (cid:98) N/ 2 (cid:99) and s − 1 ( i + (cid:98) N/ 2 (cid:99) )
=
i for i = 1 , . .
, 2 (cid:98) N/ 2 (cid:99) , we have
[Table]
[Table] [Table]
= (cid:88)
[Table]
[Equation]
where Z (cid:15) N := (cid:16) (cid:0) X i , Y s (cid:15)i ( i ) (cid:1) (cid:17) 1 ≤ i ≤ N with Y s (cid:15)i ( i ) ∈
{ Y i , Y i + (cid:98) N/ 2 (cid:99) } for i = 1 , . . .
, (cid:98) N/ 2 (cid:99) and
Y s (cid:15)i ( i ) ∈
{ Y i , Y i −(cid:98) N/ 2 (cid:99) } for i =
+ (cid:98) N/ 2 (cid:99) , . .
, 2 (cid:98) N/ 2 (cid:99) .
Now, under the null the variables( X i ) 1 ≤ i ≤ N and ( Y i ) 1 ≤ i ≤ N are independent, so Z (cid:15)N is distributed like Z N .
We deduce that
HSIC k,(cid:96) (cid:0) Z N ; D (cid:98) N/ 2 (cid:99) (cid:1) and HSIC k,(cid:96) (cid:0) Z (cid:15)N ; D (cid:98) N/ 2 (cid:99) (cid:1) have the same distribution under the null, andhence, that the terms in Equations (17) and (18) also have the same distribution under the null.
Wededuce that U 1 λ , . .
, U B 1 +1 λ for the independence problem are exchangeable under the null, whichcompletes the proof.
D Proof of Lemma 1
Consider the case of ﬁxed design.
Using the variance expression of Lee (1990, Theorem 2, p.190), we have
[Equation]
where f i is the number of pairs of sets in the design D that have i elements in common.
The pairs of sets in D with 2 elements in common are { ( { i, j } , { i, j } ) :
( i, j ) ∈ D} , so f 2 = |D| .
We now calculate the number of pairs of sets in D with 1 element in common.
We start with a pair ( i, j ) ∈ D (there are |D| such pairs).
The number of pairs in D which have one element in common with ( i, j ) is upper bounded by the number of pairs in i N 2 which have one element in common with ( i, j ) , those are {{ i, r } : 1 ≤ r ≤ N, r (cid:54) = i } ∪ {{ j, r } : 1 ≤ r ≤ N, r (cid:54) =
j } of size smaller than 2 N .
We deduce that f 1 ≤ 2 N |D| .
Combining those results, we obtain
[Equation]
as desired.
Let us now consider the random design case.
Recall that using the variance expression of the complete
U -statistic U of Lee (1990, Theorem 3 p.12), we can obtain that
[Equation]
as done by Kim et al.
(2022, Appendix E) and Albert et al.
(2022, Lemma 10).
Using the result of
Lee (1990, Theorem 4 p.193), the variance of the incomplete U -statistic U can be expressed in terms of the variance of the complete U -statistic U .
For random design with replacement, we have
[Equation]
Letting S :
= N ( N − 1) / 2 , for random design without replacement, we have var (cid:0) U (cid:1) =
[Equation]
E Proof of Lemma 2
We rely on the concentration bound for i.i.d.
Rademacher chaos of de la Peña and Giné (1999,
Corollary 3.2.6) which as presented in Kim et al.
(2022, Equation 39) takes the form
[Equation]
 for some constant C > 0 and for every t ≥ 0 , where (cid:15) 1 , . .
, (cid:15) N are i.i.d.
Rademacher random variables taking values in {− 1 , 1 } .
Letting
[Equation]
for ( i,
[Equation]
∈
[Equation]
N
[Equation]
we obtain
[Equation]
( i,j ) ∈
i Nh ( Z i , Z j )
 − 1   which concludes the proof.
F Proof of Theorem 1
We start by reviewing the steps of proofs of Albert et al.
(2022) and Schrab et al.
(2021, 2022) whoprove that, for each of the three respective testing frameworks, a sufﬁcient condition to ensure controlof the probability of type II error for the quadratic-time test is
[Equation]
Those quadratic-time tests use the complete U -statistics deﬁned in Equations (1), (3) and (5), whichwe denote as U λ .
The key results for their proofs rely on deriving variance and quantile bounds.
The variance bound is of the form
[Equation]
(20) where they show that for h λ ∈ (cid:8) h MMD k λ , h HSIC k λ ,
[Equation]
, h KSD k λ
[Equation]
p (cid:9) deﬁned in Equations (2), (4) and (6)
[Equation]
and
[Equation]
C
[Equation]
(21) where the last inequality holds only for h λ ∈ (cid:8) h MMD k λ , h
[Equation]
µ
[Equation]
The quantile bound (Schrab et al., 2021, Proposition 4) is of the form
[Equation]
for δ ∈ (0 , 1) , where (cid:98) q λ,U,B 1 1 − α is the quantile obtained using B 1 wild bootstrapped similarly to the onedeﬁned in Equation (13) but using the complete U -statistic.
Relying on Dvoretzky–Kiefer–Wolfowitz inequality (Dvoretzky et al., 1956; Massart, 1990), Schrab et al.
(2021, Proposition 4) show that itsufﬁces to prove the bound
[Equation]
for the true wild bootstrap quantile (cid:98) q λ,U, ∞ 1 − α without ﬁnite approximation.
Combining those variance and quantile bounds using Chebyshev’s inequality (Chebyshev, 1899), they obtain a condition guaranteeing power in terms of the MMD, HSIC and KSD.
By expressing these three measures as an RHKS inner product
[Equation]
they obtain the condition in Equation (19) which guarantees high power in terms of (cid:107) p − q (cid:107) 22 .
Albertet al.
(2022) and Schrab et al.
(2021) then derive the minimax rate (1 /N )
2 s/ (4 s + d ) over the Sobolevball S
sd ( R ) for the independence and two-sample tests using the bandwidths λ ∗ i :=
(1 /N )
2 / (4 s + d )for i = 1 , . . .
, d .
For Theorem 1, we need to obtain the condition in Equation (19) with 1 /N replaced by N/L .
Hence,following their reasoning, in order to prove Theorem 1 (i) & (ii), it sufﬁces to derive variance andquantiles bounds for incomplete U -statistics which have the form of Equations (20) and (22) with1 /N replaced by N/L , which we now do.
Using the variance bound for incomplete U -statistics
U λ of Lemma 1, together with the fact that the design size L := |D| is smaller than N 2 so that 1 /L
=
L/L 2 ≤ N 2 /L 2 , we obtain for ﬁxed design that
[Equation]
we get the same bound for random design since
[Equation]
as desired.
For the quantile bound, we use Lemma 2 which, for A 2 λ :
= L − 2 (cid:80) ( i,
j ) ∈
i N 2 h λ ( Z i , Z j ) 2 , gives
[Equation]
Setting α := 2 exp( − Ct/A λ ) , we obtain
[Equation]
ln(1 /α ) for a different constant C > 0 since α ∈ (0 , e − 1 ) .
For δ ∈ (0 , 1) , using Markov’s inequality, weobtain
[Equation]
where
[Equation]
using Equation (21).
We deduce that
[Equation]
as desired, which concludes the proof.
G Proof of Theorem 3G.1 Proof of Theorem 3 (i)
In this setting, we consider as proposed by Kim et al.
(2022, Equation 32) the permuted HSIC complete U -statistic
[Equation]
for a permutation π of the indices { 1 , . .
. , N } , and for h HSIC k,(cid:96) as deﬁned in Equation (4).
Applying the exponential concentration bound of Kim et al.
(2022, Theorem 6.3), which uses theresult of de la Peña and Giné (1999, Theorem 4.1.12), we obtain that there exist constants C 1 , C 2 > 0such that
[Equation]
(23) where
[Equation]
and
[Equation]
for some constant C > 0 since the functions K 1 , . . .
K d x and L 1 , . . .
, L d y are bounded, and wherewe recall our notational convention that d := d x + d y and λ d x + i :
= µ i for i = 1 , . . .
, d y .
Using the reasoning of Schrab et al.
(2021, Proposition 3), we see that the results of Albert et al.(2022, Equations C.17, C.18 & C.19) hold not only for the Gaussian kernel but more generally forany kernels of the form of Equation (12).
Those results give us that there exists a constant C > 0depending on M and d such that
[Equation]
d .
We deduce that
[Equation]
As explained in Appendix F, by relying on Dvoretzky–Kiefer–Wolfowitz inequality (Dvoretzky et al., 1956; Massart, 1990) as done by Schrab et al.
(2021, Proposition 4), it is sufﬁcient to prove upper bounds for the true permutation quantile (cid:98) q λ, ∞
1 − α without ﬁnite approximation.
From Equation (23), we obtain that this quantile satisﬁes
[Equation]
Using Markov’s inequality and bounds obtained above, we get that
[Equation]
holds with probability at least 1 − δ where δ ∈ (0 , 1) , and where the constants are different on eachline.
Now, recall that 4 s > d and λ ∗ i = N − 2 / (4 s + d ) for i = 1 , . . .
, d , so that
[Equation]
which gives
[Equation]
holding with probability at least 1 − δ , since α ∈ (cid:0) 0 , e − 1 (cid:1) .
By combining this result with the reasoning of Albert et al.
(2022) as explained in Appendix F, we obtain that the probability of type II error of the test is controlled by β ∈ (0 , 1) when
[Equation]
We have recovered the correct dependency with respect to N and λ with an improved α -dependencyof ln(1 /α ) 3 / 2 compared to the α − 1 / 2 dependency obtained by Kim et al.
(2022, Proposition 8.7).
The proof of minimax optimality of the quadratic-time test with ﬁxed bandwidth λ ∗ does not dependon the α -dependency and can be derived in both our setting and the one of Kim et al.
(2022) usingquantiles obtained from permutations by following the reasoning of Albert et al.
(2022, Corollary2).
We obtain that the uniform separation rate over the Sobolev ball S sd ( R ) is, up to a constant,(1 /N )
2 s/ (4 s + d ) .
The improved α -dependency is crucial for deriving the rate of the aggregatedquadratic-time test over Sobolev balls because the weights appear in the α -term (i.e. α is replaced byαw λ which depends on the sample size N ).
G.2 Proof of Theorem 3 (ii)
Similarly to the proofs of Albert et al.
(2022, Corollary 3) and Schrab et al.
(2021, Corollary 10), consider (cid:96) ∗ :=
(cid:108) 4 s + d log(cid:16)
N ln(ln( N ))
(cid:17)(cid:109) ≤ (cid:108)d log(cid:16)
N ln(ln( N ))
(cid:17)(cid:109) and the bandwidth λ ∗ := (cid:0) 2 − (cid:96) ∗ , . . .
, 2 − (cid:96) ∗ (cid:1) ∈
Λ which satisﬁes ln (cid:18)w λ ∗ (cid:19)
≤
C ln ( (cid:96) ∗ ) ≤ C ln(ln( N ))
as w λ ∗ := 6 π − 2 ( (cid:96) ∗ ) − 2 , and  (cid:18) ln(ln( N ))
N (cid:19) 2 / (4 s + d ) ≤ λ ∗ i ≤ (cid:18) ln(ln( N ))
N (cid:19) 2 / (4 s + d ) for i = 1 , . . .
, d .
Since 4 s > d , we have
[Equation]
all with different constants.
By Equation (24), we get that
[Equation] [Equation]
holds with probability at least 1 − δ for δ ∈ (0 , 1) .
If the largest term is the ﬁrst one, then we get
[Equation] [Equation]
where the constant also depends on α .
The result then follows exactly as in the proofs of Albert et al.(2022, Corollary 3) and Schrab et al.
(2021, Corollary 10).
So, we consider the case where the secondterm is the largest one, so that
[Equation]
We have recovered the same dependency as in Equation (26) when considering the ﬁrst term as the largest one, and the proof then follows exactly the ones of Albert et al.
(2022, Corollary 3) and Schrab et al.
(2021, Corollary 10).
We have treated both cases in Equation (25), we conclude that the uniform separation rate over the Sobolev balls (cid:8) S sd ( R ) : R > 0 , s > d/ 4 (cid:9) of the quadratic-time aggregated test using a quantile obtained with permutations is (up to a constant)
[Equation]
H Additional experimentsH.1 Updated experiments of Figure 1
In Figure 2, we reproduce the experiments of Figure 1 with the following changes.
All MMDAggInc and KSDAggInc tests use the collection of 21 bandwidths
[Equation]
where
[Equation]
For HSICAggInc, we work with the collection of 25 bandwidths
[Equation]
We have updated OST PSI which uses the same collections of bandwidths as the AggInc tests.
The
AggInc tests retain high power using those larger collections of bandwidths.
For goodness-of-ﬁt, we also compare against L1 IMQ and Cauchy RFF, two tests proposed by
Huggins and Mackey (2018).
L1 IMQ performs similarly to FSSD in our RBM experiments, whichis in line with the experiments of Huggins and Mackey (2018, Figure 4d).
The Cauchy RFF performsvery well in almost all settings; it has the same test power as KSDAggInc with R = 200 , and itobtains slightly higher power in a few settings.
However, we found that L1 IMQ and Cauchy RFFtake 16 seconds to run while our tests run in less than a second. also
H.2 Aggregation experiments
As we increase the number of bandwidths, the power ﬁrst increases (as the bandwidths we add aremore relevant) and then the power reaches a plateau.
This is also the case for HSICAggInc until 49pairs of bandwidths are included.
After this point, the power starts decreasing quickly as can be seenin Figure 4.
H.3 MNIST Experiments
We run experiments on the real-world MNIST dataset consisting of images of digits in dimension784.
For the two sample problem, one distribution P consists of all digits and the other distribution is Q iwhere Q 1 is only the odd digits, Q 2 is Q 1 with 0, Q 3 is Q 2 with 2, ..., Q 5 consists of images of alldigits expect 8.
This experiment was also performed by Schrab et al.
For the independence problem, we pair images of digits with the value of the digit.
To make theproblem more difﬁcult, for some percentage of the data we pair an image of a digit with the value ofa random digit.
For the goodness-of-ﬁt problem, we use samples of the true MNIST dataset and as model we use a
Normalizing Flow trained on the MNIST dataset (which admits a density).
We had access only to thesamples and the values of the score function evaluated at those (not the score function itself).
In thissetting where the score function does not admit a closed form, using FSSD, L1 IMQ or Cauchy RFFis very challenging and was not considered for this rebuttal.
Overall, we observe the same trends in Figure 5 as we did in Figure 2: AggInc tests perform well inthis high-dimensional real-world setting as well.