A permutation-free kernel two-sample test
Anonymous Author(s)
AffiliationAddress email
Abstract
The kernel Maximum Mean Discrepancy (MMD) is a popular multivariate distancemetric between distributions that has found utility in two-sample testing.
The usualkernel-MMD test statistic is a degenerate U-statistic under the null, and thus it hasan intractable limiting distribution.
Hence, to design a level- α test, one usuallyselects the rejection threshold as the (1 − α )
-quantile of the permutation distribution.
The resulting nonparametric test has finite-sample validity but suffers from largecomputational cost, since every permutation takes quadratic time.
We propose thecross-MMD, a new quadratic-time MMD test statistic based on sample-splittingand studentization.
We prove that under mild assumptions, the cross-MMD has alimiting standard Gaussian distribution under the null.
Importantly, we also showthat the resulting test is consistent against any fixed alternative, and has minimaxrate-optimal power against local alternatives (with Gaussian kernels).
For largesample-sizes, our new cross-MMD provides a significant speedup over the MMD,for only a slight loss in power.
Introduction
We study the two-sample testing problem; given X =
( X 1 , . . .
X n ) i.i.d.
∼
P and Y =(
Y 1 , . . .
, Y m ) i.i.d.
∼ Q , we test the null hypothesis H 0 : P = Q against the alternative H 1 : P ̸ = Q .
This is a nonparametric hypothesis problem with a composite null hypothesis and a compositealternative hypothesis.
It finds applications in diverse areas such as testing microarray data, clinicaldiagnosis and database attribute matching (Gretton et al., 2012a).
A popular approach to solving this problem is based on the kernel-MMD distance between the twoempirical distributions (Gretton et al., 2006).
Given a positive definite kernel k , the kernel-MMDdistance between two distributions P and Q on X , denoted by MMD( P, Q ) , is defined as
MMD( P, Q ) =
∥ µ − ν ∥ k , where µ ( · ) =
(cid:90)
[Equation]
[Equation]
Above, µ and ν are commonly called “kernel mean maps”, and denote the kernel mean embeddingsof the distributions P and Q into the reproducing kernel Hilbert space (RKHS) associated with thepositive-definite kernel k , and ∥ · ∥ k denotes the corresponding RKHS norm.
Under mild conditionson the positive definite kernel k (Sriperumbudur et al., 2011), MMD is a metric on the space ofprobability distributions.
Gretton et al.
(2006) suggested using an empirical estimate of the squareddistance as the test statistic.
In particular, given X and Y , define the test statistic [Equation] Y
[Equation]
where h ( x, x ′ , y, y ′ )
= k ( x, x ′ )
− k ( x, y ′ )
− k
( y, x ′ )
+ k
( y, y ′ ) .
The above statistic has an alternative form that only takes quadratic time to calculate.
The MMD test rejects the null if (cid:92) MMD 2 exceeds a suitable threshold τ ≡ τ ( α ) that ensures the falsepositive rate is at most α .
For “characteristic kernels”, this test is consistent against fixed alternatives,meaning the power (the probability of rejecting the null when P ̸ = Q ) increases to one as m, n → ∞ .
The difficulty in practically determining τ will play a key role in this paper.
It is well known thatwhen P = Q , (cid:92) MMD 2 is an instance of a “degenerate two-sample U-statistic”, meaning that:
Under H 0 , E P [ h ( x, X ′ , y, Y ′ )]
=
E Q [ h ( X, x ′ , Y, y ′ )]
= 0 .
(Above, x, y, x ′ , y ′ are fixed, and the expectations are over X, Y, X ′ ,
Y ′ i.i.d.
∼ P .)
As a consequence,its (limiting) null distribution is unwieldy; it is an infinite sum of independent χ 2 random variablesweighted by the eigenvalues of an operator that depends on the kernel k and the underlying distribution
P (see equation (10) in Appendix A).
Since P is unknown, one cannot explicitly calculate τ .
In practice, a permutation-based approach is commonly used, where τ is set as the (1 − α )
-quantileof the kernel-MMD statistic computed on B permuted versions of the aggregate data ( X , Y ) .
Theresulting test has finite-sample validity, but its practical applicability is reduced due to the highcomputational complexity; if B = 200 permutations are used, the (permuted) test statistic must berecomputed 201 times, rather than once (usually, B is chosen between 100 and 1000).
Due to the high computational complexity of the permutation test, some permutation-free alternativesfor selecting τ have been proposed.
However, as we discuss in Section 1.2, these alternatives areeither too conservative in practice (using concentration inequalities), or heuristics with no theoreticalguarantees (Pearson curves and Gamma approximation) or are only shown to be consistent in thesetting where the kernel k does not vary with n (spectral approximation).
We later recap somecomputationally efficient alternatives to (cid:92) MMD 2 , but these have significantly lower power.
As far as we are aware, there exists no method in literature based on the kernel-MMD that is (i)permutation-free (does not require permutations), (ii) consistent against any fixed alternative, (iii)achieves minimax rate-optimality against local alternatives, and (iv) is correct for both the fixedkernel setting ( k is fixed as m, n → ∞ ) and the changing kernel setting ( k changes as a function ofm, n , for instance, by selecting the scale parameter of a Gaussian kernel in a data-driven manner).
Our work delivers a novel and simple test satisfying all four desirable properties.
We propose anew variant of the kernel-MMD statistic that (after studentization) has a standard Gaussian limitingdistribution under the null in both the fixed and changing kernel settings, in low- and high-dimensionalsettings.
There is a computation-statistics tradeoff: our permutation-free test loses about a √ 2 factorin power compared to the standard kernel-MMD test, but it is hundreds of times faster.
Remark 1.
Let P ( X ) denote the set of all probability measures on the observation space X , wherewe often use X = R d for some d ≥ 1 .
For simplicity, in the above presentation, the distributions
P, Q , kernel k and dimension d did not change with sample size, and this is the setting considered inthe majority of the literature.
Later, we prove several of our results in a significantly more generalsetting where P, Q, d, k can vary with n, m .
Under the null, this provides a much more robust type-Ierror control in high-dimensional settings, even with data-dependent kernels.
Under the alternative,this provides a more fine-grained power result.
To elaborate on the latter, we assume that for everyn, m , the pair ( P, Q )
=
( P n , Q n ) ∈ P (1) n ⊂ P ( X ) × P ( X ) for some sequence {P (1) n : n, m ≥ 2 } .
The class P (1) n is such that with increasing n and m , it contains pairs ( P ′ , Q ′ )
that are increasinglycloser in some distance measure ϱ ; thus the alternatives can approach the null and be equal in thelimit.
That is, ∆ n,m := inf ( P ′ ,Q ′ )
∈P (1) n ϱ ( P ′ , Q ′ ) decreases with n, m , and such alternatives arecalled local alternatives (as opposed to fixed alternatives).
This framework allows us to characterizethe detection boundary of a test, that is, the smallest perturbation from the null (in terms of ∆ n,m )that can be consistently detected by a test.
Paper outline.
We present an overview of our main results in Section 1.1 and discuss relatedwork in Section 1.2.
In Section 2, we present the cross-MMD statistic and obtain its limiting nulldistribution in Section 2.1.
We demonstrate its consistency against fixed alternatives and minimax rateoptimality against smooth local alternatives in Section 2.2.
Section 3 contains numerical experimentsthat demonstrate our theoretical claims.
All our proofs are in the supplement.
1.1 Overview of our main results
We propose a variant of the quadratic time kernel-MMD statistic of (1) that relies on two key ideas:(i) sample splitting and (ii) studentization .
In particular, we split the sample X of size n ≥ 2 into 82 X 1 and X 2 of sizes n 1 ≥ 1 and n 2 ≥ 1 , respectively (and Y of size m ≥ 2 into Y 1 and Y 2 of sizesm 1 ≥ 1 and m 2 ≥ 1 ), and define the two-sample cross kernel-MMD statistic c (cid:92) MMD 2 as follows:
[Equation]
Our final test statistic is ¯ c (cid:92)
[Equation]
, where (cid:98) σ is an empirical variance introduced in (4).
Our first set of results show that quite generally, ¯ c (cid:92) MMD 2 has an N (0 , 1) asymptotic null distribution.
Theorem 4 obtains this result in the setting where both the kernel k and null distribution P are fixed.
This is then generalized to deal with changing kernels (for instance, Gaussian kernels with data-drivenbandwidth choices) in Theorem 5.
Finally, in Theorem 14 in Appendix A, we significantly expandthe scope of these results by also allowing the null distribution to change with n , and also weakeningthe moment conditions required by Theorem 5.
n chosen by the median heuristic for different choices of n , m and d , and with samples X and Y drawn from a multivariate Gaussian distribution with identity covariance matrix.
The second figure compares the power curves of the two-sample test using the ¯ c (cid:92) MMD 2 statistic with the kernel-MMD permutation test (with 200 permutations).
The final figure plots the power vs computation time for the two tests.
The size of the markers are proportional to the sample-size used in the test.
Our main methodological contribution is the “ cMMD test”, denoted Ψ , which rejects the null if¯ c (cid:92) MMD 2 exceeds z 1 − α , which is the (1 − α ) -quantile of N (0 , 1) .
Formally,
[Equation]
By the previous results, Ψ has type-I error at most α , meaning that E [Ψ( X , Y )] ≤ α under the null.
We next study the power of the cMMD test Ψ in Section 2.2.
First, in the fixed alternative case, i.e.,when the distributions P ̸
= Q do not change with n , we show in Theorem 6, that the cMMD testimplemented with any characteristic kernel is consistent under a bounded fourth moment condition.
Next, we consider the more challenging case of local alternatives, i.e., when the distributions,
P n ̸ = Q n , change with n .
In Theorem 7, we first identify general sufficient conditions for the cMMDtest to be uniformly consistent over a class of alternatives.
Then, we specialize this to the casewhen P n and Q n admit densities p n and q n with ∥ p n − q n ∥
L 2 ≥ ∆ n for some ∆ n → 0 .
We showin Theorem 8, that the cMMD test with a Gaussian kernel k s n ( x, y ) =
exp( − s n ∥ x − y ∥ 22 ) , withscale parameter s n increasing at an appropriate rate can consistently detect the local alternatives{ ∆ n : n ≥ 1 } decaying at the minimax rate.
Finally, we note that while our primary focus in the paper is on the special case of kernel-MMDstatistic, the ideas involved in defining the cMMD statistic can be extended to the case of generaltwo-sample U-statistics.
We describe this in Appendix D.1, and obtain sufficient conditions forasymptotic Gaussian limit of the resulting statistic, possibly of independent interest.
1.2 Comparisons to related work
Attempts to avoid permutations.
There have been some prior attempts to avoid permutations, butthey are either heuristics (no provable type-I error control) or have poor power (higher type-II error).
The first approach to obtaining a rejection threshold is based on large deviation bounds for the MMDstatistic (Gretton et al., 2006, § 3) or the permuted MMD statistic (Kim, 2021, § 5).
The resulting testsare distribution-free, but they tend to be too conservative (type-I error much less than α , resultingin low power).
Another approach involves choosing the threshold based on parametric estimatesof the limiting null distribution.
For example, Gretton et al.
(2006) suggested fitting to the Pearsonfamily of densities based on the first four moments, while Gretton et al.
(2009) introduced a morecomputationally efficient method using a two-parameter Gamma approximation.
However, both ofthese methods are heuristic and do not have any consistency guarantees.
Gretton et al.
(2009) introduced a spectral method for approximating the null distribution using theeigendecomposition of the gram matrix.
They showed that the resulting distribution converges to thetrue null distribution as long as the square roots of the eigenvalues associated with the kernel operatorare summable.
While this method is asymptotically consistent, the conditions imposed on the kernelare more stringent than that used in our work.
Furthermore, this method was shown to be consistentonly in the fixed kernel (or low-dimensional) setting.
Hence, it is unknown whether the results carryover to the case of kernels varying with sample size or high-dimensional settings.
This method is alsocomputationally nontrivial due to the need for a full eigendecomposition.
Keeping only the top feweigenvectors is another heuristic, but this introduces an extra hyperparameter and loses theoreticalguarantees; as a result this method is rarely used in practice.
Our methods are simpler (no extrahyperparameter), faster (closed-form threshold), and more robust (type I error guarantees also holdfor changing kernels, and in high-dimensional settings).
Changing the statistic: block-MMD and linear-MMD statistics.
An idea more closely related toours is that changing the test statistic itself would help make it cheaper to compute and also yield atractable limiting distribution.
One approach splits the observations into disjoint blocks, compute thekernel-MMD statistic on every block, and the final test statistic averages over all the blocks.
If the sizeof each block is fixed, we get a linear-time kernel-MMD (Gretton et al., 2012a,b).
The case of blocksizes increasing with n, m was studied by Zaremba et al.
; Ramdas et al.
Depending onthe block size ( b ), the computational complexity of block-MMD statistic varies from linear (constantb ) to quadratic ( b = Ω( n ) ).
Further, if b = o ( n ) , then one gets a Gaussian null distribution as well.
Our proposed statistic is fundamentally different from the block-MMD statistics, despite both beingincomplete U-statistics (Lee, 1990).
In particular, the block-MMD statistics can be understoodas building a block-diagonal approximation of the gram matrix.
On the other hand, our proposedcross-MMD statistic uses the off-diagonal blocks of the gram matrix, exactly the blocks that theblock-MMD with two blocks ( b = n/ 2 ) excludes!
The reason that this is a sensible thing to do isnontrivial, and our test is motivated quite differently from the block-MMD.
In fact, when b = n/ 2 ,the block-MMD does not have a Gaussian null, but the cross-MMD does.
For the block-MMD with b = o ( n ) , the Gaussian null distribution is achieved at the cost of suboptimalpower, as observed empirically in Zaremba et al.
(2013), and
proved by Reddi et al.
(2015) for thecase of linear-MMD and Ramdas et al.
(2015) for general block-MMD statistics.
In particular, theirpower is worse by factors scaling with n , which means that they are not minimax rate optimal.
Incontrast, our test uses exactly half the elements of the gram matrix, and its power is about a √ 2 worsethan the MMD test, independent of n and d , and we prove explicitly that it achieves the minimax rate.
Beyond the kernel-MMD.
The literature on two-sample testing is vast, and one can move evenfurther away from the kernel-MMD (than the block-MMD) while retaining some of its intuition.
Forexample, Chwialkowski et al.
(2015) proposed a linear-time test statistic by computing the averagesquared-distance between the empirical kernel embeddings at J randomly drawn points.
When thekernel k being used is analytic, in addition to being characteristic and integrable, they showed thatthe limiting distribution under null for this statistic is a combination of J independent χ 2 randomvariables.
However, similar to the spectral method of Gretton et al.
(2009), the high-dimensionalbehavior of this statistic is unknown.
Further, the authors only proved consistency against fixedalternatives, but its power is not known to be minimax rate optimal.
In contrast, our statistic has thesame limiting distribution in low- and high-dimensional settings even with changing kernels, and isprovably minimax rate optimal, and it is much more direct tweak of the usual MMD test.
One-sample (goodness-of-fit) testing.
Kim and Ramdas (2020) proposed and analyzed a similarstudentized cross U-statistic in the simpler one-sample setting.
Our work has different motivations:our primary goal in this paper is to design a permutation-free kernel-MMD test that does notsignificantly sacrifice the power, while Kim and Ramdas (2020) pursued the related but differentgoal of dimension-agnostic inference, which means having the same limiting distribution in lowdimensional and high-dimensional settings.
Nevertheless, our results can be seen as an extension oftheir methods to two-sample testing.
Our proofs also build on their advances, but we require a moreinvolved analysis since in their case the second distribution is known (making it a point null).
Deriving the cross-MMD test
In this section, we present our test statistic and investigate its limiting distribution.
First note that thesquared kernel-MMD distance between two probability measures P and Q can be expressed as aninner product, namely ⟨ µ − ν, µ − ν ⟩ k .
The usual kernel-MMD statistic is obtained by plugging theempirical kernel embeddings into this inner product expression and removing the diagonal terms tomake it unbiased.
Our proposal instead considers pairs of empirical estimates ( (cid:98) µ 1 , (cid:98) µ 2 ) and ( (cid:98) ν 1 , (cid:98) ν 2 )constructed via sample splitting, and use the inner product between (cid:98) µ 1 − (cid:98) ν
1 and (cid:98) µ 2 − (cid:98) ν 2 instead.
This careful construction allows us to obtain a Gaussian limiting distribution after studentization.
To elaborate, recall from Section 1.1 that we partition X into X 1 and X 2 , and similarly Y into 180 Y 1 and Y 2 .
We then compute empirical kernel embeddings based on each partition, yielding
[Equation]
n
[Equation]
· ) , (cid:98) ν 1 := m − 1 1 (cid:80) m 1 j =1 k ( Y j , · ) and (cid:98) ν 2 :=m − 1 2 (cid:80) m 2 j ′ =1 k
( Y j ′ , · ) .
Using these embeddings coupled with the kernel trick, the cross U-statistic (2)can be written as c (cid:92) MMD 2 = ⟨ (cid:98) µ 1 − (cid:98) ν 1 , (cid:98) µ 2 − (cid:98) ν 2 ⟩ k .
To further motivate our test statistic, denote U X,i := ⟨ k
( X i , · ) , (cid:98) µ 2 − (cid:98) ν 2 ⟩ k for i = 1 , . . . , n 1 and U Y,j := ⟨ k
( Y j , · ) , (cid:98) µ 2 − (cid:98) ν
2 ⟩ k forj = 1 , . . .
, m 1 .
Then the cross U-statistic can be viewed as the difference between two sample means:
[Equation]
U
[Equation]
j .
Since the summands are independent conditional on
X 2 and Y 2 , one may expect that c (cid:92) MMD 2 is approximately Gaussian after studentization.
Our resultsin Section 2.1 formalize this intuition under standard moment conditions, where it takes some care toremove the above conditioning, since we care about the unconditional distribution.
Let us further denote the sample means of U X,
i ’s and U Y,j ’s by ¯ U X
and ¯ U Y , respectively, and define
[Equation]
Now we have completed the description of our studentized cross U-statistic ¯ c (cid:92) MMD 2 =
c (cid:92) MMD 2 / (cid:98) σ ,and the resulting test Ψ in (3).
The asymptotic validity of the cMMD test is guaranteed by Theorem 14that establishes the asymptotic normality of ¯ c (cid:92) MMD 2 under the null.
Remark 2 (Computational Complexity) .
The overall cost of computing the statistic ¯ c (cid:92) MMD 2 is
O (cid:0) ( n + m ) 2 (cid:1) , and in particular, both c (cid:92) MMD 2 and (cid:98) σ have quadratic complexity.
To see this, notethat c (cid:92) MMD 2 can be expanded into ⟨ (cid:98) µ 1 , (cid:98) µ 2 ⟩ k + ⟨ (cid:98) ν 1 , (cid:98) ν 2 ⟩ k − ⟨ (cid:98) µ 1 , (cid:98)
ν 2 ⟩ k − ⟨ (cid:98) ν 1 , (cid:98) µ 2 ⟩ k .
Each of theseterms can be computed in O (cid:0) ( n + m ) 2 (cid:1) .
Similarly, each term in the summations defining (cid:98) σ 2 X and(cid:98)
σ 2 Y also require O (cid:0) ( n + m ) 2 (cid:1) computation, implying that the (cid:98) σ also has O (( n + m ) 2 ) complexity.
Remark 3.
To simplify notation in what follows, we denote m as m n , where m n is some unknownnondecreasing sequence such that lim n →∞ m n = ∞ .
This still permits m, n to be separate quantitiesgrowing to infinity at potentially different rates, but it allows us to index the sequence of problemswith the single index n (rather than m, n ).
We will use k n , d n , X n , P n and Q n to indicate thatquantities could (but do not have to) change as n increases, and drop the subscript when they are fixed.
Furthermore, unless explicitly stated, we will focus on the balanced splitting scheme, i.e., n 1 =
⌊ n/ 2 ⌋and m 1 =
⌊ m/ 2 ⌋ in what follows, because we currently see no apriori reason to split asymmetrically.
2.1 Gaussian limiting distribution under the null hypothesis
As shown in Figure 1, the empirical distribution of ¯ c (cid:92) MMD 2 resembles a standard normal distributionfor various choices of m , n and dimension d under the null.
In this section, we formally prove thisstatement.
Recalling the mean embedding µ from (1), define
[Equation]
Theorem 4.
Suppose that k and P do not change with n .
If 0 < E P [¯ k
( X, X ′ ) 4 ] < ∞ for
X, X ′ i.i.d.
∼ P , then ¯ c (cid:92) MMD 2 d −→ N (0 , 1) .
We next present a more general result (that implies Theorem 4) for data-dependent kernels k n ,motivated by the common practice of picking kernel hyperparameters in a data-driven manner.
Theorem 5.
Suppose P is fixed, but the kernel k n changes with n .
If
[Equation]
and
[Equation] exists [Equation] where ( λ l,n ) ∞ l =1 denote the eigenvalues of ¯ k introduced in (16) , then we have ¯ c (cid:92) MMD 2 d −→ N (0 , 1) .
It is easy to check that condition (6) is trivially satisfied if the kernels { k n : n ≥ 1 } are uniformlybounded by some constant; prominent examples are the Gaussian or Laplace kernel with a sample-sizedependent bandwidth.
Thus, the above condition really exists to handle unbounded kernels and heavytailed distributions.
To motivate this requirement, we recall Bentkus and Götze (1996)
(see Fact 10in Appendix A) who proved a studentized CLT for i.i.d.
random variables in a triangular arraysetup: W 1 ,n , W 2 ,n , . .
. , W n,n i.i.d.
∼ P n .
Define V n = √ n (cid:80) ni
=1 W i,n / (cid:113)(cid:80) i
( W i,n − ¯ W n ) 2 where¯ W n = ( (cid:80) i W i,n ) /n .
They showed that a sufficient condition for the asymptotic normality of V n isthat lim n →∞ E P n [ W 31 ,n ] / (cid:113) E P n [ W 21 ,n ] 3 n = 0 .
(This last condition is trivially true if P n does notchange with n , meaning that the triangular array setup is irrelevant and W 1 ,n can be replaced by W 1 .)
Our requirement is slightly stronger: condition (6) with ¯ k n ( X 1 , X 2 ) replaced by W 1 ,n implies theprevious condition of Bentkus and Götze (1996) (details in Remark 11 in Appendix A).
We needthis stronger condition, because the terms in the definition of c (cid:92) MMD 2 are not i.i.d.
(indeed, not evenindependent), and thus we cannot directly apply the result of Bentkus and Götze (1996).
Instead,we take a different route by first conditioning on the second half of data ( X 2 , Y 2 ) , then showing theconditional asymptotic normality of the standardized c (cid:92) MMD 2 (i.e., divided by conditional standarddeviation instead of empirical), and finally showing that the ratio of conditional and empirical standarddeviations converge in probability to 1 (see Appendix B).
Finally, we note that the result of Theorem 5 can be further generalized in several ways: (i) instead ofa fixed P and changing k n , we can consider a sequence of pairs { ( P n , k n ) : n ≥ 1 } changing withn , (ii) we can let P n ∈ P (0) n , for a class of distributions changing with n , and obtain the Gaussianlimit uniformly over all elements of P (0) n , and finally, (iii) the moment requirements on ¯ k n stated incondition in (6) can also be slightly weakened.
We state and prove this significantly more generalversion of Theorem 5 in Appendix B.
2.2 Consistency against fixed and local alternatives
Here, we show that the cMMD test Ψ introduced in (3) is consistent against a fixed alternative andalso has minimax rate-optimal power against smooth local alternatives separated in L 2 norm.
We first show that analogous to Theorem 4, cMMD is consistent against fixed alternatives.
Theorem 6.
Suppose P, Q, k do not change with n , and P ̸ = Q .
If k is a characteristic kernelsatisfying 0 < E P [ ¯ k
( X 1 , X 2 ) 4 ] < ∞ , and 0 < E Q [ ¯ k ( Y 1 , Y 2 ) 4 ] < ∞ , then the cMMD test isconsistent, meaning it has asymptotic power 1 .
The moment conditions required above are mild, and are satisfied trivially, for instance, by boundedkernels such as the Gaussian kernel.
The “characteristic” condition is also needed for the consistencyof the usual MMD test (Gretton et al., 2012a), and is also satisfied by the Gaussian kernel.
Recalling Remark 1, we next consider the more challenging setting where d n , k n can change withn , and ( P n , Q n ) can vary within a class P (1) n ⊂ P ( X n ) × P ( X n ) that can also change with n .
Wepresent a sufficient condition under which the cMMD test Ψ is consistent uniformly over P (1) n .
Defineγ n := MMD( P n , Q n ) , which is assumed nonzero for each n but could approach zero in the limit.
Theorem 7.
Let { δ n : n ≥ 2 } denote any positive sequence converging to zero.
If
[Equation]
V denotes variance,
[Equation]
cMMD [Equation] is consistent.
Note that while any sequence { δ n } converging to zero suffices for the general statement above, thecondition (7) is easiest to satisfy for slowly decaying δ n , such as δ n = 1 / log log n for instance.
The sufficient conditions for consistency of Ψ stated in terms of (cid:98) σ and c (cid:92) MMD 2 in (7) can also betranslated into equivalent conditions on the kernel function k n , similar to (6), and we present thedetails in Appendix C. Importantly, if P n , Q n , d n are fixed and k n is bounded, then both E [ (cid:98) σ 2 ] and
V ( c (cid:92) MMD 2 ) are O
(1 /n ) , and γ n is a constant, so the condition is trivially satisfied, and in fact theabove condition is even weaker than the fourth-moment condition of the previous theorem.
2.3 Minimax rate optimality against smooth local alternatives
We now apply the general result of Theorem 7 to the case where the distributions P n and Q n admit
Lebesgue densities p n and q n that lie in the order β Sobolev ball for some β > 0 , defined as
W β, 2 ( M ) :
= { f : X → R | f is a.s. continuous, and (cid:82) (1 + ω 2 ) β/ 2 ∥F ( f )( ω )
∥ 2 dw < M < ∞} .
Formally, we define the null and alternative class of distributions as follows:
[Equation]
with density
[Equation]
and
[Equation]
with densities
[Equation]
for some sequence ∆ n decaying to zero.
In particular, we assume that under H 0 , P n = Q n and
P n ∈ P (0) n , while under H 1 , we assume that ( P n , Q n ) ∈ P (1) n .
Our next result shows that for suitably chosen scale parameter, the cMMD test Ψ with the Gaussiankernel is minimax rate-optimal for the above class of local alternatives.
For simplicity, we statethis result with n = m , noting that the result easily extends to the case when there exist constants0 < c ≤ C , such that c ≤ n/m ≤ C .
Theorem 8.
Consider the case when n = m , and let { ∆ n : n ≥ 1 } be a sequence such thatlim n →∞ ∆ n n 2 β/ ( d +4 β )
= ∞ .
On applying the cMMD test Ψ with the Gaussian kernel k s n ( x, y ) =
exp( − s n ∥ x − y ∥ 22 ) , if we choose the scale as s n ≍ n 4 / ( d +4 β ) , then we have
[Equation]
The proof of this statement is in Appendix C, and it follows by verifying that the conditions requiredby Theorem 7 are satisfied for the above choices of ∆ n and s n .
Remark 9.
Li and Yuan (2019, Theorem 5 (ii)) showed a converse of the above statement: iflim n →∞ ∆ n n 2 β/ ( d +4 β ) < ∞ , then there exists an α ∈ (0 , 1) such that any asymptotically level α test(cid:101)
Ψ must have lim n →∞ inf ( P,Q ) ∈P (∆ n ) E P,Q [ (cid:101) Ψ
( X , Y )]
< 1 .
Hence, the sequence of { ∆ n : n ≥ 1 }used in Theorem 8 represents the smallest L 2 -deviations that can be detected by any test, and (8)shows that our cMMD test Ψ can detect such changes, establishing its minimax rate-optimality.
Experiments
We now present experimental validation of the theoretical claims of the previous section.
In particular,our experiments demonstrate that (i) the limiting null distribution of ¯ c (cid:92) MMD 2 is N (0 , 1) under a wide range of choices of dimension d , sample sizes n, m and the kernel k , and (ii) the power of our cMMDtest is competitive with the kernel-MMD permutation test.
We now describe the experiments in moredetail.
Additional experimental results are reported in Appendix E.
Limiting null distribution of ¯ c (cid:92) MMD 2 .
We showed in Theorem 14 that the statistic ¯ c (cid:92) MMD 2 has alimiting normal distribution under some mild assumptions.
We empirically test this result when Xand Y are drawn from N ( 0 ,
I d ) with 0 denoting the all-zeros vector in R d , and in particular, studythe effects of (i) dimension: d = 10 versus d = 500 , (ii) skewness of the samples: n/m = 1 versusn/m = 0 .
1 , and (iii) choice of kernel: Gaussian versus Quadratic, both with data-dependent scaleparameters using median heuristic.
As shown in the first row of Figure 2, the distribution of ¯ c (cid:92) MMD 2 is robust to all these effects, and isclose to N (0 , 1) in all cases.
In contrast, the distribution of the kernel-MMD statistic scaled by itsempirical standard deviation (obtained using 200 bootstrap samples) in the bottom row of Figure 2shows strong changes with these parameters.
We present additional figures and details of theimplementation in Appendix E.
Evaluation of the power of Ψ .
For d ≥ 1 and j ≤ d , let a ϵ,j denote the element of R d withfirst j coordinates equal to ϵ , and others equal to 0 .
We consider the two-sample testing problemwith P = N ( 0 , I d ) Q = N ( a ϵ,j , I d ) for different choices of ϵ and d and j .
We compare theperformance of our proposed test Ψ with the kernel-MMD permutation test, implemented with
B = 200 permutations, and plot the power-curves (over 400 trials) in Figure 3.
We also proposea heuristic for predicting the power of the permutation test (denoted by ρ perm ) using the power ofΨ (denoted by ρ Ψ ) as follows (with Φ denoting the standard normal cdf, and z α its α -quantile):
[Equation]
This heuristic is motivated by the power expressions derived by Kim and Ramdas (2020) for theproblems of one-sample Gaussian mean and covariance testing (we discuss this further in Appendix E).
The term √ 2 in the above expression quantifies the price to pay for sample-splitting.
As shownin Figure 3, this heuristic gives us an accurate estimate of the power of the kernel-MMD permutationtest, without incurring the computational burden.
We now use ROC curves to compare the tradeoff between type-I and type-II errors for the usual MMD,linear and block-MMDs with our ¯ c (cid:92) MMD 2 .
We use the same distributions P = N ( 0 , I d ) and Q =
N ( a ϵ,j , I d ) as before, and plot results for ( d, j, ϵ ) ∈ { (10 , 5 , 0 . 2) , (100 , 20 , 0 . 15) , (500 , 100 , 0 . 1) }in Figure 4.
Due to sample splitting, the tradeoff achieved by our proposed statistic is slightlyworse than that of (cid:92) MMD 2 , but significantly better than other computationally efficient variants of kernel-MMD statistics.
More details about the implementation, and additional figures are presentedin
Appendix E.
Conclusion and future work
[Figure] We proposed a variant of the kernel-MMD statistic, called cross-MMD, based on the ideas of samplesplitting and studentization, and showed that it has a standard normal limiting null distribution.
Usingthis key result, we introduced a permutation-free (and hence computationally efficient) MMD testfor the two-sample problem.
Experiments indicate that the power achieved by our test is within a 324 √ 2 -factor of the power of the kernel-MMD permutation test (that requires recomputing the statistichundreds of times).
In other words, our results achieve the following favorable tradeoff: we get asignificant reduction in computation at the price of a small reduction in power .
Sejdinovic et al.
(2013) establish in some generality that distance-based two-sample tests (like theenergy distance (Székely and Rizzo, 2013)) can be viewed as kernel-MMD tests with a particularchoice of kernel k .
Hence our results are broadly applicable to distance-based statistics as well.
Since two-sample testing and independence testing can be reduced to each other, it is an interestingdirection for future work to see if the ideas developed in our paper can be used for designingpermutation-free versions of kernel-based independence tests like HSIC (Gretton et al., 2007) ordistance covariance (Székely and Rizzo, 2009; Lyons, 2013).
Our techniques seem to rely on the specific structure of two-sample U-statistics of degree 2 .
Extendingthese to more general U-statistics of higher degrees is another important question for future work.
A final question is to figure out whether it is possible to achieve minimax optimal power using asub-quadratic time test statistic.
One potential approach would be to work with a kernel approximatedby random Fourier features (Rahimi and Recht, 2007; Zhao and Meng, 2015).
Depending on thenumber of random features, our test statistic can be computed in sub-quadratic time and it wouldbe interesting to see whether the resulting test can still be minimax optimal in power.
We leave thisimportant question for future work. Checklist
For all authors...
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] In cases where computation costis not an issue, the power achieved by our test will be slightly smaller (approximatelyby a factor of √ 2 ) as compared to the kernel-MMD permutation test.
(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d)
Have you read the ethics review guidelines and ensured
that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs are inthe appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will includethe code with the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We include all the details of the experiments in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The experiments were run on aworkstation, whose specifications are provided in Appendix E4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]