A permutation-free kernel two-sample test
Abstract
The kernel Maximum Mean Discrepancy (MMD) is a popular multivariate distance metric between distributions that has found utility in two-sample testing.
The usual kernel-MMD test statistic is a degenerate U-statistic under the null, and thus it has an intractable limiting distribution.
Hence, to design a level- α test, one usually selects the rejection threshold as the (1 − α )
-quantile of the permutation distribution.
The resulting nonparametric test has finite-sample validity but suffers from large computational cost, since every permutation takes quadratic time.
We propose the cross-MMD, a new quadratic-time MMD test statistic based on sample-splitting and studentization.
We prove that under mild assumptions, the cross-MMD has a limiting standard Gaussian distribution under the null.
Importantly, we also show that the resulting test is consistent against any fixed alternative, and when using the Gaussian kernel, it has minimax rate-optimal power against local alternatives.
For large sample sizes, our new cross-MMD provides a significant speedup over the MMD, for only a slight loss in power.
Introduction
We study the two-sample testing problem; given X =
( X 1 , . . .
X n ) i.i.d.
∼
P and Y =
( Y 1 , . . .
, Y m ) i.i.d.
∼ Q , we test the null hypothesis H 0 : P = Q against the alternative H 1 : P ̸ = Q .
This is a nonparametric hypothesis problem with a composite null hypothesis and a composite alternative hypothesis.
It finds applications in diverse areas such as testing microarray data, clinical diagnosis and database attribute matching (Gretton et al., 2012a).
A popular approach to solving this problem is based on the kernel-MMD distance between the two empirical distributions (Gretton et al., 2006).
Given a positive definite kernel k , the kernel-MMD distance between two distributions P and Q on X , denoted by MMD( P, Q ) , is defined as
[Equation]
k , where
[Equation]
Above, µ and ν are commonly called “kernel mean maps”, and denote the kernel mean embeddings of the distributions P and Q into the reproducing kernel Hilbert space (RKHS) associated with the positive-definite kernel k , and ∥ · ∥ k denotes the corresponding RKHS norm.
Under mild conditions on the positive definite kernel k (Sriperumbudur et al., 2011), MMD is a metric on the space of probability distributions.
Gretton et al.
(2006) suggested using an empirical estimate of the squared distance as the test statistic.
In particular, given X and Y , define the test statistic
[Equation]
1 ≤ i ̸ = i ′ [Equation] n (cid:88) 1 ≤ j ̸ = j ′
[Equation]
m h ( X i [Equation] X i ′
[Equation] [Equation] where h ( x, x ′ , y, y ′ ) = k ( x, x ′ )
− k ( x, y ′ )
− k
( y, x ′ )
+ k
( y, y ′ ) .
The above statistic has an alternative form that only takes quadratic time to calculate.
The MMD test rejects the null if (cid:92) MMD 2 exceeds a suitable threshold τ ≡ τ ( α ) that ensures the false positive rate is at most α .
For “characteristic kernels”, this test is consistent against fixed alternatives, meaning the power (the probability of rejecting the null when P ̸ = Q ) increases to one as m, n → ∞ .
The difficulty in practically determining τ will play a key role in this paper.
It is well known that when P = Q , (cid:92) MMD 2 is an instance of a “degenerate two-sample U-statistic”, meaning that:
[Equation]
(Above, x, y, x ′ , y ′ are fixed, and the expectations are over X, Y, X ′ ,
Y ′ i.i.d.
∼ P .)
As a consequence, its (limiting) null distribution is unwieldy; it is an infinite sum of independent χ 2 random variables weighted by the eigenvalues of an operator that depends on the kernel k and the underlying distribution
P (see equation (10) in Appendix A).
Since P is unknown, one cannot explicitly calculate τ .
In practice, a permutation-based approach is commonly used, where τ is set as the (1 − α )
-quantile of the kernel-MMD statistic computed on B permuted versions of the aggregate data ( X , Y ) .
The resulting test has finite-sample validity, but its practical applicability is reduced due to the high computational complexity; if B = 200 permutations are used, the (permuted) test statistic must be recomputed 201 times, rather than once (usually, B is chosen between 100 and 1000).
Due to the high computational complexity of the permutation test, some permutation-free alternatives for selecting τ have been proposed.
However, as we discuss in Section 1.2, these alternatives are either too conservative in practice (using concentration inequalities), or heuristics with no theoretical guarantees (Pearson curves and Gamma approximation) or are only shown to be consistent in the setting where the kernel k does not vary with n (spectral approximation).
We later recap some computationally efficient alternatives to (cid:92) MMD 2 , but these have significantly lower power.
As far as we are aware, there exists no method in literature based on the kernel-MMD that is (i) permutation-free (does not require permutations), (ii) consistent against any fixed alternative, (iii) achieves minimax rate-optimality against local alternatives, and (iv) is correct for both the fixed kernel setting ( k is fixed as m, n → ∞ ) and the changing kernel setting ( k changes as a function of m, n , for instance, by selecting the scale parameter of a Gaussian kernel in a data-driven manner).
Our work delivers a novel and simple test satisfying all four desirable properties.
We propose a new variant of the kernel-MMD statistic that (after studentization) has a standard Gaussian limiting distribution under the null in both the fixed and changing kernel settings, in low- and high-dimensional settings.
There is a computation-statistics tradeoff: our permutation-free test loses about a √ 2 factor in power compared to the standard kernel-MMD test, but it is hundreds of times faster.
Remark 1.
Let P ( X ) denote the set of all probability measures on the observation space X , where we often use X = R d for some d ≥ 1 .
For simplicity, in the above presentation, the distributions
P, Q , kernel k and dimension d did not change with sample size, and this is the setting considered in the majority of the literature.
Later, we prove several of our results in a significantly more general setting where P, Q, d, k can vary with n, m .
Under the null, this provides a much more robust type-I error control in high-dimensional settings, even with data-dependent kernels.
Under the alternative, this provides a more fine-grained power result.
To elaborate on the latter, we assume that for every n, m , the pair ( P, Q ) =
( P n , Q n ) ∈ P (1) n ⊂ P ( X ) × P ( X ) for some sequence {P (1) n : n, m ≥ 2 } .
The class P (1) n is such that with increasing n and m , it contains pairs ( P ′ , Q ′ )
that are increasingly closer in some distance measure ϱ ; thus the alternatives can approach the null and be equal in the limit.
That is, ∆ n,m := inf ( P ′ ,Q ′ )
∈P (1) n ϱ ( P ′ , Q ′ ) decreases with n, m , and such alternatives are called local alternatives (as opposed to fixed alternatives).
This framework allows us to characterize the detection boundary of a test, that is, the smallest perturbation from the null (in terms of ∆ n,m ) that can be consistently detected by a test.
Paper outline.
We present an overview of our main results in Section 1.1 and discuss related work in Section 1.2.
In Section 2, we present the cross-MMD statistic and obtain its limiting null distribution in Section 2.1.
We demonstrate its consistency against fixed alternatives and minimax rateoptimality against smooth local alternatives in Section 2.2.
Section 3 contains numerical experiments that demonstrate our theoretical claims.
All our proofs are in the supplement.
1.1 Overview of our main results
We propose a variant of the quadratic time kernel-MMD statistic of (1) that relies on two key ideas: (i) sample splitting and (ii) studentization .
In particular, we split the sample X of size n ≥ 2 into X 1 and X 2 of sizes n 1 ≥ 1 and n 2 ≥ 1 , respectively (and Y of size m ≥ 2 into Y 1 and Y 2 of sizes m 1 ≥ 1 and m 2 ≥ 1 ), and define the two-sample cross kernel-MMD statistic x (cid:92) MMD 2 as follows:
[Equation]
Our final test statistic is ¯x (cid:92) MMD 2 := x (cid:92) MMD 2 / (cid:98) σ , where (cid:98) σ is an empirical variance introduced in (4).
Our first set of results show that quite generally, ¯x (cid:92) MMD 2 has an N (0 , 1) asymptotic null distribution.
Theorem 4 obtains this result in the setting where both the kernel k and null distribution P are fixed.
This is then generalized to deal with changing kernels (for instance, Gaussian kernels with data-driven bandwidth choices) in Theorem 5.
Finally, in Theorem 15 in Appendix A, we significantly expand the scope of these results by also allowing the null distribution to change with n , and also weakening the moment conditions required by Theorem 5.
−   0 .0 .0 .0 . Statistic Value
P r ob a b ilit y d e n s it y Null distribution of xMMD d= d= N(0,1) 100 200 3000 .0 .0 .0 . Sample-Size (n+m)
P o w e r Power vs Sample-Size MMD -perm xMMD 10 −10 −10 −0 .0 .0 .0 . Running time (seconds)
P o w e r
Power vs Computation MMD -perm xMMD Our main methodological contribution is the “ xMMD test”, denoted Ψ , which rejects the null if ¯x (cid:92) MMD 2 exceeds z 1 − α , which is the (1 − α ) -quantile of N (0 , 1) .
Formally,
[Equation]
By the previous results, Ψ has type-I error at most α , meaning that E [Ψ( X , Y )] ≤ α under the null.
We next study the power of the xMMD test Ψ in Section 2.2.
First, in the fixed alternative case, i.e., when the distributions P ̸ =
Q do not change with n , we show in Theorem 7, that the xMMD test implemented with any characteristic kernel is consistent under a bounded fourth moment condition.
Next, we consider the more challenging case of local alternatives, i.e., when the distributions,
P n ̸ = Q n , change with n .
In Theorem 8, we first identify general sufficient conditions for the xMMD test to be uniformly consistent over a class of alternatives.
Then, we specialize this to the case when P n and Q n admit densities p n and q n with ∥ p n − q n ∥
L 2 ≥ ∆ n for some ∆ n → 0 .
We show in Theorem 9, that the xMMD test with a Gaussian kernel k s n ( x, y ) =
exp( − s n ∥ x − y ∥ 22 ) , with scale parameter s n increasing at an appropriate rate can consistently detect the local alternatives { ∆ n : n ≥ 1 } decaying at the minimax rate.
Finally, we note that while our primary focus in the paper is on the special case of kernel-MMD statistic, the ideas involved in defining the xMMD statistic can be extended to the case of general two-sample U-statistics.
We describe this in Appendix D.1, and obtain sufficient conditions for asymptotic Gaussian limit of the resulting statistic, possibly of independent interest.
1.2 Comparisons to related work
Attempts to avoid permutations.
There have been some prior attempts to avoid permutations, but they are either heuristics (no provable type-I error control) or have poor power (higher type-II error).
The first approach to obtaining a rejection threshold is based on large deviation bounds for the MMD statistic (Gretton et al., 2006, § 3) or the permuted MMD statistic (Kim, 2021, § 5).
The resulting tests are distribution-free, but they tend to be too conservative (type-I error much less than α , resulting in low power).
Another approach involves choosing the threshold based on parametric estimates of the limiting null distribution.
For example, Gretton et al.
(2006) suggested fitting to the Pearson family of densities based on the first four moments, while Gretton et al.
(2009) introduced a more computationally efficient method using a two-parameter Gamma approximation.
However, both of these methods are heuristic and do not have any consistency guarantees.
Gretton et al.
(2009) introduced a spectral method for approximating the null distribution using the eigendecomposition of the gram matrix.
They showed that the resulting distribution converges to the true null distribution as long as the square roots of the eigenvalues associated with the kernel operator are summable.
While this method is asymptotically consistent, the conditions imposed on the kernel are more stringent than that used in our work.
Furthermore, this method was shown to be consistent only in the fixed kernel (or low-dimensional) setting.
Hence, it is unknown whether the results carry over to the case of kernels varying with sample size or high-dimensional settings.
This method is also computationally nontrivial due to the need for a full eigendecomposition.
Keeping only the top few eigenvectors is another heuristic, but this introduces an extra hyperparameter and loses theoretical guarantees; as a result this method is rarely used in practice.
Our methods are simpler (no extra hyperparameter), faster (closed-form threshold), and more robust (type I error guarantees also hold for changing kernels, and in high-dimensional settings).
Changing the statistic: block-MMD and linear-MMD statistics.
An idea more closely related to ours is that changing the test statistic itself would help make it cheaper to compute and also yield a tractable limiting distribution.
One approach splits the observations into disjoint blocks, compute the kernel-MMD statistic on every block, and the final test statistic averages over all the blocks.
If the size of each block is fixed, we get a linear-time kernel-MMD (Gretton et al., 2012a,b).
The case of block sizes increasing with n, m was studied by Zaremba et al.
; Ramdas et al.
Depending on the block size ( b ), the computational complexity of block-MMD statistic varies from linear (constant b ) to quadratic ( b = Ω( n ) ).
Further, if b = o ( n ) , then one gets a Gaussian null distribution as well.
Our proposed statistic is fundamentally different from the block-MMD statistics, despite both being incomplete U-statistics (Lee, 1990).
In particular, the block-MMD statistics can be understood as building a block-diagonal approximation of the gram matrix.
On the other hand, our proposed cross-MMD statistic uses the off-diagonal blocks of the gram matrix, exactly the blocks that the block-MMD with two blocks ( b = n/ 2 ) excludes!
The reason that this is a sensible thing to do is nontrivial, and our test is motivated quite differently from the block-MMD.
In fact, when b = n/ 2 , the block-MMD does not have a Gaussian null, but the cross-MMD does.
For the block-MMD with b = o ( n ) , the Gaussian null distribution is achieved at the cost of suboptimal power, as observed empirically in Zaremba et al.
(2013), and
proved by Reddi et al.
(2015) for the case of linear-MMD and Ramdas et al.
(2015) for general block-MMD statistics.
In particular, their power is worse by factors scaling with n , which means that they are not minimax rate optimal.
In contrast, our test uses exactly half the elements of the gram matrix, and its power is about a √ 2 worse than the MMD test, independent of n and d , and we prove explicitly that it achieves the minimax rate.
Beyond the kernel-MMD.
The literature on two-sample testing is vast, and one can move even further away from the kernel-MMD (than the block-MMD) while retaining some of its intuition.
For example, Chwialkowski et al.
(2015) proposed a linear-time test statistic by computing the average squared-distance between the empirical kernel embeddings at J randomly drawn points.
Jitkrittum et al.
(2016) proposed a variant of this statistic in which the J points are selected to maximize a lower bound on the power.
In both cases, when the kernel k being used is analytic, in addition to being characteristic and integrable, the authors showed that the limiting distribution under null for this statistic is a combination of J independent χ 2 random variables.
However, similar to the spectral method of Gretton et al.
(2009), the high-dimensional behavior of these statistics are unknown.
In fact, some preliminary experiments with d ≈ n in Appendix E.3 suggest that these linear-time statistics have a different null distribution in this regime.
Further, the authors only proved consistency of these tests against fixed alternatives, but their power is not known to be minimax rate optimal.
In contrast, our statistic has the same limiting distribution in low- and high-dimensional settings even with changing kernels, and is provably minimax rate optimal for smooth alternatives, and it is a much more direct tweak of the usual MMD test.
One-sample (goodness-of-fit) testing.
Kim and Ramdas (2020) proposed and analyzed a similar studentized cross U-statistic in the simpler one-sample setting.
Our work has different motivations: our primary goal in this paper is to design a permutation-free kernel-MMD test that does not significantly sacrifice the power, while Kim and Ramdas (2020) pursued the related but different goal of dimension-agnostic inference, which means having the same limiting distribution in lowdimensional and high-dimensional settings.
Nevertheless, our results can be seen as an extension of their methods to two-sample testing.
Our proofs also build on their advances, but we require a more involved analysis since in their case the second distribution is known (making it a point null).
Deriving the cross-MMD test
In this section, we present our test statistic and investigate its limiting distribution.
First note that the squared kernel-MMD distance between two probability measures P and Q can be expressed as an inner product, namely ⟨ µ − ν, µ − ν ⟩ k .
The usual kernel-MMD statistic is obtained by plugging the empirical kernel embeddings into this inner product expression and removing the diagonal terms to make it unbiased.
Our proposal instead considers pairs of empirical estimates ( (cid:98) µ 1 , (cid:98) µ 2 ) and ( (cid:98) ν 1 , (cid:98) ν 2 ) constructed via sample splitting, and use the inner product between (cid:98) µ 1 − (cid:98) ν
1 and (cid:98) µ 2 − (cid:98) ν 2 instead.
This careful construction allows us to obtain a Gaussian limiting distribution after studentization.
To elaborate, recall from Section 1.1 that we partition X into X 1 and X 2 , and similarly Y into Y 1 and Y 2 .
We then compute empirical kernel embeddings based on each partition, yielding (cid:98) µ 1 := n − 1 1 (cid:80) n 1 i =1 k
( X i , · ) , (cid:98) µ 2 := n − 1 2 (cid:80) n 2 i ′ =
1 k ( X i ′ , · ) , (cid:98) ν 1 := m − 1 1 (cid:80) m 1 j =1 k ( Y j , · ) and (cid:98) ν 2 := m − 1 2 (cid:80) m 2 j ′ =
1 k ( Y j ′ , · ) .
Using these embeddings coupled with the kernel trick, the cross U-statistic (2) can be written as x (cid:92) MMD 2 = ⟨ (cid:98) µ 1 − (cid:98) ν 1 , (cid:98) µ 2 − (cid:98) ν 2 ⟩ k .
To further motivate our test statistic, denote U X,i := ⟨ k
( X i , · ) , (cid:98) µ 2 − (cid:98) ν 2 ⟩ k for i = 1 , . . . , n 1 and U Y,j := ⟨ k
( Y j , · ) , (cid:98) µ 2 − (cid:98) ν
2 ⟩ k for j = 1 , . . .
, m 1 .
Then the cross U-statistic can be viewed as the difference between two sample means: x (cid:92) MMD 2 = 1 n 1 (cid:80) n 1 i =1 U X,i − 1 m 1 (cid:80) m 1 j =1 U Y,j .
Since the summands are independent conditional on
X 2 and Y 2 , one may expect that x (cid:92) MMD 2 is approximately Gaussian after studentization.
Our results in Section 2.1 formalize this intuition under standard moment conditions, where it takes some care to remove the above conditioning, since we care about the unconditional distribution.
Let us further denote the sample means of U X,
i ’s and U Y,j ’s by ¯ U X
and ¯ U Y , respectively, and define
[Equation]
[Equation] Now we have completed the description of our studentized cross U-statistic ¯x (cid:92) MMD 2 = x (cid:92) MMD 2 / (cid:98) σ , and the resulting test Ψ in (3).
The asymptotic validity of the xMMD test is guaranteed by Theoremthat establishes the asymptotic normality of ¯x (cid:92) MMD 2 under the null.
Remark 2 (Computational Complexity) .
The overall cost of computing the statistic ¯x (cid:92) MMD 2 is
O (cid:0) ( n + m ) 2 (cid:1) , and in particular, both x (cid:92) MMD 2 and (cid:98) σ have quadratic complexity.
To see this, note that x (cid:92) MMD 2 can be expanded into ⟨ (cid:98) µ 1 , (cid:98) µ 2 ⟩ k + ⟨ (cid:98) ν 1 , (cid:98) ν 2 ⟩ k − ⟨ (cid:98) µ 1 , (cid:98) ν 2 ⟩ k − ⟨ (cid:98) ν 1 , (cid:98) µ 2 ⟩ k .
Each of these terms can be computed in O (cid:0) ( n + m ) 2 (cid:1) .
Similarly, each term in the summations defining (cid:98) σ 2 X and (cid:98) σ 2 Y also require O (cid:0) ( n + m ) 2 (cid:1) computation, implying that the (cid:98) σ also has O (( n + m ) 2 ) complexity.
Remark 3.
To simplify notation in what follows, we denote m as m n , where m n is some unknown nondecreasing sequence such that lim n →∞ m n = ∞ .
This still permits m, n to be separate quantities growing to infinity at potentially different rates, but it allows us to index the sequence of problems with the single index n (rather than m, n ).
We will use k n , d n , X n , P n and Q n to indicate that quantities could (but do not have to) change as n increases, and drop the subscript when they are fixed.
Furthermore, unless explicitly stated, we will focus on the balanced splitting scheme, i.e., n 1 =
⌊ n/ 2 ⌋ and m 1 =
⌊ m/ 2 ⌋ in what follows, because we currently see no apriori reason to split asymmetrically.
2.1 Gaussian limiting distribution under the null hypothesis
As shown in Figure 1, the empirical distribution of ¯x (cid:92) MMD 2 resembles a standard normal distribution for various choices of m , n and dimension d under the null.
In this section, we formally prove this statement.
Recalling the mean embedding µ from (1), define
[Equation]
Theorem 4.
Suppose that k and P do not change with n .
If 0 < E P [¯ k
( X, X ′ ) 4 ] < ∞ for
X, X ′ i.i.d.
∼ P , then ¯x (cid:92) MMD 2 d −→ N (0 , 1) .
We next present a more general result that implies Theorem 4.
Theorem 5.
Suppose P is fixed, but the kernel k n changes with n .
If
[Equation]
and
[Equation]
where ( λ l,n ) ∞ l =1 denote the eigenvalues of ¯ k introduced in (16) , then we have ¯x (cid:92) MMD 2 d −→ N (0 , 1) .
It is easy to check that condition (6) is trivially satisfied if the kernels { k n : n ≥ 1 } are uniformly bounded by some constant; prominent examples are the Gaussian or Laplace kernel with a sample size dependent bandwidth.
Thus, the above condition really exists to handle unbounded kernels and heavytailed distributions.
To motivate this requirement, we recall Bentkus and Götze (1996)
(see Factin Appendix A) who proved a studentized CLT for i.i.d.
random variables in a triangular array setup: W 1 ,n , W 2 ,n , . .
. , W n,n i.i.d.
∼ P n .
Define V n = √ n (cid:80) ni
=1 W i,n / (cid:113)(cid:80) i
( W i,n − ¯ W n ) 2 where ¯ W n = ( (cid:80) i W i,n ) /n .
They showed that a sufficient condition for the asymptotic normality of V n is that lim n →∞ E P n [ W 31 ,n ] / (cid:113) E P n [ W 21 ,n ] 3 n = 0 .
(This last condition is trivially true if P n does not change with n , meaning that the triangular array setup is irrelevant and W 1 ,n can be replaced by W 1 .)
Our requirement is slightly stronger: condition (6) with ¯ k n ( X 1 , X 2 ) replaced by W 1 ,n implies the previous condition of Bentkus and Götze (1996) (details in Remark 12 in Appendix A).
We need this stronger condition, because the terms in the definition of x (cid:92) MMD 2 are not i.i.d.
(indeed, not even independent), and thus we cannot directly apply the result of Bentkus and Götze (1996).
Instead, we take a different route by first conditioning on the second half of data ( X 2 , Y 2 ) , then showing the conditional asymptotic normality of the standardized x (cid:92) MMD 2 (i.e., divided by conditional standard deviation instead of empirical), and finally showing that the ratio of conditional and empirical standard deviations converge in probability to 1 (see Appendix B).
Finally, we note that the result of Theorem 5 can be further generalized in several ways: (i) instead of a fixed P and changing k n , we can consider a sequence of pairs { ( P n , k n ) : n ≥ 1 } changing with n , (ii) we can let P n ∈ P (0) n , for a class of distributions changing with n , and obtain the Gaussian limit uniformly over all elements of P (0) n , and finally, (iii) the moment requirements on ¯ k n stated in condition in (6) can also be slightly weakened.
We state and prove this significantly more general version of Theorem 5 in Appendix B. Remark 6.
In the statement of the two theorems of this section, the splits ( X 1 , Y 1 ) and ( X 2 , Y 2 ) are assumed to be drawn i.i.d.
from the same distribution P .
However, a closer look at the proof of Theorem 5 indicates that the conclusions of the above two theorems hold even when the two splits are independent and drawn i.i.d.
from possibly different distributions; that is ( X 1 , Y 1 ) and ( X 2 , Y 2 ) are independent of each other and drawn i.i.d. from distributions P 1 and P 2 respectively, with P 1 ̸ = P 2 . In particular, under this more general condition, the asymptotic normality of ¯x (cid:92) MMDstill holds, and the resulting test Ψ still controls the type-1 error at the desired level. This may be useful for two-sample testing in settings where the entire set of data is not i.i.d., but two different parts of the data were collected in two different situations.
The usual MMD can also handle such scenarios by using a subset of permutations that do not exchange the data across the two situations.
2.2 Consistency against fixed and local alternatives
Here, we show that the xMMD test Ψ introduced in (3) is consistent against a fixed alternative and also has minimax rate-optimal power against smooth local alternatives separated in L 2 norm.
We first show that analogous to Theorem 4, xMMD is consistent against fixed alternatives.
Theorem 7.
Suppose P, Q, k do not change with n , and P ̸ = Q .
If k is a characteristic kernel satisfying 0 < E P [¯ k ( X 1 , X 2 ) 4 ] < ∞ , and 0 < E Q [¯ k ( Y 1 , Y 2 ) 4 ] < ∞ , then the xMMD test is consistent, meaning it has asymptotic power 1 .
The moment conditions required above are mild, and are satisfied trivially, for instance, by bounded kernels such as the Gaussian kernel.
The “characteristic” condition is also needed for the consistency of the usual MMD test (Gretton et al., 2012a), and is also satisfied by the Gaussian kernel.
Recalling Remark 1, we next consider the more challenging setting where d n , k n can change with n , and ( P n , Q n ) can vary within a class P (1) n ⊂ P ( X n ) × P ( X n ) that can also change with n .
We present a sufficient condition under which the xMMD test Ψ is consistent uniformly over P (1) n .
Define γ n := MMD( P n , Q n ) , which is assumed nonzero for each n but could approach zero in the limit.
Theorem 8.
Let { δ n : n ≥ 2 } denote any positive sequence converging to zero.
If
[Equation]
V denotes variance,
[Equation]
[Equation]
then
[Equation]
, meaning the xMMD test is consistent.
Note that while any sequence { δ n } converging to zero suffices for the general statement above, the condition (7) is easiest to satisfy for slowly decaying δ n , such as δ n = 1 / log log n for instance.
The sufficient conditions for consistency of Ψ stated in terms of (cid:98) σ and x (cid:92) MMD 2 in (7) can also be translated into equivalent conditions on the kernel function k n , similar to (6), and we present the details in Appendix C. Importantly, if P n , Q n , d n are fixed and k n is bounded, then both E [ (cid:98) σ 2 ] and
V (x (cid:92) MMD 2 ) are O
(1 /n ) , and γ n is a constant, so the condition is trivially satisfied, and in fact the above condition is even weaker than the fourth-moment condition of the previous theorem.
2.3 Minimax rate optimality against smooth local alternatives
We now apply the general result of Theorem 8 to the case where the distributions P n and Q n admit
Lebesgue densities p n and q n that lie in the order β Sobolev ball for some β > 0 , defined as
W β, 2 ( M ) :
= { f : X → R | f is a.s. continuous, and (cid:82) (1 + ω 2 ) β/ 2 ∥F ( f )( ω )
∥ 2 dw < M < ∞} .
Formally, we define the null and alternative class of distributions as follows:
[Equation]
density
[Equation]
and
[Equation]
densities
[Equation]
[Equation]
p
[Equation]
q ∥ L
[Equation]
≥ ∆ n
[Equation]
, for some sequence ∆ n decaying to zero.
In particular, we assume that under H 0 , P n = Q n and
P n ∈ P (0) n , while under H 1 , we assume that ( P n , Q n ) ∈ P (1) n .
Our next result shows that for suitably chosen scale parameter, the xMMD test Ψ with the Gaussian kernel is minimax rate-optimal for the above class of local alternatives.
For simplicity, we state this result with n = m , noting that the result easily extends to the case when there exist constants 0 < c ≤ C , such that c ≤ n/m ≤ C .
Theorem 9.
Consider the case when n = m , and let { ∆ n : n ≥ 1 } be a sequence such that lim n →∞ ∆ n n 2 β/ ( d +4 β )
= ∞ .
On applying the xMMD test Ψ with the Gaussian kernel k s n ( x, y ) =
exp( − s n ∥ x − y ∥ 22 ) , if we choose the scale as s n ≍ n 4 / ( d +4 β ) , then we have
[Equation]
The proof of this statement is in Appendix C, and it follows by verifying that the conditions required by Theorem 8 are satisfied for the above choices of ∆ n and s n .
Remark 10.
Li and Yuan (2019, Theorem 5 (ii)) showed a converse of the above statement: if lim n →∞ ∆ n n 2 β/ ( d +4 β ) < ∞ , then there exists an α ∈ (0 , 1) such that any asymptotically level α test (cid:101)
Ψ must have lim n →∞ inf ( P,Q ) ∈P (∆ n ) E P,Q [ (cid:101) Ψ
( X , Y )]
< 1 .
Hence, the sequence of { ∆ n : n ≥ 1 } used in Theorem 9 represents the smallest L 2 -deviations that can be detected by any test, and (8) shows that our xMMD test Ψ can detect such changes, establishing its minimax rate-optimality.
Experiments
We now present experimental validation of the theoretical claims of the previous section.
In particular, our experiments demonstrate that (i) the limiting null distribution of ¯x (cid:92) MMD 2 is N (0 , 1) under a wide range of choices of dimension d , sample sizes n, m and the kernel k , and (ii) the power of our xMMD test is competitive with the kernel-MMD permutation test.
We now describe the experiments in more detail.
Additional experimental results are reported in Appendix E.
Limiting null distribution of ¯x (cid:92) MMD 2 .
We showed in Theorem 15 that the statistic ¯x (cid:92) MMD 2 has a limiting normal distribution under some mild assumptions.
We empirically test this result when X and Y are drawn from N ( 0 ,
I d ) with 0 denoting the all-zeros vector in R d , and in particular, study the effects of (i) dimension: d = 10 versus d = 500 , (ii) skewness of the samples: n/m = 1 versus n/m = 0 .
1 , and (iii) choice of kernel: Gaussian versus Quadratic, both with data-dependent scale parameters using median heuristic.
As shown in the first row of Figure 2, the distribution of ¯x (cid:92) MMD 2 is robust to all these effects, and is close to N (0 , 1) in all cases.
In contrast, the distribution of the kernel-MMD statistic scaled by its empirical standard deviation (obtained using 200 bootstrap samples) in the bottom row of Figureshows strong changes with these parameters.
We present additional figures and details of the implementation in Appendix E.
Evaluation of the power of Ψ .
For d ≥ 1 and j ≤ d , let a ϵ,j denote the element of R d with first j coordinates equal to ϵ , and others equal to 0 .
We consider the two-sample testing problem with P = N ( 0 , I d ) Q = N ( a ϵ,j , I d ) for different choices of ϵ and d and j .
We compare the performance of our proposed test Ψ with the kernel-MMD permutation test, implemented with
B = 200 permutations, and plot the power-curves (over 400 trials) in Figure 3.
We also propose a heuristic for predicting the power of the permutation test (denoted by ρ perm ) using the power of Ψ (denoted by ρ Ψ ) as follows (with Φ denoting the standard normal cdf, and z α its α -quantile): [Equation]
[Equation]
This heuristic is motivated by the power expressions derived by Kim and Ramdas (2020) for the problems of one-sample Gaussian mean and covariance testing (we discuss this further in Appendix E).
The term √ 2 in the above expression quantifies the price to pay for sample-splitting.
As shown in Figure 3, this heuristic gives us an accurate estimate of the power of the kernel-MMD permutation test, without incurring the computational burden.
We now use ROC curves to compare the tradeoff between type-I and type-II errors for the usual MMD, linear and block-MMDs with our ¯x (cid:92) MMD 2 .
We use the same distributions P = N ( 0 , I d ) and Q =
P r ob a b ilit y d e n s it y xMMD
( n/m = 0 . 1) d = 10 d = N (0 , 1) xMMD ( n/m = 1) xMMD ( n/m = 10) xMMD ( n/m = 1) P r ob a b ilit y d e n s it y
MMD ( n/m = 0 . 1)
MMD ( n/m = 1)
MMD ( n/m = 10)
MMD ( n/m = 1)
[Figure]
i.i.d.
and Y is drawn from N ( a ϵ,j , I d ) where a ϵ,j is obtained by perturbing the first j ≤ d coordinates of 0 by ϵ , the kernel used is the
Gaussian kernel with scale parameter chosen via the median heuristic.
The dashed curve shows the predicted power of the kernel-MMD permutation test using the heuristic defined in (9).
N ( a ϵ,j , I d ) as before, and plot results for ( d, j, ϵ ) ∈ { (10 , 5 , 0 . 2) , (100 , 20 , 0 . 15) , (500 , 100 , 0 . 1) } in Figure 4.
Due to sample splitting, the tradeoff achieved by our proposed statistic is slightly worse than that of (cid:92) MMD 2 , but significantly better than other computationally efficient variants of kernel-MMD statistics.
More details about the implementation, and additional figures are presented in Appendix E.
Conclusion and future work
We proposed a variant of the kernel-MMD statistic, called cross-MMD, based on the ideas of samplesplitting and studentization, and showed that it has a standard normal limiting null distribution.
Using this key result, we introduced a permutation-free (and hence computationally efficient) MMD test False Positive Rate T r u e P o s iti v e R a t e ( d, ϵ, j ) =
MMDxMMD
B - MMD ( n 1 / 2 )
B - MMD ( n 1 / 3 ) L - MMD 0 . False Positive Rate ( d, ϵ, j ) = (100 , 0 . 15 , 10) 0 .  0 . False Positive Rate ( d, ϵ, j ) = (500 , 0 . 15 , 50) for the two-sample problem. Experiments indicate that the power achieved by our test is within a √ 2 -factor of the power of the kernel-MMD permutation test (that requires recomputing the statistic hundreds of times).
In other words, our results achieve the following favorable tradeoff: we get a significant reduction in computation at the price of a small reduction in power .
Sejdinovic et al.
(2013) establish in some generality that distance-based two-sample tests (like the energy distance (Székely and Rizzo, 2013)) can be viewed as kernel-MMD tests with a particular choice of kernel k .
Hence our results are broadly applicable to distance-based statistics as well.
Since two-sample testing and independence testing can be reduced to each other, it is an interesting direction for future work to see if the ideas developed in our paper can be used for designing permutation-free versions of kernel-based independence tests like HSIC (Gretton et al., 2007) or distance covariance (Székely and Rizzo, 2009; Lyons, 2013).
Our techniques seem to rely on the specific structure of two-sample U-statistics of degree 2 .
Extending these to more general U-statistics of higher degrees is another important question for future work.
A final question is to figure out whether it is possible to achieve minimax optimal power using a sub-quadratic time test statistic.
One potential approach would be to work with a kernel approximated by random Fourier features (Rahimi and Recht, 2007; Zhao and Meng, 2015).
Depending on the number of random features, our test statistic can be computed in sub-quadratic time and it would be interesting to see whether the resulting test can still be minimax optimal in power.
We leave this important question for future work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In cases where computation cost is not an issue, the power achieved by our test will be slightly smaller (approximately by a factor of √ 2 ) as compared to the kernel-MMD permutation test.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs are in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will include the code with the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We include all the details of the experiments in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The experiments were run on a workstation, whose specifications are provided in Appendix E 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]