A GENERALIZED PROBABILITY KERNEL ON DISCRETE DISTRIBUTIONS AND ITS APPLICATION IN TWOSAMPLE TEST
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a generalized probability kernel(GPK) on discrete distributions with ﬁnite support.
This probability kernel, deﬁned as kernel between distributions instead of samples, generalizes the existing discrepancy statistics such as maximum mean discrepancy(MMD) as well as probability product kernels, and extends to more general cases.
For both existing and newly proposed statistics, we estimate them through empirical frequency and illustrate the strategy to analyze the resulting bias and convergence bounds.
We further propose power-MMD, a natural extension of MMD in the framework of GPK, illustrating its usage for the task of two-sample test.
Our work connects the ﬁelds of discrete distribution-property estimation and kernel-based hypothesis test, which might shed light on more new possibilities.
I NTRODUCTION
We focus on the two-sample problem, which is given two i.i.d samples { x 1 , x 2 , ...x n } , { y 1 , y 2 , ..., y n } , could we infer the discrepancy between underlying distributions they are drawn from.
For such a problem, the option of hypothesis test(two-sample test) is most popular, and a variety of statistics in estimating the discrepancy is proposed.
In recent years,
RKHS based method such as maximum mean discrepancy(MMD) has gained a lot of attention.
(Gretton et al., 2012) has shown that in a universal-RKHS F , MMD( F , p , q ) =
if and only if p = q , thus could be used for the two-sample hypothesis test.
(Gretton et al., 2012) further provides unbiased estimator of MMD with fast asymptotic convergence rate, illustrating its advantages.
On the other hand, estimating distribution properties with plugin(empirical) estimators on discrete setting is an active research area in recent years, where people focus on problem settings with large support size but not so large sample size.
The Bernstein polynomial technique is introduced to analyze the bias of the plugin estimators in (Yi & Alon, 2020), which provides remarkable progress on bias-reduction methods of the plugin estimators.
It is thus interesting to ask if the plugin estimators could motivate new results for the RKHS-based two-sample test.
Another interesting topic is about the probability kernel, deﬁned as kernel function over probabilities, instead of over samples.
As is easily seen, any discrepancy measure of distribution p and q could potentially be valid probability kernels, not so much work focuses on this.
While (Jebara et al., 2004) introduced the so called probability product kernels which generalize a variety of discrepancy measures, its properties remain further study.
Motivated by above observations, our work focuses on a specialized probability kernel function which is a direct generalization of sample-based RKHS methods such as MMD.
We focus on using plugin-estimator as the default estimator of the kernel function we deﬁned, and illustrate that with the help of Bernstein polynomial techniques, we could analyze the bias and convergence bounds of these plugin-estimators.
Our work thus connects the ﬁelds of discrete distribution-property estimation and kernel-based hypothesis test, which brings interesting possibilities.
N OTATION
We use bold symbol p , q ∈ R k to represent a probability function over a discrete support with support size k , and p i , q i represents the i th entry of p and q .
We use { v 1 , v 2 , ..., v k } , v i ∈ R d to represent the support of p , q .
[ k ] := { 1 , 2 , 3 ..., k } represents the set of indices of elements in { v 1 , v 2 , ..., v k } .
We use φ ◦ ( p , q ) to denote an element-wise function from R k × R k to R k , where ( φ ◦ ( p , q ))
i =
φ ◦ ( p i , q i ) and φ ◦ p to denote an element-wise function from R k to R k , where ( φ ◦ p ) i =
φ ◦ p i .
With a slight abuse of notation, we denote p ρ , p − q as element-wise function deﬁned above.
We use kernel( p , q ) to denote kernel function which maps from R k × R k to real value R .
And kernel ( x, y ) , x, y ∈ R d represents a kernel function from R d × R d to real value R .
We use K to denote the gram matrix generated from kernel ( x, y ) on ﬁnite support { v 1 , v 2 , ..., v k } , where K ij = kernel ( v i , v j ) .
We use { x 1 , x 2 , ..., x n } ∼ p and { y 1 , y 2 , ..., y n } ∼ q to denote the samples from distribution p and q , where n is the sample size.
We use (cid:107) x, y (cid:107) 22 to denote L2 norm between two vector x, y ∈ R d
G ENERALIZED PROBABILITY KERNEL
Probability kernel function, deﬁned as kernel function between distributions instead of samples, is a natural extension of the idea of kernel function in sample space.
Deﬁnition 1.
Given distribution p and q belongs to a family of discrete distribution with the same ﬁnite support { v 1 , v 2 , ..., v k } , v i ∈ R d , where k is the support size, we deﬁne the probability kernel function as PK( p , q ) , which is a kernel function maps from R k × R k to real value R .
Many discrepancy measures, such as MMD, can serve as probability kernel functions, but people usually don’t use the term of probability kernel function when describing them.
The reason is that for most of the time, we only consider a limited number of distributions, and do not need or have the resources to navigate through all the distributions within the family.
For example, when looking into the two-sample problem, we usually assume two samples { x 1 , x 2 , ..., x n } ∈ R d and { y 1 , y 2 , ..., y n } ∈ R d are i.i.d drawn from two distributions p and q , and use the discrepancy measure MMD[ F , p , q ] to determine if p and q are indistinguishable in the RKHS F .
We do not consider all other distributions in F that is irrelevant to our samples!
So far the idea of kernel function between distributions is in practice not so much useful, however, here in this paper, we propose, when considering the plugin-estimator of many of the existing discrepancy measures, it is beneﬁcial to view them as probability kernel functions.
3. DEFINATION OF GENERALIZED PROBABILITY KERNEL
Deﬁnition 2 (Generalized probability kernel) .
Given the family S of discrete distribution on support { v 1 , v 2 , .., v k } where v i ∈ R d .
Let F be a unit ball in a universal-RKHS H with associated continuous kernel RK ( x, y ) , where for any x ∈ R d and y ∈ R d , RK ( x, y ) maps from R d × R d to R .
We denote gram matrix K ij =
RK ( v i , v j ) .
The generalized probability kernel function on distribution p , q ∈ S is GPK F ,φ ( p , q ) = φ ◦ ( p , q ) Kφ ◦ ( q , p ) T =
(cid:80)
i ∈
[ k ] (cid:80)
[Equation]
where φ ◦ ( p , q ) is an element-wise mapping function on discrete distribution p , q ∈ S , which maps from R k × R k to R k ,
Obviously, under this deﬁnition, the GPK is a symmetric probability kernel function where
[Equation]
Mapping function φ represent a great amount of possibilities.
For most cases, we need to narrow down the region and equipped it with some convenient properties so that the GPK measure could be useful.
One example is for the measurement of discrepancy, where we want GPK F ,φ ( p , q )
=
if and only if p = q .
Deﬁnition 3 (discrepancy probability kernel) .
Let S be a family of discrete distribution p ∈ S on support { v 1 , v 2 , ..., v k } .
A discrepancy probability kernel is a kernel function PK ( p , q ) that
PK( p , q ) =
if and only if p = q
Theorem 1.
GPK F ,φ ( p , q ) with the mapping function φ that satisﬁes:
symmetric or antisymmetric with respect to p and q : φ ◦ ( p , q ) =
φ ◦ ( q , p ) or φ ◦ ( p , q ) = − φ ◦ ( q , p
(cid:107) [Equation] ◦ ( p , q ) (cid:107)
= (cid:107) φ ◦ ( q , p ) (cid:107)
=
if and only if p = q , where (cid:107) · (cid:107) 2 represents L2 norm.
is a discrepancy probability kernel.
Proof.
[Equation]
K is a semipositive deﬁnite matrix, thus by deﬁnition of positive deﬁnite matrix, v K v T ≥ 0 , where equality holds if and only if v = 0 , and since v = φ ◦ ( p , q ) , this condition further means φ ◦ ( p , q ) = 0 , which holds if and only if p = q .
Another example is the polynomial GPK, which is our main focus of this paper.
Such a subclass of
GPK is interesting since we can build unbiased estimators of it using techniques of Bernstein polynomial in (Qian et al., 2011).
As we will show in section 5.
, we also have analyzable convergence bounds for the resulting unbiased estimators, illustrating its potential usage for applications such as two-sample test.
Deﬁnition 4 (polynomial GPK) .
The polynomial GPK is the subset of GPK that equipped with the mapping function φ that is polynomial in p and q : φ ◦ ( p , q )
= (cid:80) ol =0 (cid:80) os =
α l,s p l q s where o ∈ Z is the degree of the polynomial, and a l,s ∈ R is the coefﬁcient
Below we give some examples of polynomial GPK, which include MMD proposed in (Gretton et al., 2012), and the newly proposed power-MMD in this paper, which is a natural extension of MMD, from the view point of probability kernels.
3.1. EXAMPLE 1: MMD AS MEMBER OF POLYNOMIAL GPK
Given discrete distribution p , q with support { v 1 , v 2 , ..., v k } , we can rewrite MMD with distribution probability function p i , q i :
[Equation]
Where φ l ◦ ( p , q ) = p − q , H is the RKHS deﬁned in MMD literature, and f is the function that maps v i to H .
GPK F ,φ l ( p , q ) is a special case of polynomial GPK where α 1 ,
= 1 , α 0 , 1 = − 1 , and all other coefﬁcients are 0 .
3.1. EXAMPLE 2: PRODUCT GPK AS MEMBERS OF POLYNOMIAL GPK
Deﬁnition 5 (product GPK) .
Let p and q be probability distributions on support { v 1 , v 2 , ..., v k } , and l, s ∈
Z be nonnegative integer.
The product GPK is a subset of polynomial GPK where α l,s = 1 , and all other coefﬁcients are 0 .
the corresponding mapping function is: φ ( p , q ) =
p l q s
The probability product kernel as in (Jebara et al., 2004) is a special case of product GPK where l
= s , and K is a identity matrix.
3.1. EXAMPLE 3: POWER -MMD AS MEMBERS OF POLYNOMIAL GPK
Another interesting subset of polynomial GPK is the one extends MMD case into a power form and we denote it as power-MMD:
Deﬁnition 6 (power-MMD) .
Let p and q be probability distributions on support { v 1 , v 2 , ..., v k } and ρ ∈ Z be a positive integer.
then the power-MMD is a subset of polynomial GPK where α ρ, 0 = 1 , α 0 ,ρ = − 1 , and all other coefﬁcients are 0 .
the corresponding mapping function is: φ ( p , q ) =
p ρ − q ρ
Apparently, MMD is a special case of power-MMD where ρ = 1 , and power-MMD satisﬁes the requirement in theorem 1, thus has the potential usage of discrepancy measure.
In section 5., we will show that power-MMD has unbiased estimator with analyzable convergence bounds thus could be used for two-sample test.
3. DISCUSSION OF GPK IN DISCRETE SETTING
As one may easily notice, the deﬁnition of GPK includes a gram matrix generated by the kernel function RK ( v i , v j ) which measures the discrepancy between v i , v j ∈ { v 1 , v 2 , .., v k } .
While considering the cases of categorical distribution, values of discrete variables does not relate to any notion of distance, this raises the question: how the introduced gram matrix will be beneﬁcial in any cases?
The answer is twofold: 1.
Many natural processes produce discrete distributions where there possibly exists a similarity measure in values which imply the similarity in frequencies of occurrence(probability values).
For example, in the ﬁeld of natural language process(NLP), one may treat words as atomic units with no notion of similarity between words, as these are represented as indices in a vocabulary.
However, given large number of training samples, similarity measure between words could be made possible using techniques such as words2vec(Mikolov et al., 2013).
Such techniques generally result in better performance and have become the important preprocessing techniques for NLP tasks(Goodfellow et al., 2016).
As there are cases where the values of discrete variables are totally irrelevant, or people may use kernel function RK ( v i , v j ) which doesn’t correctly imply the similarity in probability values, the GPK framework may still capture the similarity between distributions.
One example is the case of MMD, which is, as we discussed above, an element of GPK family.
As proved in (Gretton et al., 2012), MMD is a distribution free measurement between two samples, which means no matter what kind of p , q and kernel ( x, y ) , we have, the
MMD 2 F ( p , q ) measure will be 0 if and only if p = q .
However, the bad choice of kernel function does have a negative effect on convergence bounds of the empirical estimator proposed in (Gretton et al., 2012), and will inﬂuence the results of two-sample test.
For this reason, we mainly focus on dataset with known relativity measures in our experiment section.
PLUGIN - ESTIMATOR FOR GPK
So far we have deﬁned the GPK and discussed some subsets of GPK with potential usage of two-sample test.
Next we discuss how to build an estimator, given a member of GPK.
In this section, we propose the plugin-estimator, which based on the count of occurrence of each value v i ∈ { v 1 , v 2 , ..., v k } in samples { x 1 , x 2 , ..., x n } ∈ p or q .
We illustrate that by doing so, the techniques of Bernstein polynomial in (Qian et al., 2011) could be used to help building unbiased estimators for any members of polynomial GPK.
Furthermore, we provide analyzable convergence bounds of these estimators.
We begin with the deﬁnition of plugin-estimators:
Deﬁnition 7.
Suppose we have i.i.d samples of distribution p as X n 1 :=
{ x 1 , x 2 , ..., x n 1 } ∼ p and X n 2 := { x n 1 +1 , x n 1 +2 , ..., x n 1 + n 2 } ∼ p .
And also the i.i.d samples of distribution q as
Y m 1 := { y 1 , y 2 , ..., y m 1 } ∼ q and Y m 2 := { y m 1 +1 , y m 1 +2 , ..., y m 1 + m 2 } ∼ q .
Let N ( n 1 ) i denotes the number of occurrence of value v i ∈ { v 1 , v 2 , ..., v k } in sample X n 1 , and
S i,n 1 :=
( N ( n 1 ) i , n 1 ) denotes the collection of N ( n 1 ) i and n 1 .
The same follows for X n 2 , Y m 1 and
Y m
We deﬁne the plugin-estimator of GPK F ,φ ( p , q ) as
[Equation]
where f φ is a function related to function φ , and K is the gram matrix brought by F .
Here our setting is different from the unbiased estimator MMD 2 u of (Gretton et al., 2012), where in their setting X n 1 , X n 2 represent the same sample from p and so do for Y m 1 , Y m 2 from q .
Instead, we are using the same setting as the linear time statistic MMD 2 l proposed in (Gretton et al., 2012).
Another way of viewing this is that for our setting, given two samples { x 1 , x 2 , ..., x n } , { y 1 , y 2 , ..., y n } from p and q , we depart each sample of x and y into two parts, yielding 4 different samples with size n 1 , n 2 , m 1 , m 2 , and then calculate the empirical frequencies for plugin-estimator deﬁned above.
4. POLYNOMIAL GPK WITH UNBIASED PLUGIN - ESTIMATORS
One of our main contributions of this paper is the proposal that we can always ﬁnd an unbiased plugin-estimator for any members in polynomial GPK family.
The basic idea is that we can analyze the expectation of plugin-estimators through Bernstein polynomial, and use the existing results of (Qian et al., 2011) to build the unbiased plugin estimators.
Theorem 2.
Denote
[Equation]
Then any member of polynomial GPK[ F , φ, p , q ] equipped with polynomial mapping function φ ( p , q ) =
(cid:80) ol =0 (cid:80) os =
α l,s p l q s of degree o ∈ Z , has an unbiased plugin-estimator with mapping function f φ to be:
[Equation]
Proof.
The basic idea is directly using the result of Bernstein polynomial in (Qian et al., 2011) to build unbiased estimators.
We put our formal proof in appendix
For notation simplicity, we deﬁne the plugin-estimator discussed above to be the default-pluginestimator for polynomial GPK:
Deﬁnition 8 (default-plugin-estimator for polynomial GPK) .
The plugin-estimator deﬁned in theorem 2 is the default-plugin-estimator for polynomial GPK.
This plugin-estimator, according to theorem 2, is an unbiased estimator
4. DEVIATION BOUND OF PLUGIN - ESTIMATORS
Another topic about plugin-estimator is its deviation bound.
We directly use the McDiamid’s inequality to derive the default-plugin-estimator for polynomial GPK:
Theorem 3.
The default-plugin-estimator of GPK[ F , φ, p , q ] equipped with polynomial mapping function φ ( p , q )
= (cid:80) ol =0 (cid:80) os =
α l,s p l q s of degree o ∈ Z has the convergence bound:
[Equation]
where
[Equation]
K max is the largest value of entries in K
Proof.
The basic idea is to use the McDiamid’s inequality, and we put our formal proof into the appendix.
EXAMPLE : POWER -MMD AS A NATURAL EXTENSION TO MMD FROM GPK VIEWPOINT
In this section, we mainly discuss power-MMD as deﬁned in 3.1.3.
We analyze the bias and convergence bound of its plugin-estimators using the techniques we introduced so far, illustrating that such a natural extension to MMD from GPK viewpoint could be beneﬁcial for two-sample test.
5. PLUGIN - ESTIMATORS OF POWER -MMD
As we already discussed in section 4.1.3, power-MMD is a subset of polynomial GPK.
According to theorem 2, any member GPK F ,φ ρ ( p , q ) in power-MMD has a default-plugin-estimator with the mapping function f φ ( S i,n 1 , S i,m 1 ) =
g ρ ( N i , n 1 ) − g ρ
( M i , m 1 )
Remark 3.1.
When ρ = 1 , the power-MMD return to the original MMD case.
Remarkably, the default-plugin-estimator of this case is equivalent to the linear time statistic MMD 2 l proposed in (Gretton et al., 2012): GPK E [ F , φ l , X, Y ] =
MMD 2 l [ F , X, Y ]
For details of the derivation, see appendix
5. DEVIATION BOUND OF PLUGIN - ESTIMATORS OF POWER -MMD
Corollary 3.1.
Denote τ n = sup i ∈
[ k ]
[Equation]
The default-plugin-estimator of power-MMD GPK F ,φ ρ ( p , q ) has uniform convergence bound deﬁned in theorem 3.
with τ (1) n,m = τ n and τ (2) n,m = τ m
Corollary 3.2.
Consider the case where n 1 = n 2 = m 1 = m 2 = n The default-plugin-estimator of power-MMD GPK F ,φ ρ ( p , q ) has uniform convergence bound: Pr [Equation] | GPK
[Equation]
[ F ,
[Equation]
ρ , X, Y ]
[Equation]
[GPK
[Equation]
[Equation]
, X
[Equation]
[Equation]
Proof.
The ﬁrst inequality above comes from:
[Equation]
where sup i ∈ [ k ] (cid:16) ρN i g ρ
( N i , n ) (cid:17) =
ρn only stands for extreme case such that there exist N i = n , i.e. all the samples belongs to the same value v i ∈ { v 1 , v 2 , ..., v k } .
And the second inequality above comes from:
[Equation]
For details of this derivation, see appendix.
Remark 3.2.
recall in (Gretton et al., 2012), the deviation bound for linear time estimator
MMD 2 l [ F , X, Y ] is
[Equation]
Interestingly, this bound is the same as the case ρ = 1 in Corollary 3.2.
Note that according to section 6.1, the default-plugin-estimator of power-MMD with ρ = 1 is actually in equivalent to
MMD 2 l case in (Gretton et al., 2012).
Our bound generalize the bound in (Gretton et al., 2012) and provide a tighter version.
Note that the bounds for special case of ρ = 1 has simpler derivation, and the reader may refer to appendix for more details.
5. TWO - SAMPLE TEST USING POWER -MMD
Corollary 3.3.
A hypothesis test of level α for the null hypothesis p =
q has the acceptance region
[Equation]
Where Z is deﬁned in Corollary 3.
We are ready to use power-MMD for two-sample test.
However, before we really get started, we still need to answer two questions: 1.
when applying power-MMD in practice, is the proposed statistics numerical stable?
will the performance of two-sample test gets better when ρ gets larger?
5. NUMERICAL STABILITY
Since g ρ ( N i , n ) ≤ (cid:0) N i n (cid:1) ρ , the term will exponentially decrease with the increase of ρ . This effect will cause numerical problem when N i (cid:28) n and ρ is large. One solution is to ﬁnd an upper-bound of (cid:12)(cid:12)(cid:12)
GPK E [ F ,φ ρ ,X,Y ] √ Z (cid:12)(cid:12)(cid:12) which is numerical stable.
Corollary 3.4.
Consider the simplest case where n 1 = n 2 = m 1 = m 2 = n .
Deﬁne
[Equation]
N
[Equation]
} to be the set of all counts of occurrence in the four samples X n 1 , X n 2 , X m 1 , X m 2 .
Denote S N = sup N i ∈ C N ( N i ) to be the maximium value in the set C N We have:
[Equation]
where
GPK (cid:48)
[Equation]
(cid:88) i
[Equation]
[Equation]
and Φ (cid:48) = (cid:88) i ∈ [ k ] (cid:12)(cid:12)(cid:12) g ρ
( N ( n ) i , S N ) − g ρ ( N ( m ) i , S N ) (cid:12)(cid:12)(cid:12)
For cases when N i are not far less from S N , GPK (cid:48)
E will be much more numerical stable than
GPK E [ F , φ ρ , X, Y ] . 5. THE PERFORMANCE OF TWO - SAMPLE TEST WHEN ρ GET LARGER We need to analyze the case when p (cid:54) = q , if (cid:12)(cid:12)(cid:12)
GPK E
[ F ,φ ρ ,X,Y ] √ Z (cid:12)(cid:12)(cid:12) increase with the increase of ρ .
Unfortunately, there is no clear answer to this.
A A PPENDIX A. B ERNSTEIN POLYNOMIAL
Drawing i.i.d.
samples Y m from any distribution p the expected value of the empirical estimator for a distribution property is
[Equation]
Note that for any function f, m ∈ N , and x ∈ [0 , 1] , the degree-m Bernstein polynomial of f is
[Equation]
Therefore, we can express the expectation of the empirical property estimator as
[Equation]
A. PROOF OF THEOREM 2.
Proof.
Recall the deﬁnition of polynomial GPK:
[Equation] k [Equation]
[Equation]
recall in (Qian et al., 2011)
[Equation]
where g j ( k, n ) is deﬁned in the beginning of the theorem
GPK( F , φ, p , q ) = (cid:88) i ∈ [ k ] [Equation] 
[Equation]
, m 1 )
[Equation]
N
[Equation]
n
[Equation]
, n [Equation] ) g r ( N [Equation]
m
[Equation]
j , m 2 ) 
 =
E [GPK E ( F , φ, X, Y )]
A. PROOF OF THEOREM 3.
Lemma 4.
Let S N i ,n 1 :=
( N ( n 1 ) i , n 1 ) denotes the collection of N ( n 1 ) i and n 1 .
The same follows for
X n 2 , Y m 1 and Y m 2 .
Also for notation simplicity, let S i,n 1 :=
S N i ,n For the plugin-estimator
[Equation]
with mapping function f φ ( S N i ,n 1 , S N i ,m 1 ) having the following properties:
• f φ ( S N i ,n 1 , S N i ,m 1 ) is a monotonic function related to N ( n 1 ) i and N ( m 1 ) i : ∀ N i > N (cid:48)
i or ∀ N i < N (cid:48)
i : f φ ( S N i ,n 1 , S N i ,m 1 ) >
f φ ( S N (cid:48) i ,n 1 , S N i ,m 1 )
The same follows for N ( m 1 ) i • | f φ ( S N i ± 1 ,n 1 , S N i ,m 1 ) − f φ ( S N i ,n 1 , S N i ,m 1 ) | ≤ τ nand | f φ ( S N i ,n 1 , S N i ± 1 ,m 1 ) − f φ ( S N i ,n 1 , S N i ,m 1 ) | ≤ τ m
[Equation]
where τ n 1 is a constant related to sample size n 1 , τ m 1 is a constant related to sample size m 1 , the same follows for τ n 2 and τ m
We have:
[Equation]
where
[Equation]
K max is the largest value of entries in K
Proof.
recall McDiamid’s inequality
Theorem 5.
Let Y 1 , . . .
, Y m be independent random variables taking values in ranges R 1 , . . .
, R m and let F : R 1 × . . .
×
R m → C with the property that if one freezes all but the w th coordinate of F ( y 1 , . . .
, y m ) for some 1 ≤ w ≤ m, then F only ﬂuctuates by most c w > 0 , thus | F ( y 1 , . . .
, y w − 1 , y w , y w +1 , . . .
, y m ) − F ( y 1 , . . . , y w
[Equation] , y (cid:48) w , y w +1 , . . .
, y m ) |≤ c w for all y j ∈ R j and y (cid:48) w ∈ R w for 1 ≤ j ≤ m
Then for any a > 0 , one has Pr( | F ( Y )
[Equation]
considering the plugin-estimator of GPK family:
[Equation]
Without loss of generality, we rewrite the function f φ as:
[Equation]
Assume we freeze all but one element in X n 1 :=
{ x 1 , x 2 , ..., x s , ..., x n 1 } , and only x s is allowed to change its value.
obviously, no matter how this element change, it always lies in the ﬁnite set of support { v 1 , v 2 , ..., v k } , without loss of generality, we assume x s changes its value from v i to v ii , thus the corresponding count of occurrence N i changes to N i − 1 and N ii changes to N ii +we have for x s ∈ X n
[Equation]
= sup i,ii ∈ [ k ] (cid:12)(cid:12) (cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)
[Equation]
≤
[Equation]
= τ n 1 K max Φwhere Φ 2 = (cid:80) j ∈
[ k ] (cid:12)(cid:12)
f φ ( S N j ,n 2 , S N j ,m 2 ) (cid:12)(cid:12)
Note that K i,j :=
K ij
Similarly, for x s ∈ X nc s ≤ τ n 2 K max Φwhere Φ 1 = (cid:80) j ∈
[ k ] | φ ( p 1 j , q 1 j ) | for x s ∈ Y mc s ≤ τ m 1 K max Φ for x s ∈ Y m [Equation]
m
[Equation] K max Φthus according to McDiamid’s inequality, we have
[Equation] − 2 a 2 (cid:80) n 1+ n 2+ m 1+ m 2 i =
c 2 i
We set c [Equation]
for x i ∈ { X n 1 } , c [Equation] = [Equation]
m [Equation] K max Φfor x i ∈
{ Y m 1 } , c i =
τ n 2 K max Φfor x i ∈ { X n 2 } , c
[Equation]
= τ m 2 K max Φfor x i ∈ { Y m 2 } and get[-n 1 + n 
-]
[Equation] (cid:88)
[Equation]
2 max
We are ready to proof theorem 3:
Proof.
Deﬁne
Div (1) n,l,s = | g l ( N ( n ) i + 1 , n ) g
[Equation]
N
[Equation] i [Equation] [Equation]
g l
[Equation]
N
[Equation]
N
[Equation]
m
[Equation]
i
[Equation]
| =
[Equation]
N
[Equation]
[Equation]
N
[Equation]
[Equation]
[Equation]
N i
[Equation]
n 1 ( n 1 − 1) ...
( n
[Equation]
ρ + 1) − N i ( N i − 1) ...
( N i − ρ + 1) n 1 ( n 1 − 1) ...
( n 1 − ρ + 1) (cid:12)(cid:12) (cid:12)(cid:12) · (cid:12)(cid:12) (cid:12) g s ( N ( m ) i , m ) (cid:12)(cid:12) (cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) l
N ( n ) i − l + 1 g l ( N ( n ) i , n ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) · (cid:12)(cid:12)(cid:12) g s ( N ( m ) i , m ) (cid:12)(cid:12)(cid:12) =
l
N ( n ) i − l + 1 g l ( N ( n ) i , n ) g s ( N ( m ) i , m )
Div (2) n,l,s = | g l ( N ( n ) i − 1 , n ) g s ( N ( m ) i , m ) − g l ( N ( n ) i , n ) g s ( N ( m ) i , m ) | = (cid:12)(cid:12) (cid:12)(cid:12) ( N i − 1)( N i − 2) ...
( N i − ρ ) n 1 ( n 1 − 1) ...
( n 1 − ρ + 1) −
N i ( N i − 1) ...
( N i − ρ + 1) n 1 ( n 1 − 1) ...
( n 1 − ρ + 1) (cid:12)(cid:12) (cid:12)(cid:12) · (cid:12)(cid:12) (cid:12) g s ( N ( m ) i , m ) (cid:12)(cid:12) (cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
g
[Equation] [Equation] ( n ) i , n ) [Table]
s
[Table]
N
[Table]
m )
[Table]
[Equation]
[Table]
[Equation]
[Equation]
Recall the mapping function of default-plugin-estimator of polynomial GPK f φ ( S i,n , S i,m ) :
= (cid:80) ol =0 (cid:80) os =
[Equation]
α l
[Equation]
g l
[Equation][-Recall the mapping function of default-plugin-estimator of polynomial GPK f φ ( S i,n , S i,m ) : = (cid:80) ol =0 (cid:80) os =
-]
N [Equation] i , n
[Equation]
g s ( N ( m [Equation] i , m ) Apparently f φ is a monotonic function with respect to N i and M i , thus condition 1. for theorem 5.
is satisﬁed
Since we also have:
[Equation]
thus condition 2.
for theorem 5.
is satisﬁed
A. PROOF OF COROLLERY 3.
Proof.
Deﬁne τ = sup N i
[Equation]
Since we have g ρ ( N,n ) g ρ ( M,n ) =
g ρ ( N, M ) , we could get (cid:12)(cid:12)(cid:12)(cid:12) [Equation] ] √ Z
[Equation] [Table] N ( m 2 ) j , S N ) (cid:17) K max ρ √ 2 n · (cid:80) i ∈ [ k ] (cid:12)(cid:12)(cid:12) g ρ ( N ( n ) i , S N ) − g ρ ( N ( m ) i , S N ) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
A. DETAILS OF R EMARK 3.
GPK E [ F , φ l , X, Y ] = (cid:88) i ∈ [ k ] (cid:88) j ∈ [ k ] (cid:16) g 1 ( N ( n 1 ) i , n 1 ) − g 1 ( N ( m 1 ) i , m 1 ) (cid:17) K ij (cid:16) g 1 ( N ( n 2 ) j , n 2 ) − g 1 ( N ( m 2 ) j , m 2 ) (cid:17) = (cid:88) i ∈ [ k ] (cid:88)
j ∈ [ k ] (cid:32) N ( n 1 ) i n− N ( m 1 ) i m(cid:33) K ij (cid:32) N ( n 2 ) j n− N ( m 2 ) j m(cid:33) = m 1 mm 1 (cid:88) i
= m 2 (cid:88)
j = k ( x i , x j ) + n 1 nn 1 (cid:88) i = n 2 (cid:88) j = k ( y i , y j ) − m 1 nm 1 (cid:88) i = n 2 (cid:88)
j = k ( x i , y j ) − m 2 nm 2 (cid:88) i = n 1 (cid:88) j = k ( x i , y j ) =
[Equation]
A. THE CONVERGENCE BOUNDS OF POWER -MMD WITH ρ =
When ρ = 1 , τ n 1 = ρN i g ρ ( N, n 1 )
=
1 N i · N i n 1 = 1 n 1 For simplicity consider the case where n 1 = n 2 = m
= m 2 = n , from theorem 3.
we have
[Equation]
Since Φ = (cid:80) j ∈ [ [Equation] [Equation] f
[Equation]
we have
[Equation]
recall in (Gretton et al., 2012), the deviation bound for linear time estimator MMD 2 l [ F , p , q ] is
[Equation]
Thus our bound generalize the bound in (Gretton et al., 2012) and provide a tighter version. A. I S BOUNDS GET TIGHTER WHEN ρ GETTING LARGER ? As we’ve already known, ρ = 1 case is equivalent to MMD l in (Gretton et al., 2012), one question rises: Would the performance of cases ρ > 1 better than widely used ρ = 1 case?
According to Corollary 3.2., since e − na 2 8 K 2 max ≤ e − na 2 8 ρ 2 K 2 max , the convergence bounds for ρ > 1 cases seem looser than ρ = 1 case, and this may give a negative answer to the question above.
However, the bound above is based on the worst cases where sup i ( N i ) = n , such that τ n ≤ ρn and Φ ≤ 2 .
In practice, we are less likely to come across such a phenomena, instead, we may assume the sup i ( N i ) to be far smaller. Without loss of generality, assume we have max ( sup i ( N ( n ) i ) n , sup i ( N ( m ) i ) m ) ≤ 1 α , where α ≥ 1 , it is easily seen: [Equation] and
[Equation]
deﬁne τ := max ( τ n 1 , τ n 2 , τ m 1 , τ m 2 ) and Φ := max (Φ n 1 ,m 1 , Φ m 2 ,n 2 )
We have
[Equation]
Plotting the Z b value with respect to variety value of ρ and α in Fig.
1, we can see that for α = 1 , the bound will be looser given larger ρ .
However, for α larger than around 1 . 25 , which means the sup i ( N i ) is slightly smaller than the sample size, the bound will become tighter when ρ is large.
This illustrate the beneﬁt of using power-MMD with larger ρ in practice.
We could also get a tighter bound according to Corollary 3.1.
Practically, it will be much more beneﬁcial to calculate the τ n
[Equation]
) (cid:19) and Φ n,m = (cid:80) i ∈
[ k ] (cid:12)(cid:12) (cid:12) g ρ ( N ( n ) i , n ) − g ρ
( N ( m ) i , m ) (cid:12)(cid:12) (cid:12) on the ﬂy.
That is to say, we do not estimate the convergence bounds before we receive the samples, instead, the calculation of the bounds is carried out together with the calculation of default-plugin-estimators.
Remarkably this is still a distribution-free bounds, since we make no assumptions on the probability functions we apply hypothesis test upon.
However, the issue is although Z decreases when ρ increases, GPK E [ F , φ ρ , X, Y ] also decreases when ρ increases.
It is not clear how ρ will inﬂuence the value of (cid:12)(cid:12) (cid:12)
GPK E [ F ,φ ρ ,X,Y ] √ Z (cid:12)(cid:12)(cid:12) .
A. EXPERIMENT SETTING OF POWER -MMD FOR TOY DATA
As deﬁned in Section ? ? , we use randomly generated potential landscapes to generate distribution functions we need.
The idea is that for the k ﬁxed points { v i , v 2 , ..., v k } , v i ∈ R
d (they are also randomly generated), we use different potential landscapes to represents different distribution.
These distributions thus has the same support { v i , v 2 , ..., v k } , v i ∈ R d , but different probability values { p i , p 2 , ..., p k } , p i ∈ R .
In order to generate valid probability values, we normalize the k values to 1 , thus (cid:80) ki =
p i = 1 .
The potential landscape is smooth, thus pairs of values with smaller L2 norm (cid:107) v i , v j (cid:107) 22 will have smaller difference in probability values | p
i − p
j | , satisfying the requirement of our setting.
The examples of two potential landscape with d = 2 , k = 25 are shown in Fig 3.
Once the distributions are generated, we next need to generate samples according to distribution p and q .
For this experiment, we simply use the acceptance-rejection method, where each time a sample v i is sampled, a random variable z with uniform probability between 0 to 1 is generated.
And the sample value v i is only accepted if z < p i .
The procedure is repeated until the required number of samples are generated.