Original: P OST S ELECTION I NFERENCE WITH I NCOMPLETE M AXIMUM M EAN D ISCREPANCY E STIMATOR
Revised: P OST S ELECTION I NFERENCE WITH I NCOMPLETE M AXIMUM M EAN D ISCREPANCY E STIMATOR

Original: A BSTRACT
Revised: A BSTRACT

Original: Measuring divergence between two distributions is essential in machine learning and statistics and has various applications including binary classiﬁcation, change point detection, and two-sample test.
Revised: Measuring divergence between two distributions is essential in machine learning and statistics and has various applications including binary classiﬁcation, change point detection and two-sample test.

Original: Furthermore, in the era of big data, designing divergence measure that is interpretable and can handle high-dimensional and complex data becomes extremely important.
Revised: Furthermore, in the era of big data, designing divergence measure that is interpretable and can handle high-dimensional and complex data becomes extremely important.

Original: In this paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically signiﬁcant features that discriminate two distributions.
Revised: In this paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically signiﬁcant features that discriminate two distributions.

Original: Speciﬁcally, we employ an additive variant of maximum mean discrepancy (MMD) for features and introduce a general hypothesis test for PSI.
Revised: Speciﬁcally, we employ an additive variant of maximum mean discrepancy (MMD) for features and introduce a general hypothesis test for PSI.

Original: A novel MMD estimator using the incomplete U-statistics, which has an asymptotically normal distribution (under mild assumptions) and gives high detection power in PSI, is also proposed and analyzed theoretically.
Revised: A novel MMD estimator using the incomplete U-statistics, which has an asymptotically normal distribution (under mild assumptions) and gives high detection power in PSI, is also proposed and analyzed theoretically.

Original: Through synthetic and real-world feature selection experiments, we show that the proposed framework can successfully detect statistically signiﬁcant features.
Revised: Through synthetic and real-world feature selection experiments, we show that the proposed framework can successfully detect statistically signiﬁcant features.

Original: Last, we propose a sample selection framework for analyzing different members in the Generative Adversarial Networks (GANs) family.
Revised: Last, we propose a sample selection framework for analyzing different members in the Generative Adversarial Networks (GANs) family.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Computing the divergence between two probability distributions is fundamental to machine learning and has many important applications such as binary classiﬁcation (Friedman et al., 2001), change point detection (Yamada et al., 2013a;
Revised: Computing the divergence between two probability distributions is fundamental to machine learning and has many important applications such as binary classiﬁcation (Friedman et al., 2001), change point detection (Yamada et al., 2013a;

Original: Liu et al., 2013), two-sample test (Gretton et al., 2012; Yamada et al., 2013b), and generative models such as generative adversarial networks (GANs) (Goodfellow et al., 2014; Li et al., 2015b; Nowozin et al., 2016), to name a few.
Revised: Liu et al., 2013), two-sample test (Gretton et al., 2012; Yamada et al., 2013b), and generative models such as generative adversarial networks (GANs) (Goodfellow et al., 2014; Li et al., 2015b; Nowozin et al., 2016), to name a few.

Original: Recently, interpreting the difference between distributions has become an important task in applied machine learning (Mueller & Jaakkola, 2015; Jitkrittum et al., 2016) since it can facilitate scientiﬁc discovery.
Revised: Recently, interpreting the difference between distributions has become an important task in applied machine learning (Mueller & Jaakkola, 2015; Jitkrittum et al., 2016) since it can facilitate scientiﬁc discovery.

Original: For instance, in biomedical binary classiﬁcation tasks, it is common to analyze which variables or features are different between two different distributions (classes).
Revised: For instance, in biomedical binary classiﬁcation tasks, it is common to analyze which variables or features are different between two different distributions (classes).

Original: The simplest approach to measure divergence between two probability densities would be parametric methods.
Revised: The simplest approach to measure divergence between two probability densities would be parametric methods.

Original: For example, t -test can be used if the distributions to be compared are known and well-deﬁned (Anderson, 2001).
Revised: For example, t -test can be used if the distributions to be compared are known and well-deﬁned (Anderson, 2001).

Original: However, in many real-world problems, the property of distribution is not known a priori , and therefore model assumptions are likely to be violated.
Revised: However, in many real-world problems, the property of distribution is not known a priori , and therefore model assumptions are likely to be violated.

Original: In contrast, non-parametric methods can be applied to any distributions without prior assumptions.
Revised: In contrast, non-parametric methods can be applied to any distributions without prior assumptions.

Original: The maximum mean discrepancy (MMD) (Gretton et al., 2012) is an example of non-parametric discrepancy measures and is deﬁned as the difference between the mean embeddings of two distributions in a reproducing kernel Hilbert space (RKHS).
Revised: The maximum mean discrepancy (MMD) (Gretton et al., 2012) is an example of non-parametric discrepancy measures and is deﬁned as the difference between the mean embeddings of two distributions in a reproducing kernel Hilbert space (RKHS).

Original: For RKHSs associated with characteristic kernels (Sriperumbudur et al., 2011), the difference in mean embeddings deﬁnes a proper metric, and thus MMD can capture potentially non-linear difference between distributions (i.e., higher order moments).
Revised: For RKHSs associated with characteristic kernels (Sriperumbudur et al., 2011), the difference in mean embeddings deﬁnes a proper metric, and thus MMD can capture potentially non-linear difference between distributions (i.e., higher order moments).

Original: However, since MMD considers the entire d dimensional vector, it is hard to interpret how individual features contribute to the discrepancy.

Original: [Table]
Revised: [Table]

Original: [Figure] (cid:1) [Figure]
Revised: =EG(cid:1) (cid:13)(cid:43)(cid:46)C(cid:41)IEI(cid:35)(cid:29)(cid:27)BB(cid:46)(cid:1)3EGC(cid:27)B (cid:1) (cid:14)BE(cid:29)A(cid:1)(cid:2)(cid:1) → ∞ (cid:24)(cid:27)C(cid:41)B(cid:31)(cid:1)(cid:2) D # → (cid:7) ,EC(cid:41)B(cid:31)(cid:45)(cid:35)I(cid:46) (cid:23) D (cid:9) (cid:4) (cid:23) (cid:24)(cid:27)C(cid:41)B(cid:31)(cid:1)(cid:2)(cid:4) (cid:23)

Original: (cid:24)(cid:27)C(cid:41)B(cid:31)(cid:1)(cid:2) [Figure] (cid:23)
Revised: D % (cid:14)BE(cid:29)A(cid:1)(cid:2) (cid:4) (cid:30)(cid:31)(cid:33)(cid:31)D(cid:31)G(cid:27)I(cid:31)(cid:30)(cid:1)3EGC(cid:27)B ,ECCED(cid:1)

Original: [Figure]
Revised: [Figure]

Original: (cid:30)(cid:31)(cid:33)(cid:31)D(cid:31)G(cid:27)I(cid:31)(cid:30)(cid:1)3EGC(cid:27)B
Revised: (cid:34)E(cid:35)(cid:29)(cid:31) (cid:1) (cid:14)BE(cid:29)A(cid:1)(cid:2)(cid:1)) D (cid:24)(cid:27)C(cid:41)B(cid:31)(cid:1)(cid:2)(cid:1)

Original: [Figure]
Revised: [Figure]

Original: (cid:1)
Revised: (cid:1)GD

Original: [Figure] (cid:24)(cid:27)C(cid:41)B(cid:31)(cid:1)(cid:2)(cid:1) [Figure]
Revised: However, since MMD considers the entire d dimensional vector, it is hard to interpret how individual features contribute to the discrepancy.

Original: To deal with the interpretability issue, divergence measure with feature selection has been actively studied (Yamada et al., 2013a; Mueller & Jaakkola, 2015; Jitkrittum et al., 2016).
Revised: To deal with the interpretability issue, divergence measure with feature selection has been actively studied (Yamada et al., 2013a; Mueller & Jaakkola, 2015; Jitkrittum et al., 2016).

Original: For instance, in Mueller & Jaakkola (2015), the Wasserstein divergence is employed as a divergence measure and (cid:96) 1 regularizer is used for feature selection.
Revised: For instance, in Mueller & Jaakkola (2015), the Wasserstein divergence is employed as a divergence measure and (cid:96) 1 regularizer is used for feature selection.

Original: However, these approaches focus on detecting a set of features that discriminate two distributions.
Revised: However, these approaches focus on detecting a set of features that discriminate two distributions.

Original: But for scientiﬁc discovery applications such as biomarker discovery, it might be preferable to test the signiﬁcance of each selected feature (e.g., one biomarker).
Revised: But for scientiﬁc discovery applications such as biomarker discovery, it might be preferable to test the signiﬁcance of each selected feature (e.g., one biomarker).

Original: A naive approach would be to select features from one dataset and then test the selected features using the same data.
Revised: A naive approach would be to select features from one dataset and then test the selected features using the same data.

Original: However, in such case, selection bias is included and thus the false positive rate cannot be controlled.
Revised: However, in such case, selection bias is included and thus the false positive rate cannot be controlled.

Original: Therefore, it is crucial in hypothesis testing that the selection event should be taken into account to correct the bias.
Revised: Therefore, it is crucial in hypothesis testing that the selection event should be taken into account to correct the bias.

Original: To the best of our knowledge, there is not yet an existing framework that tests the signiﬁcance of selected features that distinguish between two different distributions on the same dataset.
Revised: To the best of our knowledge, there is not yet an existing framework that tests the signiﬁcance of selected features that distinguish between two different distributions on the same dataset.

Original: In this paper, we propose mmdInf , a post selection inference (PSI) algorithm for distribution comparison, which ﬁnds a set of statistically signiﬁcant features that discriminate between two distributions.
Revised: In this paper, we propose mmdInf , a post selection inference (PSI) algorithm for distribution comparison, which ﬁnds a set of statistically signiﬁcant features that discriminate between two distributions.

Original: mmdInf enjoys several compelling properties.
Revised: mmdInf enjoys several compelling properties.

Original: First, it is a non-parametric method based on kernels, and thus it can detect features that distinguish various types of distributions.
Revised: First, it is a non-parametric method based on kernels, and thus it can detect features that distinguish various types of distributions.

Original: Second, the proposed framework is general and can be applied to not only feature selection but also other distribution comparison problems, such as dataset comparison.
Revised: Second, the proposed framework is general and can be applied to not only feature selection but also other distribution comparison problems, such as dataset comparison.

Original: In mmdInf , we employ the recently developed PSI algorithm (Lee et al., 2016) and use MMD as a divergence measure.
Revised: In mmdInf , we employ the recently developed PSI algorithm (Lee et al., 2016) and use MMD as a divergence measure.

Original: However, the standard empirical squared MMD estimator has a degenerated null distribution, which violates the requirement of a proper Normal distribution for the current PSI algorithm.
Revised: However, the standard empirical squared MMD estimator has a degenerated null distribution, which violates the requirement of a proper Normal distribution for the current PSI algorithm.

Original: To address this issue, we apply the block estimate (Zaremba et al., 2013) and the linear estimate (Gretton et al., 2012) of MMD.
Revised: To address this issue, we apply the block estimate (Zaremba et al., 2013) and the linear estimate (Gretton et al., 2012) of MMD.

Original: Furthermore, we propose a new empirical estimate of MMD based on the incomplete
Revised: Furthermore, we propose a new empirical estimate of MMD based on the incomplete

Original: U-statistics (incomplete MMD estimator) (Blom, 1976; Janson, 1984; Lee, 1990) and show that it asymptotically follows the normal distribution, and has much greater detection power (compared to the existing estimators of MMD) in PSI.
Revised: U-statistics (incomplete MMD estimator) (Blom, 1976; Janson, 1984; Lee, 1990) and show that it asymptotically follows the normal distribution, and has much greater detection power (compared to the existing estimators of MMD) in PSI.

Original: Finally, we propose a framework to analyze different members in the GAN (Goodfellow et al., 2014) family based on mmdInf .
Revised: Finally, we propose a framework to analyze different members in the GAN (Goodfellow et al., 2014) family based on mmdInf .

Original: We elucidate the theoretical properties of the incomplete U-statistics estimate of MMD and show that mmdInf can successfully detect signiﬁcant features through feature selection experiments.
Revised: We elucidate the theoretical properties of the incomplete U-statistics estimate of MMD and show that mmdInf can successfully detect signiﬁcant features through feature selection experiments.

Original: The contributions of our paper are summarized as follows: (1) We propose a non-parametric PSI algorithm mmdInf for distribution comparison.
Revised: The contributions of our paper are summarized as follows: (1) We propose a non-parametric PSI algorithm mmdInf for distribution comparison.

Original: (2) We propose the incomplete MMD estimator and investigate its theoretical properties (see Figure 1).
Revised: (2) We propose the incomplete MMD estimator and investigate its theoretical properties (see Figure 1).

Original: (3) We propose a sample selection framework based on mmdInf that can be used for analyzing generative models.
Revised: (3) We propose a sample selection framework based on mmdInf that can be used for analyzing generative models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Divergence measures in general can be divided into two categories: f -divergence (Ali & Silvey, 1966), such as the Kullback-Leibler divergence (Cover & Thomas, 2012) or the α -divergence (R´enyi et al., 1961; P´oczos & Schneider, 2011), and integral probability metric (M¨uller, 1997), including the total variation distance (Shorack & Shorack, 2000) and the Wasserstein distance in its dual (Villani, 2008).
Revised: Divergence measures in general can be divided into two categories: f -divergence (Ali & Silvey, 1966), such as the Kullback-Leibler divergence (Cover & Thomas, 2012) or the α -divergence (R´enyi et al., 1961; P´oczos & Schneider, 2011), and integral probability metric (M¨uller, 1997), including the total variation distance (Shorack & Shorack, 2000) and the Wasserstein distance in its dual (Villani, 2008).

Original: In this work we consider the maximum mean discrepancy (MMD) (Gretton et al., 2012), which falls into the second category.
Revised: In this work we consider the maximum mean discrepancy (MMD) (Gretton et al., 2012), which falls into the second category.

Original: Although these divergence measures can be used for testing the discrepancy between p ( x ) and q ( x ) , it is hard to test the signiﬁcance of one of k selected features from the entire d features, where the setup is useful for scientiﬁc discovery tasks such as biomarker discovery.
Revised: Although these divergence measures can be used for testing the discrepancy between p ( x ) and q ( x ) , it is hard to test the signiﬁcance of one of k selected features from the entire d features, where the setup is useful for scientiﬁc discovery tasks such as biomarker discovery.

Original: Recently, a MMD-based change point detection algorithm, which can compute the p -value of the largest score, has been proposed (Li et al., 2015a).
Revised: Recently, a MMD-based change point detection algorithm, which can compute the p -value of the largest score, has been proposed (Li et al., 2015a).

Original: However, this method can only test the largest score (i.e., one-feature).
Revised: However, this method can only test the largest score (i.e., one-feature).

Original: Thus, it is not clear whether the approach can be extended to feature selection in general.
Revised: Thus, it is not clear whether the approach can be extended to feature selection in general.

Original: A novel testing framework for the post selection inference (PSI) has been recently proposed (Lee et al., 2016), in which statistical inference after feature selection with Lasso (Tibshirani, 1996) is investigated.
Revised: A novel testing framework for the post selection inference (PSI) has been recently proposed (Lee et al., 2016), in which statistical inference after feature selection with Lasso (Tibshirani, 1996) is investigated.

Original: This work shows that statistical inference conditioned on the selection event can be done for linear regression models with Gaussian noise if the selection event can be written as a set of linear constraints.
Revised: This work shows that statistical inference conditioned on the selection event can be done for linear regression models with Gaussian noise if the selection event can be written as a set of linear constraints.

Original: However, the PSI algorithm needs to assume a Gaussian response, which is a relatively strong assumption, and consequently it cannot be directly applied for non-Gaussian output problems such as classiﬁcation.
Revised: However, the PSI algorithm needs to assume a Gaussian response, which is a relatively strong assumption, and consequently it cannot be directly applied for non-Gaussian output problems such as classiﬁcation.

Original: To deal with this issue, a kernel based
Revised: To deal with this issue, a kernel based

Original: PSI algorithm ( hsicInf ) for independence test has been proposed (Yamada et al., 2018), in which the Hilbert-Schmidt Independence Criterion (HSIC)
Revised: PSI algorithm ( hsicInf ) for independence test has been proposed (Yamada et al., 2018), in which the Hilbert-Schmidt Independence Criterion (HSIC)

Original: (Gretton et al., 2005) is employed to measure the independence between input and its output, and thus signiﬁcance test can be performed on non-Gaussian data.
Revised: (Gretton et al., 2005) is employed to measure the independence between input and its output, and thus signiﬁcance test can be performed on non-Gaussian data.

Original: In this paper, we propose an alternative kernel based inference algorithm for distribution comparison called mmdInf , which can be used for both feature selection and binary classiﬁcation.
Revised: In this paper, we propose an alternative kernel based inference algorithm for distribution comparison called mmdInf , which can be used for both feature selection and binary classiﬁcation.

Original: In Sections 3.4 and 5.3, we manifest how we analyze different members in the GANs (Goodfellow et al., 2014) family.
Revised: In Sections 3.4 and 5.3, we manifest how we analyze different members in the GANs (Goodfellow et al., 2014) family.

Original: Here, we discuss several evaluation metrics that have been proposed to compare generative models.
Revised: Here, we discuss several evaluation metrics that have been proposed to compare generative models.

Original: For example, the inception scores (Salimans et al., 2016) and the mode scores (Che et al., 2016) measure the quality and diversity of the generated samples, but they were not able to detect overﬁtting and mode dropping / collapsing for generated samples.
Revised: For example, the inception scores (Salimans et al., 2016) and the mode scores (Che et al., 2016) measure the quality and diversity of the generated samples, but they were not able to detect overﬁtting and mode dropping / collapsing for generated samples.

Original: The Frechet inception distance (FID) (Heusel et al., 2017) deﬁnes a score using the ﬁrst two moments of the real and generated distributions, whereas the classiﬁer two-sample tests (Lopez-Paz & Oquab, 2016) considers the classiﬁcation accuracy of a binary classiﬁer as a statistic for two-sample test.
Revised: The Frechet inception distance (FID) (Heusel et al., 2017) deﬁnes a score using the ﬁrst two moments of the real and generated distributions, whereas the classiﬁer two-sample tests (Lopez-Paz & Oquab, 2016) considers the classiﬁcation accuracy of a binary classiﬁer as a statistic for two-sample test.

Original: Although the above metrics are reasonable in terms of discriminability, robustness, and efﬁciency, the distances between samples are required to be computed in a suitable feature space.
Revised: Although the above metrics are reasonable in terms of discriminability, robustness, and efﬁciency, the distances between samples are required to be computed in a suitable feature space.

Original: We can also use the kernel density estimation (KDE); or more recently, Wu et al.
Revised: We can also use the kernel density estimation (KDE); or more recently, Wu et al.

Original: (2016) proposed to apply the annealed importance sampling (AIS) to estimate the likelihood of the decoder-based generative models.
Revised: (2016) proposed to apply the annealed importance sampling (AIS) to estimate the likelihood of the decoder-based generative models.

Original: Nevertheless, these approaches need the access to the generative model for computing the likelihood, which are less favorable comparing to the model agnostic approaches which rely only on a ﬁnite generated sample set.
Revised: Nevertheless, these approaches need the access to the generative model for computing the likelihood, which are less favorable comparing to the model agnostic approaches which rely only on a ﬁnite generated sample set.

Original: On the other hand, the maximum mean discrepancy (MMD) (Gretton et al., 2012) is preferred against its competitors (Sutherland et al., 2016; Huang et al., 2018).
Revised: On the other hand, the maximum mean discrepancy (MMD) (Gretton et al., 2012) is preferred against its competitors (Sutherland et al., 2016; Huang et al., 2018).

Original: Therefore, we propose the mmdInf based GANs analysis framework.
Revised: Therefore, we propose the mmdInf based GANs analysis framework.

Original:  P ROPOSED M ETHOD ( MMD I NF )
Revised:  P ROPOSED M ETHOD ( MMD I NF )

Original: In this section, we introduce a PSI algorithm with MMD (Gretton et al., 2012).
Revised: In this section, we introduce a PSI algorithm with MMD (Gretton et al., 2012).

Original: 3. P OST S ELECTION I NFERENCE (PSI)
Revised: 3. P OST S ELECTION I NFERENCE (PSI)

Original: Suppose we are given independent and identically distributed (i.i.d.)
Revised: Suppose we are given independent and identically distributed (i.i.d.)

Original: samples X = { x i } mi =1 ∈ R d
Revised: samples X = { x i } mi =1 ∈ R d

Original: × m from a d -dimensional distribution p and i.i.d.
Revised: × m from a d -dimensional distribution p and i.i.d.

Original: samples Y = { y j } nj =1 ∈ R d × n from another d -dimensional distribution q .
Revised: samples Y = { y j } nj =1 ∈ R d × n from another d -dimensional distribution q .

Original: Our goal is to ﬁnd k < d features that differentiate between X and Y and test whether each selected feature is statistically signiﬁcant .
Revised: Our goal is to ﬁnd k < d features that differentiate between X and Y and test whether each selected feature is statistically signiﬁcant .

Original: Let S be a set of selected features, we consider the following hypothesis test:
Revised: Let S be a set of selected features, we consider the following hypothesis test:

Original: • H 0 : (cid:80) d s =1 η s (cid:98) D
Revised: • H 0 : (cid:80) ds =1 η s (cid:98) D

Original: ( X ( s ) , Y ( s ) )
Revised: ( X ( s ) , Y ( s ) )

Original: = 

Revised: =

Original: | S was selected, • H 1 : (cid:80) ds
Revised: | S was selected, • H 1 : (cid:80) ds

Original: =1 η s (cid:98) D
Revised: =1 η s (cid:98) D

Original: ( X ( s ) , Y ( s ) )
Revised: ( X ( s ) , Y ( s ) )

Original: (cid:54) = 

Revised: (cid:54) =

Original: | S was selected,
Revised: | S was selected,

Original: where (cid:98) D ( X ( s ) , Y ( s ) ) is the estimated discrepancy measure for the selected feature s and η = [ η 1 , . . .
Revised: where (cid:98) D ( X ( s ) , Y ( s ) ) is the estimated discrepancy measure for the selected feature s and η = [ η 1 , . . .

Original: , η d ] (cid:62) ∈
Revised: , η d ] (cid:62) ∈

Original: R
Revised: R

Original: d is an arbitrary pre-deﬁned parameter.
Revised: d is an arbitrary pre-deﬁned parameter.

Original: To test the s -th feature, we can set η as a unit vector whose s -th position is 1 and zero otherwise.
Revised: To test the s -th feature, we can set η as a unit vector whose s -th position is 1 and zero otherwise.

Original: We employ the post-selection inference (PSI) framework (Lee et al., 2016) to test the hypothesis.
Revised: We employ the post-selection inference (PSI) framework (Lee et al., 2016) to test the hypothesis.

Original: Thanks to the cumulative distribution of Theorem 1, we can compute the p -value of (cid:80) [Equation]
Revised: Thanks to the cumulative distribution of Theorem 1, we can compute the p -value of (cid:80) ds =1 η s (cid:98) D

Original:  ) , Y [Equation] s ) ) under feature selection.
Revised: ( X ( s ) , Y ( s ) ) under feature selection.

Original: Theorem 1 (Lee et al., 2016) Suppose that z ∼ N ( µ , Σ ) , and the feature selection event can be expressed as Az ≤ b for some matrix A and vector b , then for any given feature represented by η ∈ R n we have
Revised: Theorem 1 (Lee et al., 2016) Suppose that z ∼ N ( µ , Σ ) , and the feature selection event can be expressed as Az ≤ b for some matrix A and vector b , then for any given feature represented by η ∈ R n we have

Original: [Equation]
Revised: [Equation]

Original: where F [ a,b ] µ,σ 2 ( x ) is the cumulative distribution function (CDF) of a truncated normal distribution with mean µ and variance σ 2 truncated at [a,b].
Revised: where F [ a,b ] µ,σ 2 ( x ) is the cumulative distribution function (CDF) of a truncated normal distribution with mean µ and variance σ 2 truncated at [a,b].

Original: Given that α =
Revised: Given that α =

Original: A Σ η η (cid:62) Σ η , the lower and upper truncation points are given by
Revised: A Σ η η (cid:62) Σ η , the lower and upper truncation points are given by

Original: [Equation]
Revised: [Equation]

Original: Feature Selection with Discrepancy Measure: Assume we have an estimate of a discrepancy measure for each feature: z = [ (cid:98) D
Revised: Feature Selection with Discrepancy Measure: Assume we have an estimate of a discrepancy measure for each feature: z = [ (cid:98) D

Original: ( X (1) , Y (1) ) , . .
Revised: ( X (1) , Y (1) ) , . .

Original: , (cid:98) D ( X ( d ) , Y ( d ) )] (cid:62) ∈
Revised: , (cid:98) D ( X ( d ) , Y ( d ) )] (cid:62) ∈

Original: R
Revised: R

Original: d ∼ N ( µ , Σ ) , where (cid:98) D ( · , · ) has large positive value when two distribution are different.
Revised: d ∼ N ( µ , Σ ) , where (cid:98) D ( · , · ) has large positive value when two distribution are different.

Original: We select the top- k features with largest discrepancy scores.
Revised: We select the top- k features with largest discrepancy scores.

Original: We denote the index set of the selected k features by S , and that of the unselected ¯ k = d − k features by ¯ S .
Revised: We denote the index set of the selected k features by S , and that of the unselected ¯ k = d − k features by ¯ S .

Original: This feature selection event can be characterized by (cid:98) D
Revised: This feature selection event can be characterized by (cid:98) D

Original: ( X ( s ) , Y ( s ) ) ≥ (cid:98) D
Revised: ( X ( s ) , Y ( s ) ) ≥ (cid:98) D

Original: ( X ( (cid:96) ) , Y ( (cid:96) ) ) , for all ( s, (cid:96) )
Revised: ( X ( (cid:96) ) , Y ( (cid:96) ) ) , for all ( s, (cid:96) )

Original: ∈ S × ¯ S .
Revised: ∈ S × ¯ S .

Original: Note that we have in total k · ¯ k constraints.
Revised: Note that we have in total k · ¯ k constraints.

Original: The selection event can be rewritten as A s,(cid:96) z ≤ 0 , for all ( s, (cid:96) )
Revised: The selection event can be rewritten as A s,(cid:96) z ≤ 0 , for all ( s, (cid:96) )

Original: ∈ S × ¯ S , where A s,(cid:96) =
Revised: ∈ S × ¯ S , where A s,(cid:96) =

Original: [0 · · · 0 − 1 (cid:124)(cid:123)(cid:122)(cid:125)
Revised: [0 · · · 0 − 1 (cid:124)(cid:123)(cid:122)(cid:125)

Original: s 0 · · ·1 (cid:124)(cid:123)(cid:122)(cid:125) (cid:96) 0 · · · 0] and A (cid:62) s,(cid:96)
Revised: s 0 · · ·1 (cid:124)(cid:123)(cid:122)(cid:125) (cid:96) 0 · · · 0] and A (cid:62) s,(cid:96)

Original: ∈ R
Revised: ∈ R

Original: d is a row vector of A ∈ R ( k · ¯ k ) ×
Revised: d is a row vector of A ∈ R ( k · ¯ k ) ×

Original: d .
Revised: d .

Original: Under such construction, Az ≤ b can be satisﬁed by setting b = 0 .
Revised: Under such construction, Az ≤ b can be satisﬁed by setting b = 0 .

Original: 3. M AXIMUM M EAN D ISCREPANCY (MMD)
Revised: 3. M AXIMUM M EAN D ISCREPANCY (MMD)

Original: We employ the maximum mean discrepancy (MMD) (Gretton et al., 2012) as divergence measure.
Revised: We employ the maximum mean discrepancy (MMD) (Gretton et al., 2012) as divergence measure.

Original: Let F be the unit ball in a reproducing kernel Hilbert space (RKHS) and K ( x , x (cid:48) )
Revised: Let F be the unit ball in a reproducing kernel Hilbert space (RKHS) and K ( x , x (cid:48) )

Original: the corresponding positive deﬁnite kernel with E x , x (cid:48) ∼ p
Revised: the corresponding positive deﬁnite kernel with E x , x (cid:48) ∼ p

Original: [ (cid:112) K ( x , x )] < ∞ , E x , x [Equation]
Revised: [ (cid:112) K ( x , x )] < ∞ , E x , x (cid:48)

Original: ∼ q [ (cid:112) K ( x , x )] < ∞ , the squared population MMD is deﬁned as
Revised: ∼ q [ (cid:112) K ( x , x )] < ∞ , the squared population MMD is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where E x , y denotes the expectation over independent random variables x with distribution p and y with distribution q .
Revised: where E x , y denotes the expectation over independent random variables x with distribution p and y with distribution q .

Original: It has been shown that if the kernel function K ( x , x (cid:48) ) is characteristic, then
Revised: It has been shown that if the kernel function K ( x , x (cid:48) ) is characteristic, then

Original: MMD [ F , X , Y ] = 

Revised: MMD [ F , X , Y ] =

Original: if and only if p = q (Gretton et al., 2012).
Revised: if and only if p = q (Gretton et al., 2012).

Original: In the following, we introduce two existing MMD estimators and then propose the incomplete U-statistics MMD estimator.
Revised: In the following, we introduce two existing MMD estimators and then propose the incomplete U-statistics MMD estimator.

Original: (Complete) U-statistics estimator (Gretton et al., 2012):
Revised: (Complete) U-statistics estimator (Gretton et al., 2012):

Original: We use the Gaussian kernel: K ( x , x (cid:48) )
Revised: We use the Gaussian kernel: K ( x , x (cid:48) )

Original: =
Revised: =

Original: [-exp (cid:16) − (cid:107) x − x (cid:48) (cid:107) 
 σ 2 x-]
Revised: [-exp (cid:16) − (cid:107) x − x (cid:48) (cid:107) 
 σ 2 x-][Equation]

Original: (cid:17) , where σ x > 0 is the Gaussian width.
Revised: (cid:17) , where σ x > 0 is the Gaussian width.

Original: When m = n , the complete U-statistics of
Revised: When m = n , the complete U-statistics of

Original: MMD is deﬁned as
Revised: MMD is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where h ( u , u (cid:48) )
Revised: where h ( u , u (cid:48) )

Original: =
Revised: =

Original: K ( x , x (cid:48) )
Revised: K ( x , x (cid:48) )

Original: + K ( y , y (cid:48) )
Revised: + K ( y , y (cid:48) )

Original: − K ( x , y (cid:48) )
Revised: − K ( x , y (cid:48) )

Original: − K ( x (cid:48) , y ) is the U-statistics kernel for MMD and u =
Revised: − K ( x (cid:48) , y ) is the U-statistics kernel for MMD and u =

Original: [ x (cid:62) y (cid:62) ] (cid:62)
Revised: [ x (cid:62) y (cid:62) ] (cid:62)

Original: ∈ R 2 d .
Revised: ∈ R 2 d .

Original: However, since the complete U-statistics estimator of MMD is degenerated under p =
Revised: However, since the complete U-statistics estimator of MMD is degenerated under p =

Original: q
Revised: q

Original: and does not follow normal distribution, this estimator cannot be used in PSI. 
Revised: and does not follow normal distribution, this estimator cannot be used in PSI. ds =

Revised: 1 η s MMD 2 inc ( X ( s ) , Y ( s ) ) =

Original: Block estimator (Zaremba et al., 2013):
Revised: Block estimator (Zaremba et al., 2013):

Original: The block estimate of MMD is given by
Revised: The block estimate of MMD is given by

Original: [Equation]
Revised: [Equation]

Original: = where X =
Revised: where X =

Original: [ X 1 , . .
Revised: [ X 1 , . .

Original: , X m/B 1 ] , Y =
Revised: , X m/B 1 ] , Y =

Original: [ Y 1 , . . .
Revised: [ Y 1 , . . .

Original: , Y n/B 2 ] , and X i ∈ R
Revised: , Y n/B 2 ] , and X i ∈ R

Original: d × B 1 and Y i ∈ R
Revised: d × B 1 and Y i ∈ R

Original: d × B 2 are i -th partitioned data.
Revised: d × B 2 are i -th partitioned data.

Original: Here, we assume that the number of blocks m/B 1 = n/B 2 is an integer and
Revised: Here, we assume that the number of blocks m/B 1 = n/B 2 is an integer and

Original: B = B 1 = B 2 .
Revised: B = B 1 = B 2 .

Original: A commonly used heuristic for the block size is to set B = √ n .
Revised: A commonly used heuristic for the block size is to set B = √ n .

Original: This estimator asymptotically follows the normal distribution when B 1 and B 2 are ﬁnite and m and n go to inﬁnity.
Revised: This estimator asymptotically follows the normal distribution when B 1 and B 2 are ﬁnite and m and n go to inﬁnity.

Original: The block estimator can be used for PSI, but the variance and normality depends on the partition of
Revised: The block estimator can be used for PSI, but the variance and normality depends on the partition of

Original: X and Y .
Revised: X and Y .

Original: Speciﬁcally, when the total number of samples is small, then a small block size would result in high variance, whereas larger block size tends to result in non-Gaussian response.
Revised: Speciﬁcally, when the total number of samples is small, then a small block size would result in high variance, whereas larger block size tends to result in non-Gaussian response.

Original: Incomplete U-statistics estimator:
Revised: Incomplete U-statistics estimator:

Original: The described problems of the block-estimator motivated us to design a new MMD estimator that is normally distributed and has smaller variance.
Revised: The described problems of the block-estimator motivated us to design a new MMD estimator that is normally distributed and has smaller variance.

Original: We therefore propose an MMD estimator based on the incomplete U-statistics (Blom, 1976; Janson, 1984; Lee, 1990).
Revised: We therefore propose an MMD estimator based on the incomplete U-statistics (Blom, 1976; Janson, 1984; Lee, 1990).

Original: The incomplete U-statistics estimator of MMD is given by
Revised: The incomplete U-statistics estimator of MMD is given by

Original: [Equation]
Revised: [Equation]

Original: where D is a subset of S n = { ( i, j ) } i (cid:54)
Revised: where D is a subset of S n = { ( i, j ) } i (cid:54)

Original: =
Revised: =

Original: j , and (cid:96) = |D| .
Revised: j , and (cid:96) = |D| .

Original: D can be ﬁxed design or random design.
Revised: D can be ﬁxed design or random design.

Original: In particular, if we design D as D = { (1 , 2) , (3 , 4) , . .
Revised: In particular, if we design D as D = { (1 , 2) , (3 , 4) , . .

Original: , ( n − 1 , n ) } then the incomplete U-statistic corresponds to the linear-time MMD estimator (Gretton et al., 2012).
Revised: , ( n − 1 , n ) } then the incomplete U-statistic corresponds to the linear-time MMD estimator (Gretton et al., 2012).

Original: 3. C OVARIANCE M ATRIX ESTIMATION
Revised: 3. C OVARIANCE M ATRIX ESTIMATION

Original: For PSI, we need to estimate the covariance matrix Σ from data.
Revised: For PSI, we need to estimate the covariance matrix Σ from data.

Original: Block estimator To estimate Σ , we ﬁrst compute H b ∈ R d × ( n/B ) whose elements are the meansubtracted MMDs.
Revised: Block estimator To estimate Σ , we ﬁrst compute H b ∈ R d × ( n/B ) whose elements are the meansubtracted MMDs.

Original: Then, we regard H b as the data matrix and use a standard covariance estimator for estimating (cid:98) Σ b .
Revised: Then, we regard H b as the data matrix and use a standard covariance estimator for estimating (cid:98) Σ b .

Original: Note that for small n , the number of blocks n/B tends to be smaller than the dimension d , and thus, the estimation accuracy of covariance matrix is low and affects the detection accuracy.
Revised: Note that for small n , the number of blocks n/B tends to be smaller than the dimension d , and thus, the estimation accuracy of covariance matrix is low and affects the detection accuracy.

Original: To handle this issue, we employ the POET algorithm
Revised: To handle this issue, we employ the POET algorithm

Original: (Fan et al., 2013).
Revised: (Fan et al., 2013).

Original: Incomplete U-statistics estimator To estimate Σ , we ﬁrst compute H inc ∈ R d × (cid:96) whose elements are the mean-subtracted U-stat kernels.
Revised: Incomplete U-statistics estimator To estimate Σ , we ﬁrst compute H inc ∈ R d × (cid:96) whose elements are the mean-subtracted U-stat kernels.

Original: Then, we regard H inc as the data matrix and can use a standard covariance estimator to compute (cid:98) Σ inc .
Revised: Then, we regard H inc as the data matrix and can use a standard covariance estimator to compute (cid:98) Σ inc .

Original: The estimation performance of the covariance estimation for the block estimate heavily depends on the block size B .
Revised: The estimation performance of the covariance estimation for the block estimate heavily depends on the block size B .

Original: That is, if B is large, we need to estimate (cid:98) Σ b from a small number of samples.
Revised: That is, if B is large, we need to estimate (cid:98) Σ b from a small number of samples.

Original: On the other hand, in the incomplete U-statistics estimator, we can set (cid:96) =
Revised: On the other hand, in the incomplete U-statistics estimator, we can set (cid:96) =

Original: rn (cid:29) n/B , where r >is a constant (See Section 4).
Revised: rn (cid:29) n/B , where r >is a constant (See Section 4).

Original: Thus, in practice, the estimated covariance matrix tends to be more accurate than the block estimator counterpart.
Revised: Thus, in practice, the estimated covariance matrix tends to be more accurate than the block estimator counterpart.

Original: Figure 6 in the supplementary material shows the
Revised: Figure 6 in the supplementary material shows the

Original: MSEs between the true and the estimated covariance matrices of the incomplete estimate and the block estimate, respectively.
Revised: MSEs between the true and the estimated covariance matrices of the incomplete estimate and the block estimate, respectively.

Original: As can be seen, the error of the incomplete estimate is smaller than that of the block estimate, which is an advantage of the incomplete MMD over the block MMD in PSI.
Revised: As can be seen, the error of the incomplete estimate is smaller than that of the block estimate, which is an advantage of the incomplete MMD over the block MMD in PSI.

Original: 3. A DDITIONAL A PPLICATIONS (GAN S ANALYSIS )
Revised: 3. A DDITIONAL A PPLICATIONS (GAN S ANALYSIS )

Original: The proposed PSI framework is general and can be used for not only feature selection but also sample selection.
Revised: The proposed PSI framework is general and can be used for not only feature selection but also sample selection.

Original: In generative modeling, the generated distribution should match the data distribution; in other words, the discrepancy between the generated data and real data should be small.
Revised: In generative modeling, the generated distribution should match the data distribution; in other words, the discrepancy between the generated data and real data should be small.

Original: We can therefore apply the selective inference algorithm and use the signiﬁcance value to evaluate the generation quality.
Revised: We can therefore apply the selective inference algorithm and use the signiﬁcance value to evaluate the generation quality.

Original: In this paper we apply mmdInf to compare the performance of GANs.
Revised: In this paper we apply mmdInf to compare the performance of GANs.

Original: We ﬁrst select the model whose generated samples has the smallest MMD score with the real data and then perform the hypothesis test.
Revised: We ﬁrst select the model whose generated samples has the smallest MMD score with the real data and then perform the hypothesis test.

Original: Let x ( s ) i ∈
Revised: Let x ( s ) i ∈

Original: R p be a feature vector generated by s -th GAN model with random seed i and y j ∈ R p is a feature vector of an original image.
Revised: R p be a feature vector generated by s -th GAN model with random seed i and y j ∈ R p is a feature vector of an original image.

Original: Image features can be extracted by pre-trained Resnet (He et al., 2016) or auto-encoders.
Revised: Image features can be extracted by pre-trained Resnet (He et al., 2016) or auto-encoders.

Original: The hypothesis test can be written as
Revised: The hypothesis test can be written as

Original: • H 0 : MMD 2 inc [ F , X ( s ) , Y ]=0 | s generates samples closest to the real distribution, • H 1 : MMD 2 inc [ F , X ( s ) , Y ] (cid:54) = 

Revised: • H 0 : MMD 2 inc [ F , X ( s ) , Y ]=0 | s generates samples closest to the real distribution, • H 1 : MMD 2 inc [ F , X ( s ) , Y ] (cid:54) =

Original: | s generates samples closest to the real distribution.
Revised: | s generates samples closest to the real distribution.

Original: Since we want to test the best generator that minimizes the discrepancy between generated and true samples (e.g., low MMD score), this sample selection event can be characterized by
Revised: Since we want to test the best generator that minimizes the discrepancy between generated and true samples (e.g., low MMD score), this sample selection event can be characterized by

Original: [Equation]
Revised: [Equation]

Original:  T HEORETICAL A NALYSIS OF I NCOMPLETE MMD
Revised:  T HEORETICAL A NALYSIS OF I NCOMPLETE MMD

Original: We investigate the theoretical properties of the incomplete MMD estimator under the random design with replacement.
Revised: We investigate the theoretical properties of the incomplete MMD estimator under the random design with replacement.

Original: For simplicity, we denote MMD 2 inc [ F , X , Y ] =
Revised: For simplicity, we denote MMD 2 inc [ F , X , Y ] =

Original: MMD 2 inc , MMD 2 [ F , p, q ] =
Revised: MMD 2 inc , MMD 2 [ F , p, q ] =

Original: MMD 2 , MMD 2 inc [ F , X ( s ) , Y ( s ) ]
Revised: MMD 2 , MMD 2 inc [ F , X ( s ) , Y ( s ) ]

Original: = MMD 2 , ( s ) inc , and MMD 

Revised: = MMD 2 , ( s ) inc , and MMD

Original: [ F , p ( s ) , q ( s ) ] = MMD 2 , ( s ) , respectively.
Revised: [ F , p ( s ) , q ( s ) ] = MMD 2 , ( s ) , respectively.

Original: We introduce the conditional expectations for the U-statistic kernel h ( u , u (cid:48) ) as h 1 = E u (cid:48)
Revised: We introduce the conditional expectations for the U-statistic kernel h ( u , u (cid:48) ) as h 1 = E u (cid:48)

Original: [ h ( u , u (cid:48) )] and h 

Revised: [ h ( u , u (cid:48) )] and h

Original: = h ( u , u (cid:48) ) .
Revised: = h ( u , u (cid:48) ) .

Original: See the supplementary material for proof.
Revised: See the supplementary material for proof.

Original: Theorem 2 Let c be the smallest integer such that h c (cid:54) =
Revised: Theorem 2 Let c be the smallest integer such that h c (cid:54) =

Original: MMD 2 , and let n and (cid:96) tend to inﬁnity such that γ = lim n,(cid:96) →∞ n − c (cid:96), 0 ≤ γ ≤ ∞ .
Revised: MMD 2 , and let n and (cid:96) tend to inﬁnity such that γ = lim n,(cid:96) →∞ n − c (cid:96), 0 ≤ γ ≤ ∞ .

Original: For sampling with replacement, we have  
Revised: For sampling with replacement, we have  

Original: [Equation]
Revised: [Equation]

Original: where V is the random variable of the limit distribution of n c 2 ( MMD 2 u − MMD 2 ) , T is the random variable of N (0 , σ 2 ) , σ 2 = V ar ( h ( u , u (cid:48) )) , and T and V are independent.
Revised: where V is the random variable of the limit distribution of n c 2 ( MMD 2 u − MMD 2 ) , T is the random variable of N (0 , σ 2 ) , σ 2 = V ar ( h ( u , u (cid:48) )) , and T and V are independent.

Original: Corollary 3 Assume lim n,(cid:96) →∞ n − 2 (cid:96) = 0 and 0 < γ = lim n,(cid:96) →∞ n − 1 (cid:96) <
Revised: Corollary 3 Assume lim n,(cid:96) →∞ n − 2 (cid:96) = 0 and 0 < γ = lim n,(cid:96) →∞ n − 1 (cid:96) <

Original: ∞ .
Revised: ∞ .

Original: For sampling with replacement, incomplete U-statistics estimator of MMD is asymptotically normally distributed as 
Revised: For sampling with replacement, incomplete U-statistics estimator of MMD is asymptotically normally distributed as (cid:40) (cid:96) 12 MMD [Equation] inc [Equation]

Revised: N

Revised: [Equation]

Revised: [Equation]

Revised: ) , if p = q.

Revised: (cid:96)

Revised: [Equation]

Revised: MMD

Original: [Equation]
Revised: [Equation]

Revised: inc − MMD

Revised: [Equation]

Revised: D

Revised: [Equation]

Original:  where σ 2 = V ar ( h ( u , u (cid:48) )) and σ 2 u = 4( E u [( E u (cid:48){+
+}
Revised: N (0 , σ [Equation] + γσ 2 u ) , if p (cid:54) = q. where σ 2 = V ar ( h ( u , u (cid:48) )) and σ 2 u = {+
+}

Original:  [ h ( u , u (cid:48) )]] − E u , u(cid:48)
Revised: ( E u [( E u (cid:48) [ h ( u , u (cid:48) )]] − E u , u

Revised: [Equation]

Original: [ h ( u , u (cid:48) )]) 2 ) .
Revised: [ h ( u , u (cid:48) )]) 2 ) .

Original: Corollary 4 Assume lim n,(cid:96) →∞ n − 1 (cid:96) = 0 .
Revised: Corollary 4 Assume lim n,(cid:96) →∞ n − 1 (cid:96) = 0 .

Original: For sampling with replacement, the incomplete Ustatistics estimator of MMD is asymptotically normally distributed as
Revised: For sampling with replacement, the incomplete Ustatistics estimator of MMD is asymptotically normally distributed as

Original: [Equation]
Revised: [Equation]

Original: Thus, in practice, by setting (cid:96) (cid:28) n 2 , the incomplete estimator is asymptotically normal and therefore can be applied in PSI.
Revised: Thus, in practice, by setting (cid:96) (cid:28) n 2 , the incomplete estimator is asymptotically normal and therefore can be applied in PSI.

Original: More speciﬁcally, we can set (cid:96) =
Revised: More speciﬁcally, we can set (cid:96) =

Original: rn (cid:28) n 2 , where r is a small constant.
Revised: rn (cid:28) n 2 , where r is a small constant.

Original: In practice, we found that r = 10 works well in general.
Revised: In practice, we found that r = 10 works well in general.

Original: Figure 2 shows the empirical distribution under p =
Revised: Figure 2 shows the empirical distribution under p =

Original: q and p (cid:54) =
Revised: q and p (cid:54) =

Original: q for the complete estimator, the block estimator and the incomplete estimator.
Revised: q for the complete estimator, the block estimator and the incomplete estimator.

Original: As can be observed, the empirical distribution of the incomplete estimator is normal for small sampling parameter r , and becomes similar to its complete counterpart if r is large; this is supported by Theorem 2 ( γ = ∞ ).
Revised: As can be observed, the empirical distribution of the incomplete estimator is normal for small sampling parameter r , and becomes similar to its complete counterpart if r is large; this is supported by Theorem 2 ( γ = ∞ ).

Original: Finally, the following theorem assure the joint normality of the z vector in Theorem 1.
Revised: Finally, the following theorem assure the joint normality of the z vector in Theorem 1.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Theorem 5 Suppose 0 ≤ γ < ∞ .
Revised: Theorem 5 Suppose 0 ≤ γ < ∞ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: converges to multivariate normal distribution.
Revised: converges to multivariate normal distribution.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compared mmdInf with a naive testing baseline ( mmd ), which ﬁrst selects features using MMD and estimates corresponding p -values with the same data of feature selection without adjustment for the selection event.
Revised: We compared mmdInf with a naive testing baseline ( mmd ), which ﬁrst selects features using MMD and estimates corresponding p -values with the same data of feature selection without adjustment for the selection event.

Original: For mmdInf , we used the three MMD estimators: the linear-time MMD (Gretton et al., 2012), the block MMD (Zaremba et al., 2013), and the incomplete MMD.
Revised: For mmdInf , we used the three MMD estimators: the linear-time MMD (Gretton et al., 2012), the block MMD (Zaremba et al., 2013), and the incomplete MMD.

Original: We used 1 / 2 of data to calculate the covariance matrix of MMD and the rest to perform feature selection and inference.
Revised: We used 1 / 2 of data to calculate the covariance matrix of MMD and the rest to perform feature selection and inference.

Original: We ﬁxed the number of selected features (prior to PSI) k to 30 .
Revised: We ﬁxed the number of selected features (prior to PSI) k to 30 .

Original: In PSI the signiﬁcance of each of the 30 selected features (from ranking MMD) is computed and features with p -value lower than the signiﬁcance level α = 0 .
Revised: In PSI the signiﬁcance of each of the 30 selected features (from ranking MMD) is computed and features with p -value lower than the signiﬁcance level α = 0 .

Original: 05 are selected as statistically signiﬁcant features .
Revised: 05 are selected as statistically signiﬁcant features .

Original: For block MMD, in each experiment we set the candidate of block size as B = { 10 , 20 , 50 } .
Revised: For block MMD, in each experiment we set the candidate of block size as B = { 10 , 20 , 50 } .

Original: For incomplete MMD, in each experiment the ratio between number of pairs ( i, j ) sampled to compute incomplete MMD score and sample size is ﬁxed at r = (cid:96) n ∈ { 0 . 5 , 5 , 10 } .
Revised: For incomplete MMD, in each experiment the ratio between number of pairs ( i, j ) sampled to compute incomplete MMD score and sample size is ﬁxed at r = (cid:96) n ∈ { 0 . 5 , 5 , 10 } .

Original: We reported the true positive rate (TPR) k (cid:48)
Revised: We reported the true positive rate (TPR) k (cid:48)

Original: k ∗ where k (cid:48) is the number of true features selected by mmdInf or mmd and k ∗ is the total number of true features in synthetic data.
Revised: k ∗ where k (cid:48) is the number of true features selected by mmdInf or mmd and k ∗ is the total number of true features in synthetic data.

Original: We further computed the false positive rate (FPR) k (cid:48)(cid:48)
Revised: We further computed the false positive rate (FPR) k (cid:48)(cid:48)

Original: k − k ∗ where k (cid:48)(cid:48) is the number of non-true features reported as positives.
Revised: k − k ∗ where k (cid:48)(cid:48) is the number of non-true features reported as positives.

Original: We ran all experiments over 200 times, and reported the average TPR and FPR.
Revised: We ran all experiments over 200 times, and reported the average TPR and FPR.

Original: 5. S YNTHETIC EXPERIMENTS (PSI)
Revised: 5. S YNTHETIC EXPERIMENTS (PSI)

Original: The number of features d is ﬁxed to 50 , and for each feature, data is randomly generated following a
Revised: The number of features d is ﬁxed to 50 , and for each feature, data is randomly generated following a

Original: Gaussian distribution with set mean and variance.
Revised: Gaussian distribution with set mean and variance.

Original: 10 out of the 50 features are set to be signiﬁcantly different by shifting the distribution of one class away from the other (mean or variance).
Revised: 10 out of the 50 features are set to be signiﬁcantly different by shifting the distribution of one class away from the other (mean or variance).

Original: More speciﬁcally, we generate the synthetic data as
Revised: More speciﬁcally, we generate the synthetic data as

Original: (a) Mean shift x ∼ N ( 0 50 , I 50 ) , y ∼ N (
Revised: (a) Mean shift x ∼ N ( 0 50 , I 50 ) , y ∼ N (

Original: [Equation]
Revised: [Equation]

Original: , I )
Revised: , I )

Original: [Equation]
Revised: [Equation]

Original: (b) Variance shift x ∼ N ( 0 50 , I 50 ) , y ∼
Revised: (b) Variance shift x ∼ N ( 0 50 , I 50 ) , y ∼

Original: [Equation]
Revised: [Equation]

Original: ( 0 , Σ )
Revised: ( 0 , Σ )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where N ( µ , Σ ) is a multivariate normal distribution with mean
Revised: where N ( µ , Σ ) is a multivariate normal distribution with mean

Original: µ ∈ R d and covariance Σ ∈ R d × d , 1 p ∈ R
Revised: µ ∈ R d and covariance Σ ∈ R d × d , 1 p ∈ R

Original: d is a vector whose elements are all one, 0 d ∈ R d is a vector whose elements are all zero, and diag (
Revised: d is a vector whose elements are all one, 0 d ∈ R d is a vector whose elements are all zero, and diag (

Original: a ) ∈ R
Revised: a ) ∈ R

Original: d × d is a diagonal matrix whose diagonal elements are a ∈ R d .
Revised: d × d is a diagonal matrix whose diagonal elements are a ∈ R d .

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Figure 3(a) and (b) show the FPRs of linear MMD, block MMD and incomplete MMD with or without PSI.
Revised: Figure 3(a) and (b) show the FPRs of linear MMD, block MMD and incomplete MMD with or without PSI.

Original: As can be clearly seen, PSI successfully controls FPR with signiﬁcance level α = 0 .
Revised: As can be clearly seen, PSI successfully controls FPR with signiﬁcance level α = 0 .

Original: for all the three estimators, whereas the naive approach tends to have higher FPRs.
Revised: for all the three estimators, whereas the naive approach tends to have higher FPRs.

Original: Figuresshows the TPRs of the synthetic data.
Revised: Figuresshows the TPRs of the synthetic data.

Original: In both cases, the TPR of incomplete MMD converges tosigniﬁcantly faster than the the other two empirical estimates.
Revised: In both cases, the TPR of incomplete MMD converges tosigniﬁcantly faster than the the other two empirical estimates.

Original: 5. R EAL - WORLD DATA (B ENCHMARK )
Revised: 5. R EAL - WORLD DATA (B ENCHMARK )

Original: We compared the proposed algorithm by using real-world datasets.
Revised: We compared the proposed algorithm by using real-world datasets.

Original: Since it is difﬁcult to decide what is a ”true feature” in real-world data, we choose a few datasets for binary classiﬁcation with small amount of features, and regard all the original features as true .
Revised: Since it is difﬁcult to decide what is a ”true feature” in real-world data, we choose a few datasets for binary classiﬁcation with small amount of features, and regard all the original features as true .

Original: We then concatenated random features to the true features (the total number of features d = 100 ).
Revised: We then concatenated random features to the true features (the total number of features d = 100 ).

Original: Table 1 shows TPRs and
Revised: Table 1 shows TPRs and

Original: FPRs of mmdInf with different MMD estimators.
Revised: FPRs of mmdInf with different MMD estimators.

Original: It can be observed that the incomplete estimator signiﬁcantly outperforms the other empirical estimates.
Revised: It can be observed that the incomplete estimator signiﬁcantly outperforms the other empirical estimates.

Original: Note that a higher TPR can be achieved with higher r , while the FPR is still controlled at 0.05 with r = 5 .
Revised: Note that a higher TPR can be achieved with higher r , while the FPR is still controlled at 0.05 with r = 5 .

Original: Additional comparison of different empirical estimates on real dataset (with varying sample size) can be found in Figures 8.
Revised: Additional comparison of different empirical estimates on real dataset (with varying sample size) can be found in Figures 8.

Original: 5. GAN S ANALYSIS (S AMPLE SELECTION )
Revised: 5. GAN S ANALYSIS (S AMPLE SELECTION )

Original: We also applied mmdInf for evaluating the generation quality of GANs.
Revised: We also applied mmdInf for evaluating the generation quality of GANs.

Original: We trained BEGAN (Berthelot et al., 2017), DCGAN (Radford et al., 2015), STDGAN (Miyato et al., 2017), Cramer
Revised: We trained BEGAN (Berthelot et al., 2017), DCGAN (Radford et al., 2015), STDGAN (Miyato et al., 2017), Cramer

Original: GAN (Bellemare et al., 2017), DFM (Warde-Farley & Bengio, 2016), DRAGAN (Kodali et al., 2017), Minibatch Discrimination GAN (Salimans et al., 2016), and WGAN-GP (Gulrajani et al., 2017), generated 5000 images (using Chainer GAN package 1 with CIFAR10 datasets), and extracted 512 dimensional features by pre-trained Resnet18 (He et al., 2016).
Revised: GAN (Bellemare et al., 2017), DFM (Warde-Farley & Bengio, 2016), DRAGAN (Kodali et al., 2017), Minibatch Discrimination GAN (Salimans et al., 2016), and WGAN-GP (Gulrajani et al., 2017), generated 5000 images (using Chainer GAN package 1 with CIFAR10 datasets), and extracted 512 dimensional features by pre-trained Resnet18 (He et al., 2016).

Original: For the true image sets, we subsampled 5000 images from CIFAR10 datasets and computed the 512 dimensional features using the same Resnet18.
Revised: For the true image sets, we subsampled 5000 images from CIFAR10 datasets and computed the 512 dimensional features using the same Resnet18.

Original: We then tested the difference between the generated images and the real images using mmdInf on extracted features (see Sec. 3.4).
Revised: We then tested the difference between the generated images and the real images using mmdInf on extracted features (see Sec. 3.4).

Original: We found that for all the members in the GAN family, the null hypothesis was rejected, i.e., the generated distribution and the real distribution are different.
Revised: We found that for all the members in the GAN family, the null hypothesis was rejected, i.e., the generated distribution and the real distribution are different.

Original: This result is consistent with the ﬁndings in (Sutherland et al., 2016), which demonstrate that optimized MMD has perfect discriminative power in GANs evaluation.
Revised: This result is consistent with the ﬁndings in (Sutherland et al., 2016), which demonstrate that optimized MMD has perfect discriminative power in GANs evaluation.

Original: As sanity check, we evaluated mmdInf by constructing an ”oracle” generative model that generates real images from CIFAR10.
Revised: As sanity check, we evaluated mmdInf by constructing an ”oracle” generative model that generates real images from CIFAR10.

Original: Next, we randomly selected 5000 images (a disjoint set from the oracle generative images) from CIFAR10 in each trial, and set the sampling ratio to r = 5 .
Revised: Next, we randomly selected 5000 images (a disjoint set from the oracle generative images) from CIFAR10 in each trial, and set the sampling ratio to r = 5 .

Original: Figure 4(a) shows the distribution of p -values computed by our algorithm.
Revised: Figure 4(a) shows the distribution of p -values computed by our algorithm.

Original: We can see that the p -values are distributed uniformly in the tests for the ”oracle” generative model, which matches the theoretical result in Theorem 1.
Revised: We can see that the p -values are distributed uniformly in the tests for the ”oracle” generative model, which matches the theoretical result in Theorem 1.

Original: Thus the algorithm is able to detect the distribution difference and control the false positive rate.
Revised: Thus the algorithm is able to detect the distribution difference and control the false positive rate.

Original: In other words, if the generated samples do not follow the original distribution, we can safely reject the null hypothesis with a given signiﬁcance level α .
Revised: In other words, if the generated samples do not follow the original distribution, we can safely reject the null hypothesis with a given signiﬁcance level α .

Original: Figure 4(b) shows the estimated MMD scores of each model.
Revised: Figure 4(b) shows the estimated MMD scores of each model.

Original: Based on the results, we could tell that
Revised: Based on the results, we could tell that

Original: DFM was the best model and DCGAN was the second best model to approximate the true distribution.
Revised: DFM was the best model and DCGAN was the second best model to approximate the true distribution.

Original: However, the difference between various members is not obvious.
Revised: However, the difference between various members is not obvious.

Original: Developing a validation pipeline based on mmdInf for GANs analysis would be one interesting line of future work.
Revised: Developing a validation pipeline based on mmdInf for GANs analysis would be one interesting line of future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a novel statistical testing framework mmdInf , which can ﬁnd a set of statistically signiﬁcant features that can discriminate two distributions.
Revised: In this paper, we proposed a novel statistical testing framework mmdInf , which can ﬁnd a set of statistically signiﬁcant features that can discriminate two distributions.

Original: Through synthetic and realworld experiments, we demonstrated that mmdInf can successfully ﬁnd important features and/or datasets.
Revised: Through synthetic and realworld experiments, we demonstrated that mmdInf can successfully ﬁnd important features and/or datasets.

Original: We also proposed a method for sample selection based on mmdInf and applied it in the evaluation of generative models.
Revised: We also proposed a method for sample selection based on mmdInf and applied it in the evaluation of generative models.

Original:  A CKNOWLEDGEMENT
Revised:  A CKNOWLEDGEMENT

Original: We would like to thank Dr. Kazuki Yoshizoe, Prof. Yuta Umezu and Prof. Suriya Gunasekar for fruitful discussions and valuable suggestions.
Revised: We would like to thank Dr. Kazuki Yoshizoe, Prof. Yuta Umezu and Prof. Suriya Gunasekar for fruitful discussions and valuable suggestions.

Original: MY was supported by the JST PRESTO program
Revised: MY was supported by the JST PRESTO program

Original: JPMJPR165A and partly supported by MEXT KAKENHI 16K16114.
Revised: JPMJPR165A and partly supported by MEXT KAKENHI 16K16114.

Original: IT was partially supported by
Revised: IT was partially supported by

Original: MEXT KAKENHI (17H00758, 16H06538), JST CREST (JPMJCR1302, JPMJCR1502), RIKEN
Revised: MEXT KAKENHI (17H00758, 16H06538), JST CREST (JPMJCR1302, JPMJCR1502), RIKEN

Original: Center for Advanced Intelligence Project and JST support program for starting up innovation-hub on materials research by information integration initiative.
Revised: Center for Advanced Intelligence Project and JST support program for starting up innovation-hub on materials research by information integration initiative.

Original: KF was supported partly by JSPS KAKENHI 18K19793.
Revised: KF was supported partly by JSPS KAKENHI 18K19793.

Original: S UPPLEMENTARY MATERIALS T HEORETICAL A NALYSIS OF I NCOMPLETE MMD
Revised: S UPPLEMENTARY MATERIALS T HEORETICAL A NALYSIS OF I NCOMPLETE MMD

Original: We investigate the theoretical properties of the incomplete MMD estimator under the random design with replacement.
Revised: We investigate the theoretical properties of the incomplete MMD estimator under the random design with replacement.

Original: For simplicity, we denote MMD 2 inc [ F , X , Y ] =
Revised: For simplicity, we denote MMD 2 inc [ F , X , Y ] =

Original: MMD 2 inc and MMD 2 [ F , p, q ] =
Revised: MMD 2 inc and MMD 2 [ F , p, q ] =

Original: MMD 2 , respectively.
Revised: MMD 2 , respectively.

Original: We introduce the conditional expectations for the U-statistic kernel h ( u , u (cid:48) )
Revised: We introduce the conditional expectations for the U-statistic kernel h ( u , u (cid:48) )

Original: =
Revised: =

Original: K ( x , x (cid:48) )
Revised: K ( x , x (cid:48) )

Original: + K ( y , y (cid:48) )
Revised: + K ( y , y (cid:48) )

Original: −
Revised: −

Original: K (
Revised: K (

Original: [Equation]
Revised: [Equation]

Original: , y (cid:48)
Revised: , y (cid:48)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, we deﬁne c be the smallest integer such that h c (cid:54) =
Revised: Then, we deﬁne c be the smallest integer such that h c (cid:54) =

Original: MMD 2 .
Revised: MMD 2 .

Original: For p = q , since the U-statistics is degenerated:
Revised: For p = q , since the U-statistics is degenerated:

Original: [Equation]
Revised: [Equation]

Original: and MMD 2 = 0 , we have c = 2 .
Revised: and MMD 2 = 0 , we have c = 2 .

Original: On the other hand, since h 1 (cid:54) = MMD 2 for p (cid:54) =
Revised: On the other hand, since h 1 (cid:54) = MMD 2 for p (cid:54) =

Original: q , we have c = 1 .
Revised: q , we have c = 1 .

Original: Theorem 2 Let c be the smallest integer such that h c (cid:54) =
Revised: Theorem 2 Let c be the smallest integer such that h c (cid:54) =

Original: MMD 2 , and let n and (cid:96) tend to inﬁnity such that γ = lim n,(cid:96) →∞ n − c (cid:96), 0 ≤ γ ≤ ∞ .
Revised: MMD 2 , and let n and (cid:96) tend to inﬁnity such that γ = lim n,(cid:96) →∞ n − c (cid:96), 0 ≤ γ ≤ ∞ .

Original: For sampling with replacement, we have
Revised: For sampling with replacement, we have

Original: [Equation]
Revised: [Equation]

Original: where V is the random variable of the limit distribution of n c 2 ( MMD 2 u − MMD 2 ) , T is the random variable of N (0 , σ 2 ) , σ 2 = V ar ( h ( u , u (cid:48) )) , and T and V are independent.
Revised: where V is the random variable of the limit distribution of n c 2 ( MMD 2 u − MMD 2 ) , T is the random variable of N (0 , σ 2 ) , σ 2 = V ar ( h ( u , u (cid:48) )) , and T and V are independent.

Original: Proof: Use Corollary 1 of Janson (1984) (or Theorem 1 of Lee (1990), pp. 200).
Revised: Proof: Use Corollary 1 of Janson (1984) (or Theorem 1 of Lee (1990), pp. 200).

Original: In MMD, c =for p = q and c = 1 for p (cid:54) = q .
Revised: In MMD, c =for p = q and c = 1 for p (cid:54) = q .

Original: [Equation]
Revised: [Equation]

Original: Corollary 3 Assume lim n,
Revised: Corollary 3 Assume lim n,

Original: [Equation]
Revised: [Equation]

Original: →∞ n − 2 (cid:96) = 0 and 0 < γ = lim
Revised: →∞ n − 2 (cid:96) = 0 and 0 < γ = lim

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n −
Revised: n −

Original: [Equation]
Revised: [Equation]

Original: < ∞ .
Revised: < ∞ .

Original: For sampling with replacement, the incomplete U-statistics estimator of MMD is asymptotically normally distributed as
Revised: For sampling with replacement, the incomplete U-statistics estimator of MMD is asymptotically normally distributed as

Original: [Equation]
Revised: [Equation]

Original: where σ 2 = V ar ( h ( u , u (cid:48) )) and σ 2 u = 4( E u [( E u (cid:48)
Revised: where σ 2 = V ar ( h ( u , u (cid:48) )) and σ 2 u = 4( E u [( E u (cid:48)

Original: [ h ( u , u (cid:48) )]] − E u , u (cid:48)
Revised: [ h ( u , u (cid:48) )]] − E u , u (cid:48)

Original: [ h ( u , u (cid:48) )]) 2 ) .
Revised: [ h ( u , u (cid:48) )]) 2 ) .

Original: Proof: Under p = q ( c = 2 ), since lim n,(cid:96) →∞ n − 2 (cid:96) = 0 and MMD 2 = 0 , we can immediately obtain the limit distribution by Theorem 2.
Revised: Proof: Under p = q ( c = 2 ), since lim n,(cid:96) →∞ n − 2 (cid:96) = 0 and MMD 2 = 0 , we can immediately obtain the limit distribution by Theorem 2.

Original: Under p (cid:54) =
Revised: Under p (cid:54) =

Original: q ( c = 1 ), MMD u converges in distribution to a Gaussian according to (Gretton et al., 2007)
Revised: q ( c = 1 ), MMD u converges in distribution to a Gaussian according to (Gretton et al., 2007)

Original: [Equation]
Revised: [Equation]

Original: where σ 2 u = 

Revised: where σ 2 u =

Original: ( E u [( E u (cid:48)
Revised: ( E u [( E u (cid:48)

Original: [ h ( u , u (cid:48) )]] − E u , u (cid:48)
Revised: [ h ( u , u (cid:48) )]] − E u , u (cid:48)

Original: [ h ( u , u (cid:48) )]) 2 ) .
Revised: [ h ( u , u (cid:48) )]) 2 ) .

Original: Based on Theorem 2, under the given assumption, we can obtain the distribution of MMD 2 inc since T and V are independent.
Revised: Based on Theorem 2, under the given assumption, we can obtain the distribution of MMD 2 inc since T and V are independent.

Original: [Equation]
Revised: [Equation]

Original: Corollary 4 Assume lim n,(cid:96) →∞ n − 1 (cid:96) = 0 .
Revised: Corollary 4 Assume lim n,(cid:96) →∞ n − 1 (cid:96) = 0 .

Original: For sampling with replacement, the incomplete Ustatistics estimator of MMD is asymptotically normally distributed as
Revised: For sampling with replacement, the incomplete Ustatistics estimator of MMD is asymptotically normally distributed as

Original: [Equation]
Revised: [Equation]

Original: Proof: Since lim n,(cid:96) →∞ n − 1 (cid:96) = 0 and lim n,(cid:96) →∞ n − 2 (cid:96) = 0 , the limit distribution of (cid:96) 1 / 2 ( MMD 2 inc − MMD 2 ) is N (0 , σ 2 ) based on Theorem 2.
Revised: Proof: Since lim n,(cid:96) →∞ n − 1 (cid:96) = 0 and lim n,(cid:96) →∞ n − 2 (cid:96) = 0 , the limit distribution of (cid:96) 1 / 2 ( MMD 2 inc − MMD 2 ) is N (0 , σ 2 ) based on Theorem 2.

Original: (cid:3)
Revised: (cid:3)

Original: Thus, in practice, by setting (cid:96) (cid:28) n 2 , the incomplete estimator is asymptotically normal and therefore can be applied in PSI.
Revised: Thus, in practice, by setting (cid:96) (cid:28) n 2 , the incomplete estimator is asymptotically normal and therefore can be applied in PSI.

Original: More speciﬁcally, we can set (cid:96) =
Revised: More speciﬁcally, we can set (cid:96) =

Original: rn (cid:28) n 2 , where r is a small constant.
Revised: rn (cid:28) n 2 , where r is a small constant.

Original: In practice, we found that r = 10 works well in general.
Revised: In practice, we found that r = 10 works well in general.

Original: Theorem 5 Suppose 0 ≤ γ < ∞ .
Revised: Theorem 5 Suppose 0 ≤ γ < ∞ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: converges to multivariate normal distribution.
Revised: converges to multivariate normal distribution.

Original: Proof: We use the fact that convergence in distribution of multivariate random variables results in convergence in distribution of univariate random variables.
Revised: Proof: We use the fact that convergence in distribution of multivariate random variables results in convergence in distribution of univariate random variables.

Original: From Theorem29.4 in Billingsley (2008)(Cram´er-Wold device), it is sufﬁcient to prove that for any η = [ η 1 , . . .
Revised: From Theorem29.4 in Billingsley (2008)(Cram´er-Wold device), it is sufﬁcient to prove that for any η = [ η 1 , . . .

Original: , η d ] (cid:62) ∈ R d , (cid:80) ds =1 η s MMD 2 inc [ F , X ( s ) , Y ( s ) ]
Revised: , η d ] (cid:62) ∈ R d , (cid:80) ds =1 η s MMD 2 inc [ F , X ( s ) , Y ( s ) ]

Original: D −→ (cid:80) ds =

Revised: D −→ (cid:80) ds =

Original: η s Z s , where Z s , s = 1 , 2 , . . .
Revised: η s Z s , where Z s , s = 1 , 2 , . . .

Original: , d are normal distributions.
Revised: , d are normal distributions.

Original: Since each incomplete U-statistic MMD 2 inc [ F , X ( s ) , Y ( s ) ] converges to normal distribution derived in Corollary 3 when 0 ≤ γ < ∞ , and from the continuous mapping theorem for g ( x ) =
Revised: Since each incomplete U-statistic MMD 2 inc [ F , X ( s ) , Y ( s ) ] converges to normal distribution derived in Corollary 3 when 0 ≤ γ < ∞ , and from the continuous mapping theorem for g ( x ) =

Original: η (cid:62) x , we obtain the desired result.
Revised: η (cid:62) x , we obtain the desired result.

Original: I LLUSTRATIVE EXPERIMENTS
Revised: I LLUSTRATIVE EXPERIMENTS

Original: Figure 5 shows the empirical distribution under p =
Revised: Figure 5 shows the empirical distribution under p =

Original: q and p (cid:54) =
Revised: q and p (cid:54) =

Original: q for the complete estimator, the block estimator and the incomplete estimator.
Revised: q for the complete estimator, the block estimator and the incomplete estimator.

Original: As can be observed, the empirical distribution of the incomplete estimator is normal for small sampling parameter r , and becomes similar to its complete counterpart if r is large; this is supported by Theorem 2 ( γ = ∞ ).
Revised: As can be observed, the empirical distribution of the incomplete estimator is normal for small sampling parameter r , and becomes similar to its complete counterpart if r is large; this is supported by Theorem 2 ( γ = ∞ ).

Original: Moreover, compared to the block estimator, the incomplete estimator tends to have a better trade-off between variance and normality.
Revised: Moreover, compared to the block estimator, the incomplete estimator tends to have a better trade-off between variance and normality.

Original: Figure 6 shows the MSEs between the true and the estimated covariance matrices of the incomplete estimate and the block estimate, respectively.
Revised: Figure 6 shows the MSEs between the true and the estimated covariance matrices of the incomplete estimate and the block estimate, respectively.

Original: As can be seen, the error of the incomplete estimate is smaller than that of the block estimate, which is an advantage of the incomplete MMD over the block MMD in PSI.
Revised: As can be seen, the error of the incomplete estimate is smaller than that of the block estimate, which is an advantage of the incomplete MMD over the block MMD in PSI.

Original: Moreover, for the incomplete U-statistics estimator, we have (cid:96) =
Revised: Moreover, for the incomplete U-statistics estimator, we have (cid:96) =

Original: rn samples to estimate the covariance matrix, while we have only n/B samples for the block estimate.
Revised: rn samples to estimate the covariance matrix, while we have only n/B samples for the block estimate.

Original: That is, the number of samples that can be used for covariance estimation of the incomplete U-statistics estimator is rB times larger than that of the block estimator.
Revised: That is, the number of samples that can be used for covariance estimation of the incomplete U-statistics estimator is rB times larger than that of the block estimator.

Original: For example, for n = 1000 , we can use 5000 samples for the incomplete U-statistics, while we have only 50 samples for the block estimate.
Revised: For example, for n = 1000 , we can use 5000 samples for the incomplete U-statistics, while we have only 50 samples for the block estimate.

Original: This is problematic for high-dimensional case (e.g., d = 100 ), and the estimated covariance matrix is not full rank.
Revised: This is problematic for high-dimensional case (e.g., d = 100 ), and the estimated covariance matrix is not full rank.

Original: To alleviate this issue, we employ the POET algorithm for the block estimator.
Revised: To alleviate this issue, we employ the POET algorithm for the block estimator.

Original: Figure 7(a) shows the Type II error comparison for two-sample test with one dimensional Gaussian mean shift data.
Revised: Figure 7(a) shows the Type II error comparison for two-sample test with one dimensional Gaussian mean shift data.

Original: The Type II error is computed when the Type I error is ﬁxed at 0.05, and the incomplete MMD outperforms other estimators.
Revised: The Type II error is computed when the Type I error is ﬁxed at 0.05, and the incomplete MMD outperforms other estimators.

Original: Figure 7 (b) compares the computational time of the empirical estimates, and for small r the computational time of incomplete MMD is much less than that of the block MMD.
Revised: Figure 7 (b) compares the computational time of the empirical estimates, and for small r the computational time of incomplete MMD is much less than that of the block MMD.

Original: Overall, the incomplete MMD has favorable properties in practice.
Revised: Overall, the incomplete MMD has favorable properties in practice.

Original: Figure 8 show the true positive rate (TPR) and the false positive rate (FPR) of the wine (white) dataset experiment.
Revised: Figure 8 show the true positive rate (TPR) and the false positive rate (FPR) of the wine (white) dataset experiment.

Original: Here, we change the number of samples as 200 , 400 , . . .
Revised: Here, we change the number of samples as 200 , 400 , . . .

Original: As can be seen, both PSI algorithms can control the FPR with 2000 samples.
Revised: As can be seen, both PSI algorithms can control the FPR with 2000 samples.

Original: Moreover, it is clear that the incomplete estimator can get better performance than the block estimator in both TPR and FPR with smaller number of samples.
Revised: Moreover, it is clear that the incomplete estimator can get better performance than the block estimator in both TPR and FPR with smaller number of samples.

Original: E FFECTIVENESS OF PSI
Revised: E FFECTIVENESS OF PSI

Original: Here, we compared the PSI and non-PSI counterpart on benchmark data.
Revised: Here, we compared the PSI and non-PSI counterpart on benchmark data.

Original: As clearly seen, if we do not use the PSI, we cannot control FPR, while the proposed algorithm can successfully control FPR values.
Revised: As clearly seen, if we do not use the PSI, we cannot control FPR, while the proposed algorithm can successfully control FPR values.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]