Original: Using Perturbation to Improve Goodness-of-Fit Tests based on Kernelized Stein Discrepancy
Revised: Using Perturbation to Improve Goodness-of-Fit Tests based on Kernelized Stein Discrepancy

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely employedin goodness-of-fit tests.
Revised: Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely employed in goodness-of-ﬁt tests.

Original: It can be used even when the target distribution has an unknown normalising factor, such as in Bayesian analysis.
Revised: It is applicable even when the target distribution has an unknown normalising factor, such as in Bayesian analysis.

Original: We show theoretically andempirically that the power of the KSD test can be low when the target distributionhas well-separated modes, which is due to insufficient data in regions where thescore functions of the alternative and the target distributions differ the most.
Revised: We show theoretically and empirically that the power of the KSD test can be low when the target distribution has well-separated modes, which is due to insufﬁcient data in regions where the score functions of the alternative and the target distributions differ the most.

Original: Toimprove its test power, we propose to perturb the target and alternative distributionsbefore applying the KSD test.
Revised: To improve its test power, we propose to perturb the target and alternative distributions before applying the KSD test.

Original: The perturbation uses a Markov transition kernelthat leaves the target invariant but perturbs alternatives.
Revised: The perturbation uses a Markov transition kernel that leaves the target invariant but perturbs alternatives.

Original: We provide numericalevidence that the proposed approach can lead to a substantially higher power thanthe KSD test when the target and the alternative are mixture distributions that differonly in mixing weights.
Revised: We provide numerical evidence that the proposed approach can lead to a substantially higher power than the KSD test when the target and the alternative are mixture distributions that differ only in mixing weights.

Original:  Introduction
Revised:  Introduction

Original: Goodness-of-fit (GOF) testing concerns the question: given a target distribution P and a finite sampledrawn from a alternative distribution Q , is there evidence against the null hypothesis H 0 : Q = P ?
Revised: Goodness-of-ﬁt (GOF) testing concerns the question: given a target distribution P and a ﬁnite sample drawn from a alternative distribution Q , is there evidence against the null hypothesis H 0 : Q = P ?

Original: A popular test statistic for GOF tests is kernelized Stein discrepancy (KSD), which is a score-basedstatistical discrepancy and is applicable even if the model is unnormalised (Chwialkowski et al.
Revised: A popular test statistic for GOF tests is kernelized Stein discrepancy (KSD), which is a score-based statistical divergence and is applicable even if the model is unnormalised (Chwialkowski et al., 2016;

Original: ,2016; Liu et al., 2016).
Revised: Liu et al., 2016).

Original: However, KSD can fail to detect discrepancies when the target probability measure has well-separatedmodes.
Revised: However, KSD can fail to detect discrepancies when the target distribution has well-separated modes.

Original: For example, when Q and P are mixtures with the same components but  different mixingproportions, the KSD statistic based on a finite sample can be numerically close to 0.
Revised: For example, when Q and P are mixtures with the same components but with different mixing proportions, it has been observed that the KSD test would suffer from a low test power.

Original: This is knownas the blindness  to isolated components in Wenliang and Kanagawa (2020), which study this issuefor score-based methods in a general setting.
Revised: This is a consequence of the blindness of KSD to isolated components, which is a well-known problem for KSD and other score-based applications, such as density estimation (Wenliang et al., 2019; Zhang et al., 2022) and quality measures for MCMC (Gorham et al., 2019).

Original: However, a formal treatment ofits implication on GOFtests based on KSD remains unavailable.
Revised: Neverthless, how the blindness of KSD manifests itself in GOF tests is yet to be formalised.

Original: The contribution of this paper is twofold.
Revised: The contribution of this paper is twofold.

Original: First, we show theoretically and numerically with abimodal Gaussian example that the power of KSD tests can converge to the test level when the targetdistribution has well-separated modes (Fig. 1a).
Revised: First, we show theoretically and numerically on a bimodal Gaussian example that the power of KSD tests can converge to the test level as the mode separation increases (Fig. 1a).

Original: This is different from the works of Wenliang and 
Revised: This is different from the analyses in Wenliang and Kanagawa (2020) and

Original: Kanagawa (2020) and Gorham and Mackey (2017), which focus on the convergence of the sample KSD statistic but not its limiting null distribution.
Revised: Gorham and Mackey (2017), which focus on the convergence of the sample KSD statistic but not the test power.

Original: Second, we address this issue by introducing aperturbation operator .
Revised: Second, we address this issue by introducing a perturbation operator .

Original: Therole of this operator is  to perturb the alternative and the target distributionssimultaneously so thatthe discrepancy between the perturbed ones can be more easily detected by KSD.
Revised: The operator is designed to be a Markov transition kernel that leaves the target invariant but perturbs alternatives to create discrepancies that can be more easily detected by KSD.

Original: We propose to use as the perturbation operator a Markov transition kernel that is invariant tothe target P , which ensures the resulting GOF test provably controls Type-I error. The transitionkernel is non-irreducible and relies on an inter-modal jump proposal, which can increase the testpower against multi-modal targets, sometimes substantially from the nominal level to almost 1.
Revised: We discuss how to perform GOF tests with the proposed discrepancy, and show numerically that it can increase the test power against multi-modal targets, sometimes substantially from the nominal level to almost 1.

Original: [Figure]
Revised: [Figure]

Original:  Background
Revised:  Background

Original: Given two Borel probability measuresQ and P supported on X ⊂ R d , the Stein discrepancy (SD)(Stein, 1972; Gorham and Mackey, 2015) between Q and P is defined to be
Revised: Kernelized Stein discrepancy Denote by Q and P two Borel probability measures supported on X = R d . We assume P admits a positive, continuously differentiable density p with respect to the

Revised: Lebesgue measure.

Revised: A statistical divergence that measures how well samples from Q agree with P is the Stein discrepancy (SD) (Stein, 1972; Gorham and Mackey, 2015)

Original: [Equation]
Revised: [Equation]

Original:  where F is a set of functions on X  and A P is an operator acting on F .
Revised: (1) where F is a set of functions on X , and A P is an operator acting on F .

Original: We assume X = R d and P admits a positive, continuously differentiable density p with respect to the Lebesgue measure, inwhich case the Langevin-Stein operator is a natural candidate for A p f ( x ) :
Revised: A natural candidate for A p f ( x ) :

Original: = ⟨∇ log p ( x ) , f ( x ) ⟩ +⟨∇ , f ( x ) ⟩ , where f : R d → R d is a continuously differentiable, vector-valued function.
Revised: = ⟨∇ log p ( x ) , f ( x ) ⟩ + ⟨∇ , f ( x ) ⟩ , where f : R d → R d is a continuously differentiable, vector-valued function.

Original: Kernelized Stein discrepancy Liu et al.

Original: Chwialkowski et al.

Original: (2016) proposed torestrict Fto the unit ball of a reproducing kernel Hilbert space (RKHS) (Berlinet and Thomas-Agnan, 2011) toensure S ( Q, P ; F ) is a statistical discrepancy, i.e., D ( Q, P ; F ) = 

Revised: When restricting F to the unit ball of a reproducing kernel Hilbert space (RKHS) (Berlinet and Thomas-Agnan, 2011), S ( Q, P ; F ) is a statistical discrepancy, i.e., D ( Q, P ; F ) =

Original: ⇐⇒ Q = P , and (1) can besolved efficiently.
Revised: ⇐⇒ Q = P , and (1) can be solved efﬁciently (Liu et al., 2016;

Revised: Chwialkowski et al., 2016).

Original: Let H be an RKHS associated with positive definite kernel k : X × X → R , andLet F dbe the unit ball of the d -times Cartesian product H d := H×· · · H .
Revised: Formally, let H be an RKHS associated with positive deﬁnite kernel k : X × X → R , and F d the unit ball of the d -times Cartesian product H d := H × · · · H .

Original: Choosing F = F d and theoperator A P yields the (Langevin) kernelized Stein discrepancy (KSD): D ( Q, P ) :
Revised: Choosing F = F d and the operator A P yields the (Langevin) kernelized Stein discrepancy (KSD): D ( Q, P ) :

Original: = S ( Q, P ; F d ) .
Revised: = S ( Q, P ; F d ) .

Original: Assuming the kernel k has continuous first-order derivatives with respect to both arguments, KSDattains a closed form: D ( Q, P ) =
Revised: Assuming the kernel k has continuous ﬁrst-order derivatives with respect to both arguments, KSD attains a closed form: D ( Q, P ) =

Original: E x,x ′ ∼ Q [ u P ( x, x ′ )] , where x, x ′ ∼ Q are independent randomvariables, and u P is the Stein kernel : u P ( x, x ′ ) :
Revised: E x,x ′ ∼ Q [ u P ( x, x ′ )] , where x, x ′ ∼ Q are independent random variables, and u P is the Stein kernel : u P ( x, x ′ ) :

Original: = s p ( x ) ⊤ k ( x, x ′ )
Revised: = s p ( x ) ⊤ k ( x, x ′ )

Original: s p ( x ′ )
Revised: s p ( x ′ )

Original: + s p ( x ) ⊤ ∇ x ′ k
Revised: + s p ( x ) ⊤ ∇ x ′ k

Original: ( x, x ′ )
Revised: ( x, x ′ )

Original: +∇ x k ( x, x′ )
Revised: + ∇ [Equation] x, x

Original: ⊤ s p ( x ′ ) + (cid:80) di = ∂ 2 ∂x
Revised: [Equation]

Original: i ∂x ′ i k 
Revised: k [Equation]

Original: ( x, x ′ )
Revised: x, x [Equation]

Original:  (Chwialkowski et al., 2016, Thm. 2.1).
Revised: ) (Chwialkowski et al., 2016, Thm. 2.1).

Original: Notably, u P(hence also D ( Q, P ) ) depends on p only through s p ( x ) = ∇ log p ( x ) , so KSD is computable evenwithout the knowledge of the (possibly intractable) normalising constant of p .
Revised: Notably, u P (hence also D ( Q, P ) ) depends on p only through s p ( x ) = ∇ log p ( x ) , so KSD is computable even without the knowledge of the (possibly intractable) normalising constant of p .

Revised: Given a ﬁnite sample { x i } ni =1 from Q , a consistent and unbiased estimator for D ( Q, P ) is the U-statistic ˆ D P := 1 n ( n − 1)

Revised: P 1 ≤ i ̸ =

Revised: j ≤ n u P ( x i , x j ) .

Original: GOF testing based on KSD Under mild regularity conditions (see e.g., Chwialkowski et al.(2016, Thm. 2.1), Liu et al.
Revised: GOF testing based on KSD Under mild regularity conditions (see e.g., Chwialkowski et al. Thm. 2.1), Liu et al.

Original: (2016, Prop. 3.3)), KSD guarantees D ( Q, P ) =[-
-]
Revised: (2016, Prop. 3.3)), KSD guarantees D ( Q, P ) =[-
-]

Original: ⇐⇒ Q = P ,and hence testing H 0 : Q = P against H 1 : Q ̸
Revised: ⇐⇒ Q = P , and hence testing H 0 : Q = P against H 1 : Q ̸

Original: = P is equivalent to H 0 : D ( Q, P ) = 0 against
Revised: = P is equivalent to H 0 : D ( Q, P ) = 0 against

Original: H 1 : D ( Q, P ) ̸
Revised: H 1 : D ( Q, P ) ̸

Original: = 0 .
Revised: = 0 .

Original: Given a finite sample { x i } n i =1 from Q , a consistent and unbiased estimator for

Original: D ( Q, P ) is the U-statistic ˆ D P :=

Original: 1 n ( n − 1) (cid:80) 1 ≤ i ̸ =

Original: j ≤ n u P ( x i , x j ) . The KSD test uses ˆ D P as ateststatistic to test for evidence against H 0 . The asymptotic distribution of ˆ D P under H 0 has no closedform, but can be approximated using a bootstrap procedure; see, e.g., Liu et al. 
Revised: The KSD test uses ˆ D P as a test statistic, and uses a bootstrap procedure to approximate the (intractable) asymptotic distribution of ˆ D P under H 0 ; see, e.g., Liu et al. (2016).

Original:  Limitations of KSD in GOF tests
Revised:  Limitations of KSD in GOF tests

Original: As all other hypothesis tests, GOF tests based on KSD  can be insensitive to certain forms ofdiscrepancies between the null and alternative distributions.
Revised: As all other hypothesis tests, the KSD test can be insensitive to certain families of alternatives.

Original: One example is when the target and thealternative distributions are mixture distributions that differ only in the mixing weights, which isa consequence of the “blindness” of KSD as a score-based discrepancy (Wenliang and Kanagawa,2020).
Revised: One example is when the target and the alternative distributions are mixtures that differ only in the mixing weights. This is a consequence of the “blindness” of KSD as a score-based discrepancy (Wenliang and Kanagawa, 2020).

Original: This can be seen from the fact that KSD is upper-bounded by the Fisher divergence (Johnson,2004): F ( q, p ) :
Revised: It can be seen from the fact that KSD is upper-bounded by the Fisher divergence (Johnson, 2004): F ( q, p ) :

Original: = E x ∼ Q [ ∥ s p ( x ) − s q ( x ) ∥ 22 ] (see Liu et al.
Revised: = E x ∼ Q [ ∥ s p ( x ) − s q ( x ) ∥ 22 ] (see Liu et al.

Original: (2016, Thm. 5.1)). If the set where thescore difference ∥ s p ( x ) − s q ( x ) ∥ 22 is large has low Q -probability, the Fisher divergence, thus also the KSD, will be small.
Revised: (2016, Thm. 5.1)), so if the set where the score difference ∥ s p ( x ) − s q ( x ) ∥ 22 is large has low Q -probability, the Fisher divergence, thus also the KSD, will be small.

Original: Although this issue has been studied in Wenliang and Kanagawa (2020); Zhanget al. (2022), its implication on the power of theKSD test  has not been formalised.
Revised: The blindness of KSD has been highlighted in a number of works (Wenliang and Kanagawa, 2020; Matsubara et al., 2021; Zhang et al., 2022; Gorham et al., 2019); however, its implication to the test power in GOF testing has yet been formalised.

Original: We show in Prop. 1 (proved in Appendix A) that this issue will cause the test power to converge to theprescribed test level as mode separation increases, unless the sample size grows exponentially fast.
Revised: We show in Prop. 1 (proved in Appendix A) that this issue will cause the test power to converge to the prescribed test level as mode separation increases, unless the sample size grows exponentially fast.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let Q = N (0 , I d ) and P = P ∆ =
Revised: Let Q = N (0 , I d ) and P = P ∆ =

Original: π N (0 , I d ) + (1 − π ) N (∆ , I d ) , respectively,where π ∈ [0 , 1] is the mixing proportion and ∆ ∈ R d . Suppose the kernel k satisfies
Revised: π N (0 , I d ) + (1 − π ) N (∆ , I d ) , respectively, where π ∈ [0 , 1] is the mixing proportion and ∆ ∈ R d . Suppose the kernel k satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Let x 1 , x 2 , . .
Revised: Let x 1 , x 2 , . .

Original: . be a sequence of i.i.d.
Revised: . be a sequence of i.i.d.

Original: samples from Q .
Revised: samples from Q .

Original: Suppose ∆ 1 , ∆ 2 , · · · ∈ R d is a sequence with∥ ∆ ν ∥ 2 → ∞ as ν → ∞ .
Revised: Suppose ∆ 1 , ∆ 2 , · · · ∈ R d is a sequence with ∥ ∆ ν ∥ 2 → ∞ as ν → ∞ .

Original: Let n 1 , n 2 , · · · ∈ N be such that n ν = o(cid:16) e ∥
Revised: Let n 1 , n 2 , · · · ∈ N be such that n ν = o

Original: [Equation]
Revised: [Equation]

Original: ∥ 22 / 64 (cid:17) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where ˆ D P ∆ ν is computed using x 1 , . .
Revised: where ˆ D P ∆ ν is computed using x 1 , . .

Original: , x n ν , Z j ∼ N [Equation] 0 , 1) i.i.d.
Revised: , x n ν , Z j ∼ N (0 , 1) i.i.d.

Original: and { c j } j are the eigenvalues of the Stein kernel u P under Q , i.e.the solutions of c j ϕ j ( x ) =
Revised: and { c j } j are the eigenvalues of the Stein kernel u P under Q , i.e. solutions of c j ϕ j ( x ) =

Original: E x ′ ∼ Q [ u P ( x, x ′ )
Revised: E x ′ ∼ Q [ u P ( x, x ′ )

Original: ϕ
Revised: ϕ

Original: j ( x ′ )] for non-zero ϕ j .
Revised: j ( x ′ )] for non-zero ϕ j .

Original: Remark 1.1.
Revised: Remark 1.1.

Original: The RHS of (3) is the limiting distribution of ˆ D P ∆ under H 0 (see, e.g., Liu et al.
Revised: The RHS of (3) is the limiting distribution of ˆ D P ∆ under H 0 (see, e.g., Liu et al.

Original: (2016));hence, Prop. 1 highlights that, if the sample size n does not grow fast enough, the power of the KSDtest will converge to the test level as the two modes of p becomes more and more separated.
Revised: ; hence, Prop. 1 implies that, unless the sample size n grows prohibitively fast, the power of the KSD test will converge to the test level as the two modes of p becomes more and more separated.

Original: Remark 1.2.
Revised: Remark 1.2.

Original: Assumption (2) is mild and holds for Inverse Multi-Quadrics (IMQ) and Radial Basis
Revised: Assumption (2) is mild and holds for Inverse Multi-Quadrics (IMQ) and Radial Basis

Original: Function (RBF) kernels when Q has a finite second moment.
Revised: Function (RBF) kernels when Q has a ﬁnite second moment.

Original: IMQ kernels are preferred as they havedesired tail properties to ensure a convergence determining KSD for distantly dissipative densities,which include Gaussian mixtures with common covariance like the example in Prop. 1, c.f.
Revised: IMQ kernels are preferred as they have desired tail properties to ensure a convergence determining KSD for distantly dissipative densities, which include Gaussian mixtures with common covariance like the example in Prop. 1, c.f.

Original: Gorhamet al.
Revised: Gorham et al.

Original: ; Gorham and Mackey (2017); Hodgkinson et al.
Revised: ; Gorham and Mackey (2017); Hodgkinson et al.

Original: Prop. 1 does not contradict thisresult, as it considers a different regime where a sequence of target distributions is of interest.
Revised: Prop. 1 does not contradict this result, as it considers a different regime where a sequence of target distributions is of interest.

Original: Fig.
Revised: Fig.

Original: 1 provides numerical evidence for Prop. 1 by showing the rejection rate over 100 repetitions atlevel α = 0 .
Revised: 1 provides numerical evidence for Prop. 1 by showing the rejection rate over 100 repetitions at level α = 0 .

Original: We observe that the power of the KSD test (with IMQ kernel whose bandwidth ischosen by median heuristic (Gretton et al., 2012)) becomes indistinguishable from the prescribedlevel for ∆ ≥ 6 .
Revised: We observe that the power of the KSD test (with IMQ kernel whose bandwidth is chosen by median heuristic (Gretton et al., 2012)) becomes indistinguishable from the prescribed level for ∆ ≥ 6 .

Original: The KSDA GG of Schrab et al.
Revised: The KSDA GG of Schrab et al.

Original: (2022), which uses an aggregated technique toavoid bandwidth selection via heuristics, behaved similarly.
Revised: (2022), which uses an aggregated technique to avoid bandwidth selection via heuristics, behaved similarly.

Original: Notably, this issue persists even if thesamples are drawn from both components but with an incorrect weight; see Fig. 3 in Appendix D.1.
Revised: Notably, this issue persists even if the samples are drawn from both components but with an incorrect weight; see Fig. 3 in Appendix E.1.

Original: In contrast, our proposed test, called pKSD, achieved an almost perfect power.
Revised: In contrast, our proposed test, called pKSD, achieved an almost perfect power.

Original:  Kernelized Stein discrepancy test with perturbation
Revised:  Kernelized Stein discrepancy test with perturbation

Original: Prop. 1 highlights the “locality” of the KSD test: it can only detect  discrepancies in regions wherewe have observations.
Revised: Prop. 1 highlights the myopia of the KSD test: it can only detect “local” discrepancies in regions where we have observations.

Original: If the discrepancy is “global”, such as a missing mode or incorrect mixingweight, it is necessary to turn this into a local discrepancy in order to improve the test power.
Revised: If the discrepancy is “global”, such as a missing mode or incorrect mixing weight, it is necessary to turn this into a local discrepancy in order to improve the test power.

Original: Wepropose to increase the power by perturbing both the alternative and the target distributions witha Markov transition kernel K (Robert and Casella, 2004, Chapter 6) and perform KSD tests onthe perturbed distributions .
Revised: We propose to do so by perturbing both the alternative and the target distributions with a Markov transition kernel K (Robert and Casella, 2004, Chapter 6) and performing KSD test on the perturbed distributions .

Original: Given a probability measure Q , the perturbed measure under K is( K Q )( · ) := (cid:82) X K ( x, · ) Q ( dx ) .
Revised: Given a probability measure Q , the perturbed measure under K is ( K Q )( · ) := R X K ( x, · ) Q ( dx ) .

Original: In our paper, K may also be an iterated composition of an underlyingkernel, e.g. a Metropolis-Hastings kernel.
Revised: In our paper, K may also be an iterated composition of an underlying kernel, e.g. a Metropolis-Hastings kernel.

Original: Perturbed kernel Stein discrepancy Given a Markov transition kernel K , the perturbed kernelized Stein discrepancy (pKSD), D ( Q, P ; K ) , is defined as
Revised: Perturbed kernel Stein discrepancy Given a Markov transition kernel K , the perturbed kernelized Stein discrepancy (pKSD), D ( Q, P ; K ) , is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: assuming K P admits a continuously differentiable density so that its score function is well defined.
Revised: assuming K P admits a continuously differentiable density so that its score function is well deﬁned.

Original: Notably, K Q need not have a (Lebesgue) density for (4) to exist.
Revised: Notably, K Q need not have a (Lebesgue) density for (4) to exist.

Original: For a general K , the pKSD isno longer be a statistical discrepancy as D ( Q, P ; K ) =
Revised: The next result (proved in Appendix B) shows that pKSD with an appropriately chosen K admits a closed-form solution.

Original: ̸ = ⇒ Q = P . 
Revised: Proposition 2 (pKSD close form) . Assume K is a perturbation operator for which K P attains a (Lebesgue) density that is continuously differentiable.

Original: However, so long as 109 D ( Q, P ; K ) = [-
 whenever-] Q = P  ,  the test with KSD replaced by pKSD will still be valid as itcontrols the Type-I error.
Revised: If E x ∼K Q [ u K P ( x, x )] < ∞ , then D ( Q, P ; K ) = [-
 whenever-]E x,x ′ ∼K Q [ u K P ( x, x ′ )] , where u K P is the Stein kernel for K P as deﬁned in Sec.

Original: The properties ofpKSD are dictated by the operator K .
Revised: The choice of K is paramount.

Original: A desirable choiceof K should ensure(i) pKSD, and in particular s K p , are well defined, (ii) (4) can be evaluated efficiently, (iii) the testbased on pKSD controls the Type-I error, and (iv) the test achieves a high power against distributionswith well-separated modes.
Revised: A desirable choice should ensure (i) the score function of K P is well deﬁned and efﬁciently evaluatable, and (ii) the test can achieve a high power against alternatives with wrong mixing weights.

Original: Given these desiderata , we propose to use a transition kernelK that is P -invariant and uses a jump proposal .
Revised: To this end, we propose to use a transition kernel that is P -invariant and relies on a jump proposal to perturb alternatives.

Original: A transition kernel K is P -invariant if K P = P .
Revised: A transition kernel K is P -invariant if K P = P .

Original: This meansthe score function s K p =
Revised: This means the score function s K p =

Original: s p is unchanged after perturbation, so D ( Q, P ; K ) = 0 whenever Q = P ,and (i) to (iii) are trivially satisfied.
Revised: s p is unchanged after perturbation, thus (i) is trivially satisﬁed.

Original: For iv) , the “jump proposal” makes use of geometric informationabout the modes of p and proposes inter-modal jumps to create local discrepancy that KSD can detect.
Revised: For (ii) , the “jump proposal” leverages local geometric information of the modes of p and proposes inter-modal jumps to create local discrepancy that KSD can detect.

Revised: The resulting pKSD no longer separates probability measures; we will address this major limitation in the full version of the paper.

Original: Choosing the transition kernel We consider transition kernels of the Metropolis-Hastings (MH)type (Metropolis et al., 1953; Hastings, 1970).
Revised: Choosing the transition kernel We consider transition kernels of the Metropolis-Hastings (MH) type (Metropolis et al., 1953; Hastings, 1970).

Original: At a current state x , a new state x ′ is proposedby first randomly selecting two modes µ u 1 ,µ u 2of p , then mapping  to ( x ′ , u ′ )
Revised: At a current state x , a new state x ′ is proposed by ﬁrst randomly selecting two modes of p indexed by u 1 , u 2 , then mapping ( x, u ) to ( x ′ , u ′ )

Original: =
Revised: =

Original: h ( x, u ) , whereu := ( u 1 , u 2 ) ,u ′ := ( u 2 , u 1 ) , and h is a deterministic, invertible function that is differentiablewith differentiable inverse.
Revised: h ( x, u ) , where u := ( u 1 , u 2 ) , and h is a deterministic, invertible function that is differentiable with differentiable inverse.

Original: The proposed state x ′ = x ′ ( x, u ) is hence deterministic given x and u .
Revised: The proposed state x ′ = x ′ ( x, u ) is hence deterministic given x and u .

Original: The transition kernel is K ( x, A ) =
Revised: The transition kernel is K ( x, A ) =

Original: (cid:80) u ∈U δ x ′ ( A ) g ( u ) α
Revised: P u ∈U δ x ′ ( A ) g ( u ) α

Original: ( x, x ′ )
Revised: ( x, x ′ )

Original: + δ x ( A ) r ( x ) , where α ( x, x ′ ) is anaccept-reject rule, δ x ( A )
Revised: + δ x ( A ) r ( x ) , where α ( x, x ′ ) is an acceptreject rule, δ x ( A )

Original: = 1 if x ∈ A and 0 otherwise, r ( x ) = 1 − (cid:80) u ∈U g ( u ) α
Revised: = 1 if x ∈ A and 0 otherwise, r ( x ) = 1 − P u ∈U g ( u ) α

Original: ( x, x ′ ) , and thesum is over pairs of distinct mode indices U :=
Revised: ( x, x ′ ) , and the sum is over pairs of distinct mode indices U :=

Original: { ( i, j ) : 1 ≤ i ̸ =
Revised: { ( i, j ) : 1 ≤ i ̸ =

Original: j ≤ M } .
Revised: j ≤ M } .

Original: Crucially, α is designedto satisfy the detailed balance condition to guarantee P -invariance.
Revised: Crucially, α is designed to satisfy the detailed balance condition to guarantee P -invariance.

Original: See Appendix B.2 for details.
Revised: See Appendix C.2 for details.

Original: Choosing the proposal  We use a proposal  that allows local discrepancyto be created by makinginter-modal jumps.
Revised: Choosing the proposal h We use a proposal h that creates local discrepancy by making intermodal jumps.

Original: Denote by µ 1 , . .
Revised: Denote by µ 1 , . .

Original: µ M ∈ R d the mode vectors, and A 1 , . . .
Revised: µ M ∈ R d the mode vectors, and A 1 , . . .

Original: , A M ∈ R d × d theinverse of the Hessian matrices of − log p at those points.
Revised: , A M ∈ R d × d the inverse of the Hessian matrices of − log p at those points.

Original: Given u
Revised: Given u

Original: = ( u 1 , u 2 ) ∼ Uniform ( U ) ,the proposal is defined as h ( x, u ) =
Revised: = ( u 1 , u 2 ) ∼ Uniform ( U ) , the proposal is deﬁned as h ( x, u ) =

Original: ( A 1 / 2 u 2 A − 1 / 2 u( x − θµ u 1 ) + θµ u 2 , u ) , where θ > 0 is a fixedconstant.
Revised: ( A 1 / 2 u 2 A − 1 / 2 u( x − θµ u 1 ) + θµ u 2 , u ) , where θ > 0 is a ﬁxed constant.

Original: Intuitively, when p is a mixture of elliptic distributions such as Gaussian or multivariatet -distributions, each A m is the covariance matrix of a component, and h sends x from mode µ u 1 to the“corresponding” location in µ u 2 .
Revised: Intuitively, when p is a mixture of elliptic distributions such as Gaussian or multivariate t -distributions, each A m is the covariance matrix of a component, and h sends x from mode µ u 1 to the “corresponding” location in µ u 2 .

Original: The proposed method relies on estimates of the mode vectors and Hessians and two hyperparameters (jump scale θ and number of transition steps T ).
Revised: The proposed method relies on estimates of the mode vectors and Hessians, and two hyperparameters (jump scale θ and number of transition steps T ).

Original: Appendix B.3 to B.4 discuss how to estimate/tune them in practice, as well as limitations of the proposed method.
Revised: Appendix C.3 to C. discuss how to estimate/tune them in practice, as well as limitations of the proposed method.

Original: GOF Tests with pKSD
Revised: GOF Tests with pKSD

Original: The P -invariance of K ensures that pKSD admits a closed-form expressionand that the null distribution of the pKSD statistic remains identical to that of KSD. Therefore, thep -value can be estimated with the same bootstrap technique in KSD tests and the perturbed sample{ ˜ x i } ni =1 ∼ K Q , which can be drawn by running 1-step transitions under K starting fromeach x i  .
Revised: The P -invariance implies that pKSD reduces to KSD under H 0 . Therefore, the null distribution of the pKSD statistic remains identical to that of KSD, and the p -value can be estimated with the same bootstrap technique in KSD tests given sample { ˜ x i } ni =1 from the perturbed distribution K Q , which can be drawn by running 1-step transitions under K starting from x i ∼ Q .

Original: The complete algorithm of GOF testing with pKSD is given in Algorithm 1 in Appendix.
Revised: The complete algorithm of GOF testing with pKSD is given in Algorithm 1 in Appendix.

Original:  Experiments
Revised:  Experiments

Original: [Figure]
Revised: [Figure]

Original: We apply our method to hyperparameter selection for a MCMC sampler in a Bayesian inference task of sensors localisation (Tak et al.,2018).
Revised: We apply our method to hyperparameter selection for a MCMC sampler in a Bayesian inference task of sensors localisation (Tak et al., 2018).

Original: The goal is to approximate a posterior distribution on the locations of 4 sensorsgiven noisy observations of pairwise distances.
Revised: The goal is to approximate a posterior distribution on the locations of 4 sensors given noisy observations of pairwise distances.

Original: The full experimental setup is in Appendix D.2.
Revised: The full experimental setup is in Appendix E.2.

Original: Tak et al.
Revised: Tak et al.

Original: (2018) approached this problem witha Metropolis algorithm called RAM, which isbased on alternating repelling-attracting proposals and depends on hyperparameter σ > 0 (calledthe scale ).
Revised: (2018) approached this problem with a Metropolis algorithm called RAM, which is based on alternating repelling-attracting proposals and depends on hyperparameter σ > 0 (called scale ).

Original: Tak et al.
Revised: Tak et al.

Original: (2018) used σ = 1 . 08 .
Revised: (2018) used σ = 1 . 08 .

Original: Weaim to test the estimation quality of RAM using different values of σ under a fixed computationalbudget.
Revised: We aim to test the estimation quality of RAM using different values of σ under a ﬁxed computational budget.

Original: We see from Table 1 that, for σ = 0 . 1 and 1.08, KSD test and KSDA GG test only rejectonce or none in 10 repetitions.
Revised: We see from Table 1 that, for σ = 0 . 1 and 1.08, KSD test and KSDA GG test only reject once or none in 10 repetitions.

Original: This is inconsistent with the posterior plot (Fig. 2) as some modes areclearly missing (before perturbation).
Revised: This is inconsistent with the posterior plot (Fig. 2) as some modes are clearly missing (before perturbation).

Original: On the other hand, pKSD rejects most tests by creating localdiscrepancy around the missing modes.
Revised: On the other hand, pKSD rejects most tests by creating local discrepancy around the missing modes.

Original: In contrast, with σ = 0 . 5 , there is no clear evidence that theposterior samples have missed any modes, and the samples before and after perturbation also looksimilar.
Revised: In contrast, with σ = 0 . 5 , there is no clear evidence that the posterior samples have missed any modes, and the samples before and after perturbation also look similar.

Original: The results match this observation, with at most one rejection for all three methods.
Revised: The results match this observation, with at most one rejection for all three methods.

Original:  Conclusion
Revised:  Conclusion

Original: [Table]
Revised: [Table]

Original: We formalise the failure of GOF tests basedon KSD  when the target distribution has wellseparated modes via a concrete example of bimodal Gaussian.
Revised: We formalise the failure of the KSD test when the target distribution has well-separated modes via a concrete example of bimodal Gaussian.

Original: To increase its power, we propose to perturb both the alternative and the target distributions using a Markov process to create localdiscrepancies that KSD can detect.
Revised: To increase its power, we propose to perturb both the alternative and the target distributions using a Markov process, which creates local discrepancies that KSD can detect but no longer guarantee separation of probability measures.

Original: The proposed test provably controls Type-I error.
Revised: In the full version of the paper, we will extent pKSD to a statistical divergence that indeed separates measures, and include more experiments to examine the test power against different types of alternatives. Acknowledgments and Disclosure of Funding

Original: Empirical resultssuggest that our method can achieve higher test power against incorrect mixing weights (includingthe case of missing modes), particularlywhen the mixing components are elliptic distributions.
Revised: XL was supported by the Presidents PhD Scholarships of Imperial College London and the EPSRC StatML CDT programme EP/S023151/1. AD was supported by Wave 1 of The UKRI Strategic Priorities Fund under the EPSRC Grant EP/T001569/1 and EPSRC Grant EP/W006022/1, particularly the Ecosystems of Digital Twins theme within those grants & The Alan Turing Institute.

Original: AppendixA Proof of Proposition1
Revised: Appendix A Proof of Proposition

Original: Prop. 1 states that, when a sample completely misses one mode of a bimodal Gaussian target and thesample size n does not grow fast enough with the inter-modal distance ∥
Revised: Prop. 1 states that, when a sample completely misses one mode of a bimodal Gaussian target and the sample size n does not grow fast enough with the inter-modal distance ∥

Original: ∆ ∥ 2 , the distribution of thekernelized Stein discrepancy statistic will converge to the null distribution, even though the sample isclearly not representative for the target.
Revised: ∆ ∥ 2 , the distribution of the kernelized Stein discrepancy statistic will converge to the null distribution, even though the sample is clearly not representative for the target.

Original: To prove this result, we need the following lemma.
Revised: To prove this result, we need the following lemma.

Original: Lemma 2.
Revised: Lemma 3.

Original: Under the same assumptions in Prop. 1, we have E x ∼ Q [ ∥ s p ∆ ( x ) − s q ( x ) ∥ 22 ]
Revised: Under the same assumptions in Prop. 1, we have E x ∼ Q [ ∥ s p ∆ ( x ) − s q ( x ) ∥ 22 ]

Original: =o (cid:16) e −∥ ∆ ∥ 22 / 32 (cid:17) .
Revised: = o (cid:16) [Equation]

Original: The above lemma states that the expected square error between the score functions (i.e. the Fisherdivergence ) between Q and P ∆ decays with a rate at least exponentially fast in the inter-modaldistance ∥ ∆ ∥ 2 .
Revised: The above lemma states that the expected square error between the score functions (i.e. the Fisher divergence ) between Q and P ∆ decays with a rate at least exponentially fast in the inter-modal distance ∥ ∆ ∥ 2 .

Original: We now use this lemma to prove Prop. 1; the proof of this lemma is provided later inthis section.
Revised: We now use this lemma to prove Prop. 1; the proof of this lemma is provided later in this section.

Original: Proof of Prop. 1.
Revised: Proof of Prop. 1.

Original: Fixinga positive integer ν , we can write n ν ˆ D P ∆ ν = n ν ˆ D Q + n ν
Revised: Fixing positive integer ν , we can write n ν ˆ D P ∆ ν = n ν ˆ D Q + n ν

Original: (ˆ D P ∆ ν − ˆ D Q ) .
Revised: (ˆ D P ∆ ν − ˆ D Q ) .

Original: Under the stated assumption of the kernel k and, one can check that E x,x ′ ∼ Q [ u Q ( x, x ′ ) 2 ] < ∞ when Q is the standard normal distribution, so we can apply Liu et al.
Revised: Under the stated assumption of the kernel k and, one can check that E x,x ′ ∼ Q [ u Q ( x, x ′ ) 2 ] < ∞ when Q is the standard normal distribution, so we can apply Liu et al.

Original: (2016, Thm 4.1) to conclude that, asν → ∞ ,
Revised: (2016, Thm 4.1) to conclude that, as ν → ∞ ,

Original: [Equation]
Revised: [Equation]

Original:  where z j , c j are as defined in Prop. 1.
Revised: (5) where z j , c j are as deﬁned in Prop. 1.

Original: If we could show that n ν (ˆ D P ∆ ν − ˆ D Q ) → 0 in probability asν → ∞ , then the desired result would follow from Slutsky’s Theorem (Casella and Berger, 2001)and (5).To show n ν (ˆ D P ∆ ν − ˆ D Q ) → 0 in probability [Equation]
Revised: If we could show that n ν (ˆ D P ∆ ν − ˆ D Q ) → 0 in probability as ν → ∞ , then the desired result would follow from Slutsky’s Theorem (Casella and Berger, 2001) and (5).

Original:  we fix ϵ > 0 and denote by Pr Q the probability under Q .
Revised: To do so, we ﬁx ϵ > 0 and denote by Pr Q the probability under Q .

Original: We also omit the dependence of n and ∆ on ν for brevity.
Revised: We also omit the dependence of n and ∆ on ν for brevity.

Original: The Markov inequality yields
Revised: The Markov inequality yields

Original: [Equation]
Revised: [Equation]

Original: We bound each of the three terms individually.
Revised: We bound each of the three terms individually.

Original: For the first term
Revised: For the ﬁrst term

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: , where the last line follows from the fact that ( a + b ) 2 ≤ 2 a 

Revised: where the last line follows from the fact that ( a + b ) 2 ≤ 2 a

Original: + 2 b 2 for any a, b ∈ R .
Revised: + 2 b 2 for any a, b ∈ R .

Original: Now,
Revised: Now,

Original: [Equation]
Revised: [Equation]

Original: where the first line follows from the Cauchy-Schwarz inequality, and the last line holds because 
Revised: where the ﬁrst line follows from the Cauchy-Schwarz inequality, and the last line holds because ∥

Original: [Equation]
Revised: [Equation]

Original: By Lemma 2 we know that E x ∼ Q [ ∥ s p ∆ ( x ) − s q ( x ) ∥ 22 ]
Revised: By Lemma 3 we know that E x ∼ Q [ ∥ s p ∆ ( x ) − s q ( x ) ∥ 22 ]

Original: = o (cid:16) e −∥ ∆ ∥ 22 / 32 (cid:17) .
Revised: = o [Equation] ∆ ∥ [Equation]

Original:  Also, direct computationgives that E x ∼ Q [ ∥ s q ( x ) ∥ 22 ] = 1 .
Revised: [Equation] Also, direct computation gives that E x ∼ Q [ ∥ s q ( x ) ∥ 22 ] = 1 .

Original: A similar argument shows that T 2 =
Revised: A similar argument shows that T 2 =

Original: o (cid:16) e −∥ ∆ ∥ 22 / 32 (cid:17) for someconstant a 2 .
Revised: o (cid:16) [Equation] −∥ ∆ ∥ [Equation]

Original:  We therefore conclude that the first term of (6) is o (cid:16) e −∥ ∆ ∥ 22 / 64 (cid:17) .
Revised: [Equation] for some constant a 2 . We therefore conclude that the ﬁrst term of (6) is o [Equation]

Original: With a similar argument and by the boundedness assumptions on the kernel k , we conclude that thesecond and third terms of (6) are both o ( e −∥ ∆ ∥ 22 / 64 ) .
Revised: With a similar argument and by the boundedness assumptions on the kernel k , we conclude that the second and third terms of (6) are both o ( e −∥ ∆ ∥ 22 / 64 ) .

Original: Combining all the above, we conclude thatthere exists a universal constant a 3 such that, for sufficiently large ∥ ∆ ∥ 2 ,
Revised: Combining all the above, we conclude that there exists a universal constant a 3 such that, for sufﬁciently large ∥ ∆ ∥ 2 ,

Original: [Equation]
Revised: [Equation]

Original: which goes to 0 if n = o (cid:16)e ∥ ∆ ∥ 22 / 64 (cid:17) . This shows that n ν (ˆ D P ∆ ν − ˆ D Q ) → 0 in probability asν → ∞ , so the desired result follows. Algorithm Require:
Revised: which goes to 0 if n = o (cid:16)

Original: Training set { x train ,j } n train j =1 , test set { x test ,j } n test j =1 , target P , number of transition steps T , candidate jump scales { θ 1 , . . . , θ L } , sizes of training and testing sets n train and n test , resp. Output Whether the null hypothesis is rejected. Estimate the mode { µ 1 , µ 2 , . . . , µ M } and Hessians { A 1 , . . .
Revised: [Equation]

Original: , A M } using Algorithm 2. Use { x train ,j } n train j =1 to find θ ∗ ∈ arg max θ ∈{ θ 1 ,...,θ L } ˆ D P, K ( T ) θ / ˆ σ u . Perturb { x test ,j } n test j =1 with the selected kernel K ( T ) θ ∗ , and use (8) to compute test statistic ˆ D P, K ( T ) θ ∗ . Compute bootstrap samples using (9) and find the (1 − α )
Revised: [Equation]

Original: -quantile ˆ γ 1 −α . Reject the null hypothesis if ˆ D P, K ( T ) θ ∗ ≥ ˆ γ 1 − α .
Revised: This shows that n ν (ˆ D P ∆ ν − ˆ D Q ) → 0 in probability as ν → ∞ , so the desired result follows.

Original: Proof of Lemma 2.
Revised: Proof of Lemma 3.

Original: For any δ > 0 , define B δ := { x ∈ R d : ∥ x ∥ 2 ≤ δ } .
Revised: For any δ > 0 , deﬁne B δ := { x ∈ R d : ∥ x ∥ 2 ≤ δ } .

Original: We have the followingdecomposition
Revised: We have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: The rest of the proof proceeds with bounding the two terms separately.
Revised: The rest of the proof proceeds with bounding the two terms separately.

Original: We first note that standardcomputation gives
Revised: We ﬁrst note that standard computation gives

Original: [Equation]
Revised: [Equation]

Original: , and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: For x ∈ B δ , Cauchy-Schwarz inequality implies ∆ ⊺ x ≤ ∥ ∆ ∥ 

Revised: For x ∈ B δ , Cauchy-Schwarz inequality implies ∆ ⊺ x ≤ ∥ ∆ ∥

Original: ∥ x ∥ 2 ≤ δ ∥ ∆ ∥ 2 .
Revised: ∥ x ∥ 2 ≤ δ ∥ ∆ ∥ 2 .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: and the first term of (7) can be bounded as
Revised: and the ﬁrst term of (7) can be bounded as

Revised: E x ∼ Q

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the fact that E x ∼ Q [ δ x ( B δ )] ≤ 1 .
Revised: where the last inequality follows from the fact that E x ∼ Q [ δ x ( B δ )] ≤ 1 .

Original: To bound the second term of (7), we note that for x ∈ R d \ B δ ,
Revised: To bound the second term of (7), we note that for x ∈ R d \ B δ ,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Revised: E

Original: [Equation]
Revised: [Equation]

Original: , by the tail probability of Gaussian distributions (see e.g. Wainwright (2019, Prop. 2.5)).
Revised: by the tail probability of Gaussian distributions (e.g., Wainwright (2019, Prop. 2.5)).

Original: Combiningthese results we have
Revised: Combining these results we have

Original: [Equation]
Revised: [Equation]

Original: where the last line follows by choosing δ = ∆ / 3 .
Revised: where the last line follows by choosing δ = ∆ / 3 .

Original: Noting the RHS of the last inequality iso (cid:0) − 132 ∥ ∆ ∥ 22 (cid:1) completes the proof.
Revised: Noting the RHS of the last inequality is o (cid:0) − 132 ∥ ∆ ∥ 22 (cid:1) completes the proof.

Revised: Algorithm 1 Goodness-of-Fit Test with pKSD (the proposed method).

Revised: Require:

Revised: Training set { x train ,j } n train j =1 , test set { x test ,j } n test j =1 , target P , number of transition steps T , candidate jump scales { θ 1 , . . .

Revised: , θ L } , sizes of training and testing sets n train and n test , resp.

Original:  Algorithm 
Revised: Output Whether the null hypothesis is rejected. Estimate the mode { µ 1 , µ 2 , . . . , µ M } and Hessians { A 1 , . . . , A M } using Algorithm 2. Use { x train ,j } n train j =1 to ﬁnd θ ∗ ∈ arg max θ ∈{ θ 1 ,...,θ L } ˆ D P, K ( T ) θ / ˆ σ u .

Revised: Perturb { x test ,j } n test j =1 with the selected kernel K ( T ) θ ∗ , and use (8) to compute test statistic ˆ D P, K ( T ) θ ∗ .

Revised: Compute bootstrap samples using (9) and ﬁnd the (1 − α )

Revised: -quantile

Revised: ˆ γ 1 − α .

Revised: Reject the null hypothesis if ˆ D P, K ( T ) θ ∗ ≥ ˆ γ 1 − α .

Revised: Algorithm

Revised: (Pompe et al., 2020, Algorithm 3)

Original: Require:
Revised: Require:

Original: Initial points s 1 , . . .
Revised: Initial points s 1 , . . .

Original: , s M 0 , small positive value β .
Revised: , s M 0 , small positive value β .

Original: Output Approximates for mode vectors { µ 1 , . .
Revised: Output Approximates for mode vectors { µ 1 , . .

Original: . , µ M } .
Revised: . , µ M } .

Original: Run BFGS to minimise − log p ( x ) with initial points s 1 , . . .
Revised: Run BFGS to minimise − log p ( x ) with initial points s 1 , . . .

Original: , s M 0 .
Revised: , s M 0 .

Original: Denote the estimated local optima by m 1 , . . .
Revised: Denote the estimated local optima by m 1 , . . .

Original: , m M 0 and their corresponding Hessian matrices by
Revised: , m M 0 and their corresponding Hessian matrices by

Original: A 1 , . . .
Revised: A 1 , . . .

Original: , A M 0 .
Revised: , A M 0 .

Original: Set µ 1 := m 1 , A µ 1 :
Revised: Set µ 1 := m 1 , A µ 1 :

Original: = A 1 , M = 1 .
Revised: = A 1 , M = 1 .

Original: for i = 2 , . . .
Revised: for i = 2 , . . .

Original: , M 0 do if min j ∈{ 1 ,...,M } 12 (( µ j − m i ) ⊺
Revised: , M 0 do if min j ∈{ 1 ,...,M } 12 (( µ j − m i ) ⊺

Original: A µ j ( µ j − m i ) + ( µ j − m i ) ⊺
Revised: A µ j ( µ j − m i ) + ( µ j − m i ) ⊺

Original: A i ( µ j − m i ))
Revised: A i ( µ j − m i ))

Original: <
Revised: <

Original: β then k := arg min j ∈{ 1 ,...,M } 12 (( µ j − m i ) ⊺
Revised: β then k := arg min j ∈{ 1 ,...,M } 12 (( µ j − m i ) ⊺

Original: A µ j ( µ j − m i ) + ( µ j − m i ) ⊺
Revised: A µ j ( µ j − m i ) + ( µ j − m i ) ⊺

Original: A i ( µ j − m i )) .
Revised: A i ( µ j − m i )) .

Original: if p ∗ ( µ k ) <
Revised: if p ∗ ( µ k ) <

Original: p ∗ ( m i ) then
Revised: p ∗ ( m i ) then

Original: Set µ k :
Revised: Set µ k :

Original: =
Revised: =

Original: m i and A µ k :=
Revised: m i and A µ k :=

Original: A i .
Revised: A i .

Original: end if else µ M +1 :
Revised: end if else µ M +1 :

Original: = m i and A µ M +1 :
Revised: = m i and A µ M +1 :

Original: = A i .
Revised: = A i .

Original: M :
Revised: M :

Original: =
Revised: =

Original: M + 1 . end if end for
Revised: M + 1 . end if end for

Original: B Details of Proposed Method
Revised: B Proof of Proposition

Original: B.1 pKSD Closed Form

Original: Proposition 3 (pKSD close form) .

Original: Assume K is a perturbation operator for which K P attains a (Lebesgue) density that is continuously differentiable.

Original: If E x ∼K Q [ u K P ( x, x )] < ∞ , then 293 D ( Q, P ; K ) =

Original: E x,x ′ ∼K Q [ u K P ( x, x ′ )] , where u K P is the Stein kernel for K P as defined in Sec. 2).

Original: Proof. This follows  from Chwialkowski et al.
Revised: Under the stated assumptions, this follows directly from Chwialkowski et al.

Original: (2016, Thm. 2.1) applied to K Q and K P .
Revised: (2016, Thm. 2.1) applied to K Q and K P .

Revised: C Details of Proposed Method C.1 Estimating pKSD

Original: Proposition 3 is a general result and applies to any perturbation operator K for which the density of 
Revised: Proposition 2 is a general result and applies to any perturbation operator K for which the density of K

Original: K P is well-defined.
Revised: P is well-deﬁned.

Original: In particular, it applies to pKSD where K is a P -invariant Markov transitionkernel.
Revised: In particular, it applies to pKSD where K is a P -invariant Markov transition kernel (in which case K P = P ).

Original: Given i.i.d.
Revised: Given i.i.d.

Original: { x i } ni ∼ Q , a sample { ˜ x i } ni =1 from K Q can be drawn by running 1-step transitions under K starting from each x i .
Revised: { x i } ni ∼ Q , a sample { ˜ x i } ni =1 from K Q can be drawn by running 1-step transitions under K starting from each x i .

Original: Prop.3 then suggests estimating D ( Q, P ; K ) by the following U-statistic:
Revised: Prop.2 then suggests estimating D ( Q, P ; K ) by the following Ustatistic: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Estimating the p -value
Revised: Estimating the p -value

Original: In the vanilla KSD tests, the asymptotic distribution of the test statisticˆ D P under H 0 has no closed form.
Revised: In the standard KSD tests, the asymptotic distribution of the test statistic ˆ D P under H 0 has no closed form.

Original: Liu et al.
Revised: Liu et al.

Original: (2016) proposed to approximate it using a bootstraptechnique (Huskova and Janssen, 1993) via the bootstrap samples
Revised: (2016) proposed to approximate it using a bootstrap technique (Huskova and Janssen, 1993) via the bootstrap samples

Original: [Equation]
Revised: [Equation]

Original: [Equation] where ( w b 1 , . . . , w bn ) ∼ Mult (cid:0) n ; 1 n , . . 
Revised: (9) where ( w b 1 , . . . , w bn ) ∼ Mult (cid:0) n ; 1 n , . . .

Original: . , 1 n (cid:1) follows a multinomial distribution.
Revised: , 1 n (cid:1) follows a multinomial distribution.

Original: Since pKSD reducesto KSD under H 0 due to P -invariance, the null distribution of pKSD statistic can be approximatedusing the same bootstrap technique, with { x i } ni =1 replaced with the perturbed sample { ˜ x i } ni =1 (see
Revised: Since pKSD reduces to KSD under H 0 due to P -invariance, the null distribution of pKSD statistic can be approximated using the same bootstrap technique, with { x i } ni =1 replaced with the perturbed sample { ˜ x i } ni =1 (see

Original: Section 4).
Revised: Section 4).

Original: B.2 Choosing the transition kernel
Revised: C.2 Choosing the transition kernel

Original: Denoting by B ( X ) the Borel σ -algebra on X , a Markov transition kernel is a function K : X × B ( X ) → [0 , 1] such that (i) for all x ∈ X , K
Revised: Denoting by B ( X ) the Borel σ -algebra on X , a Markov transition kernel is a function K : X × B ( X ) → [0 , 1] such that (i) for all x ∈ X , K

Original: ( x, · ) is a probability measure on ( X , B ( X )) , and (ii)for all A ∈ B ( X ) , K ( · , A ) is a measurable function on X .
Revised: ( x, · ) is a probability measure on ( X , B ( X )) , and (ii) for all A ∈ B ( X ) , K ( · , A ) is a measurable function on X .

Original: In our example, K may also be an iteratedcomposition of an underlying kernel, e.g. a Metropolis-Hastings kernel.
Revised: In our example, K may also be an iterated composition of an underlying kernel, e.g. a Metropolis-Hastings kernel.

Original: Given a probability measure Q , the perturbed measure is ( K Q )( A ) :=
Revised: Given a probability measure Q , the perturbed measure is ( K Q )( A ) :=

Original: (cid:82) X K ( x, A ) Q ( dx ) , for measurable set A .
Revised: R X K ( x, A ) Q ( dx ) , for measurable set A .

Original: As discussed in Section 4, we choose a transition kernel of the form
Revised: As discussed in Section 4, we choose a transition kernel of the form

Original: [Equation]
Revised: [Equation]

Original: where x ′ = x ′ ( x, u ) is the proposed state, α ( x, x ′ ) is an accept-reject rule that guarantees Pinvariance, δ x ( A )
Revised: where x ′ = x ′ ( x, u ) is the proposed state, α ( x, x ′ ) is an accept-reject rule that guarantees Pinvariance, δ x ( A )

Original: = [-
-] if x ∈ A and 0 otherwise, and r ( x ) = 1 − (cid:80) u ∈U g ( u ) α
Revised: = [-
-]1 if x ∈ A and 0 otherwise, and r ( x ) = 1 − P u ∈U g ( u ) α

Original: ( x, x ′ ) .
Revised: ( x, x ′ ) .

Original: Wheng is defined on a continuous space, the same argument follows by replacing the summation withintegration.
Revised: When g is deﬁned on a continuous space, the same argument follows by replacing the summation with integration.

Original: The accept-reject rule α is designed to satisfy the detailed balance condition : 
Revised: The accept-reject rule α is designed to satisfy the detailed balance condition : R

Original: [Equation]
Revised: [Equation]

Original: for all A, B ∈ B ( X ) .
Revised: for all A, B ∈ B ( X ) .

Original: One valid choice of α ( x, x ′ ) for which (10) holds is
Revised: One valid choice of α ( x, x ′ ) for which (10) holds is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Table]

Original:  where ∂ ( x ′ , u ′ ) /∂
Revised: , (11) where ∂ ( x ′ , u ′ ) /∂

Original: ( x, u ) denotes the Jacobian of the transformation from ( x, u ) to ( x ′ , u ′ ) .
Revised: ( x, u ) denotes the Jacobian of the transformation from ( x, u ) to ( x ′ , u ′ ) .

Original: See
Revised: See

Original: Appendix B.5 for the proof.
Revised: Appendix C.5 for the proof.

Original: Accept-reject rules of the form (11) are also used in Reversible-Jump
Revised: Accept-reject rules of the form (11) are also used in Reversible-Jump

Original: MCMC (Green, 1995; Green and Hastie, 2009) and generalise the well-known MH rule, for whichthe determinant of the Jacobian is 1.
Revised: MCMC (Green, 1995; Green and Hastie, 2009) and generalise the well-known MH rule, for which the determinant of the Jacobian is 1.

Original: A similar trick can be used to generalise other accept-reject rulessuch as the Barker’s rule (Peskun, 1973; Tierney, 1998; Livingstone and Zanella, 2021).
Revised: A similar trick can be used to generalise other accept-reject rules such as the Barker’s rule (Peskun, 1973; Tierney, 1998; Livingstone and Zanella, 2021).

Original: B.3 Choosing the proposal
Revised: C.3 Choosing the proposal

Original: In our proposal h ( x, u ) =
Revised: In our proposal h ( x, u ) =

Original: ( A 1 / 2 u 2 A − 1 / 2 u( x − θµ u 1 ) + θµ u 2 , u ) , two modes (indexed by u = ( u 1 , u 2 ) )are chosen randomly for a given current state, so a proposed state can potentially lie in a low-densityregion.
Revised: ( A 1 / 2 u 2 A − 1 / 2 u( x − θµ u 1 )+ θµ u 2 , u ) , two modes (indexed by u = ( u 1 , u 2 ) ) are chosen randomly for a given current state, so a proposed state can potentially lie in a low-density region.

Original: This can hence lead to a low acceptance probability.
Revised: This can hence lead to a low acceptance probability.

Original: A similar proposal considered in Pompeet al.
Revised: A similar proposal considered in Pompe et al.

Original: (2020), called deterministic jumps , addresses this problem by recording an auxiliary variable forthe mode index and augmenting the state space to X × { 1 , 2 , . .
Revised: (2020), called deterministic jumps , addresses this problem by recording an auxiliary variable for the mode index and augmenting the state space to X × { 1 , 2 , . .

Original: , M ( M − 1) } ,
Revised: , M ( M − 1) } ,

Original: so that at every stepit is guaranteed to propose a new state belonging to a different mode.
Revised: so that at every step it is guaranteed to propose a new state belonging to a different mode.

Original: However, the same trick cannotbe used in our case because the augmented density no longer has a well-defined score function.
Revised: However, the same trick cannot be used in our case because the augmented density no longer has a well-deﬁned score function.

Original: Inour experiment, we find that our jump proposal can achieve a significant increase in power againstmixtures of elliptical distributions that disagree in the weights.
Revised: In our experiment, we ﬁnd that our jump proposal can achieve a signiﬁcant increase in power against mixtures of elliptical distributions that disagree in the weights.

Original: B.4 Further DetailsTuning the jump scale Fixing T
Revised: C.4 Further Details

Original:  ≥ 1 , we can follow the same argument in Jitkrittum et al.
Revised: Tuning the jump scale Fixing T ≥ 1 , we can follow the same argument in Jitkrittum et al.

Original: (2017, Prop. 4) to approximate the test power with ˆ D P, K ( T ) θ / ˆ σ u , where ˆ σ 2 u := 
Revised: (2017, Prop. 4) to approximate the test power with ˆ D P, K ( T ) θ / ˆ σ u , where ˆ σ 2 u := 4 n

Original: [Equation]
Revised: P ni =

Original: [Equation] (cid:16)(cid:80) ni,j =1 u P (˜ x i , ˜ x j ) (cid:17) 2 with ˜ x i ∼ K ( T ) Q , which is an estimateof the asymptotic standard deviation σ u (see also Schrab et al., 2022, Eq.
Revised: (cid:16)P nj =1 u P (˜ x i , ˜ x j ) (cid:17) 2 −n(cid:16)P ni,j =1 u P (˜ x i , ˜ x j ) (cid:17)with ˜ x i ∼ K ( T ) Q , which is an estimate of the asymptotic standard deviation σ u (see also Schrab et al., 2022, Eq.

Original: Since the objective is notdifferentiable with respect to θ , we propose to choose θ from a grid of values { θ l } Ll =1 near 1, a heuristic we find to work well in our experiments.
Revised: Since the objective is not differentiable with respect to θ , we propose to choose θ from a grid of values { θ l } Ll =1 near 1, a heuristic we ﬁnd to work well in our experiments.

Original: The objective is hence max θ ∈{ θ 1 ,...,θ L } ˆ D P, K ( T ) θ / ˆ σ u .
Revised: The objective is hence max θ [Equation] [Equation] [Equation]

Original: Estimating the mode vectors and Hessians To estimate the mode locations and Hessians at thosepoints, we follow Pompe et al.
Revised: Estimating the mode vectors and Hessians To estimate the mode locations and Hessians at those points, we follow Pompe et al.

Original: (2020) to minimise − log p by running in parallel a sequence ofoptimisers initiated at different starting points.
Revised: (2020) to minimise − log p by running in parallel a sequence of optimisers initiated at different starting points.

Original: The optimiser used is BFGS (Nocedal and Wright,2006), which returns both the local optima and approximated Hessians at those points.
Revised: The optimiser used is BFGS (Nocedal and Wright, 2006), which returns both the local optima and approximated Hessians at those points.

Original: The optimaare then merged if their Mahalanobis distance weighted by the approximated Hessian is smaller thana pre-specified threshold.
Revised: The optima are then merged if their Mahalanobis distance weighted by the approximated Hessian is smaller than a pre-speciﬁed threshold.

Original: In our experiments, we initialise the optimisers from a set of size n train , halfof which is drawn randomly from the training set and the other half sampled uniformly from a hypercube [ L 1 , U 1 ] × · · · × [ L d , U d ] .
Revised: In our experiments, we initialise the optimisers from a set of size n train , half of which is drawn randomly from the training set and the other half sampled uniformly from a hyper cube [ L 1 , U 1 ] × · · · × [ L d , U d ] .

Original: The full procedure is described in Appendix C.
Revised: The full procedure is described in Appendix D.

Original: Choice of number of transitions The number of transitions T affects how many perturbed samplesare accepted across all transition steps, thus impacting the performance of the pKSD test.
Revised: Choice of number of transitions The number of transitions T affects how many perturbed samples are accepted across all transition steps, thus impacting the performance of the pKSD test.

Original: Intuitively, T should be set to a large value when the acceptance rate is low, which would happen if the estimatesof the modes and local Hessians of the target distribution are inaccurate, or the target distributioncannot be approximated by a mixture of elliptic distributions.
Revised: Intuitively, T should be set to a large value when the acceptance rate is low, which would happen if the estimates of the modes and local Hessians of the target distribution are inaccurate, or the target distribution cannot be approximated by a mixture of elliptic distributions.

Original: We propose two heuristics to choose this hyper-parameter in practice: i) tuning this parameter onthe training set by selecting from a pre-specified grid of values, or ii) setting it to a large value (e.g.,
Revised: We propose two heuristics to choose this hyper-parameter in practice: i) tuning this parameter on the training set by selecting from a pre-speciﬁed grid of values, or ii) setting it to a large value (e.g.,

Original: T = 1000 ) if the computational budget allows.
Revised: T = 1000 ) if the computational budget allows.

Original: One concern with ii) is that the candidate distribution might converge to the limiting distribution(which, for our choice of transition kernel, is the target distribution P ) as T → ∞ , so a large T could drive the perturbed distribution close to P , making it harder to distinguish.
Revised: One concern with ii) is that the candidate distribution might converge to the limiting distribution (which, for our choice of transition kernel, is the target distribution P ) as T → ∞ , so a large T could drive the perturbed distribution close to P , making it harder to distinguish.

Original: This is however not amajor concern, as the target distribution P is not the only limiting distribution of the transition kerneldue to non-irreducibility, so the data distribution Q will not necessarily converge to P as T → ∞ .
Revised: This is however not a major concern, as the target distribution P is not the only limiting distribution of the transition kernel due to non-irreducibility, so the data distribution Q will not necessarily converge to P as T → ∞ .

Original: Although this does not offer a theoretical guarantee, some empirical evidence is provided by theexperiments we conducted.
Revised: Although this does not offer a theoretical guarantee, some empirical evidence is provided by the experiments we conducted.

Original: Limitations
Revised: Limitations

Original: The jump proposal of the transition kernel used in pKSD is constructed specificallyfor targets that are mixtures of elliptic distributions and relies on estimates of the location and localgeometry of modes, so it may suffer from a low acceptance rate for targets with more complicatedgeometrical structure.
Revised: The jump proposal of the transition kernel used in pKSD is constructed specifically for targets that are mixtures of elliptic distributions and relies on estimates of the location and local geometry of modes, so it may suffer from a low acceptance rate for targets with more complicated geometrical structure.

Original: One future direction is to study alternative proposals that relaxes theseassumptions.
Revised: One future direction is to study alternative proposals that relaxes these assumptions.

Original: Another limitation is that pKSD is not a valid discrepancy, as D ( Q, P ; K ) =[-
-]
Revised: Another limitation is that pKSD is not a valid discrepancy, as D ( Q, P ; K ) =[-
-]

Original: ̸ = ⇒ Q = P .
Revised: ̸ = ⇒ Q = P .

Original: Understanding in which distribution class pKSD does have this guarantee is anotherworthwhile research question.
Revised: Understanding in which distribution class pKSD does have this guarantee is another worthwhile research question.

Original: B.5 A sufficient condition
Revised: C.5 A sufﬁcient condition

Original: We first give a sufficient condition for the detailed balance equation (10).
Revised: We ﬁrst give a sufﬁcient condition for the detailed balance equation (10).

Original: For simplicity, we writex ′ =
Revised: For simplicity, we write x ′ =

Original: x
Revised: x

Original: ′ ( x, u ) and x = x ( x ′ , u ′ ) , so that the dependence of x ′ on u and of x on x ′ is implicit.
Revised: ′ ( x, u ) and x = x ( x ′ , u ′ ) , so that the dependence of x ′ on u and of x on x ′ is implicit.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let p be a probability density function on X ⊂ R d .
Revised: Let p be a probability density function on X ⊂ R d .

Original: Suppose that h is a deterministic,invertible function that is differentiable with differentiable inverse.
Revised: Suppose that h is a deterministic, invertible function that is differentiable with differentiable inverse.

Original: Furthermore, let g be a knowndensity defined on some discrete space U .
Revised: Furthermore, let g be a known density deﬁned on some discrete space U .

Original: Consider a Markov transition kernel of the form
Revised: Consider a Markov transition kernel of the form

Original: [Equation]
Revised: [Equation]

Original: (12) where x ′ = x ( x, u ) , δ x ( A )
Revised: where x ′ = x ( x, u ) , δ x ( A )

Original: = 1 if x ∈ A and 0 otherwise, and r ( x ) = 1 − (cid:80) u ∈U g ( u ( x, x ′ ))
Revised: = 1 if x ∈ A and 0 otherwise, and r ( x ) = 1 − P u ∈U g ( u ( x, x ′ ))

Original: α ( x, x ′ ) .
Revised: α ( x, x ′ ) .

Original: Then an accept-reject rule α ( x, x ′ ) satisfies the detailed balance condition (13) if
Revised: Then an accept-reject rule α ( x, x ′ ) satisﬁes the detailed balance condition (13) if

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows largely from Green and Hastie (2009, Sec. 1.2.1), which shows the claimwhen the density g is defined on a continuous space.
Revised: The proof follows largely from Green and Hastie (2009, Sec. 1.2.1), which shows the claim when the density g is deﬁned on a continuous space.

Original: By the invertibility of the transformation h , achange-of-variable formula can be applied to the right-hand-side of (10) to yield 
Revised: By the invertibility of the transformation h , a change-of-variable formula can be applied to the right-hand-side of (10) to yield R

Original: [Equation]
Revised: [Equation]

Revised: ) (cid:12)(cid:12)(cid:12) ∂ ( x ′

Revised: [Equation]

Original: dx
Revised: u [Equation] ∂ ( x [Equation]

Revised: u ) (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Now define U B := { u : x ′ ∈ B } and similarly for U A .
Revised: Now deﬁne U B := { u : x ′ ∈ B } and similarly for U A .

Original: We have that ( x, u ) ∈
Revised: We have that ( x, u ) ∈

Original: A × U B = ⇒( x ′ , u ′ )
Revised: A × U B = ⇒ ( x ′ , u ′ )

Original: =
Revised: =

Original: h ( x, u ) ∈
Revised: h ( x, u ) ∈

Original: B × U A , and, similarly, ( x ′ , u ′ )
Revised: B × U A , and, similarly, ( x ′ , u ′ )

Original: ∈ B × U A = ⇒ (
Revised: ∈ B × U A = ⇒ (

Original: x, u ) ∈
Revised: x, u ) ∈

Original: A × U B .
Revised: A × U B .

Original: Wetherefore conclude that a sufficient condition is
Revised: We therefore conclude that a sufﬁcient condition is

Original: [Equation]
Revised: [Equation]

Original: thus showing the claim.
Revised: thus showing the claim.

Original: In particular, it follows that the detailed balance condition holds with (11) by verifying that it satisfies(13).
Revised: In particular, it follows that the detailed balance condition holds with (11) by verifying that it satisﬁes (13).

Original: This can be viewed as a generalisation of the Metropolis-Hastings (MH) rule α ( x, x ′ ) 
Revised: This can be viewed as a generalisation of the Metropolis-Hastings (MH) rule α ( x, x ′ ) =

Original: = min(cid:16) 1 , p ( x ′ )
Revised: min

Original: g ′ ( u ′ )
Revised: [Equation]

Original: p ( x ) g ( u ) (cid:17) .
Revised: [Figure]

Original: C Finding mode vectors via optimisation and merging
Revised: D Finding mode vectors via optimisation and merging

Original: Finding local modes In practice, the mode locations of a non-trivial target distribution and the
Revised: Finding local modes In practice, the mode locations of a non-trivial target distribution and the

Original: Hessians are rarely available.
Revised: Hessians are rarely available.

Original: Pompe et al.
Revised: Pompe et al.

Original: (2020) describes a general approach to estimate thesequantities by running in parallel a sequence of the optimisers initiated at different starting points,and merging the optima found.
Revised: (2020) describes a general approach to estimate these quantities by running in parallel a sequence of the optimisers initiated at different starting points, and merging the optima found.

Original: This is done by minimising the objective − log p using the BFGSalgorithm (Nocedal and Wright, 2006), which returns both the local minima and the approximated
Revised: This is done by minimising the objective − log p using the BFGS algorithm (Nocedal and Wright, 2006), which returns both the local minima and the approximated

Original: Hessian at those points.
Revised: Hessian at those points.

Original: In our experiments, we run BFGS for at most 1000 iterations with each initialpoint. [Figure]
Revised: In our experiments, we run BFGS for at most 1000 iterations with each initial point.

Original: [Figure]

Original: Mode merging Starting the optimisation procedure from each initial point will lead to an endpoint close to the true local maxima, but are numerically different from each other.
Revised: Mode merging Starting the optimisation procedure from each initial point will lead to an end point close to the true local maxima, but are numerically different from each other.

Original: Pompe et al.(2020) proposed to merge two optima points m i and m j if the Mahalanobis distance weighted bythe averaged Hessians at those points is below a given threshold β .
Revised: Pompe et al. (2020) proposed to merge two optima points m i and m j if the Mahalanobis distance weighted by the averaged Hessians at those points is below a given threshold β .

Original: The full procedure is given in
Revised: The full procedure is given in

Original: Algorithm 2 for completeness.
Revised: Algorithm 2 for completeness.

Original: Choosing starting points The BFGS can be initiated from either a random sample uniformlydrawn on a product of intervals [ L 1 , U 1 ] ×· · ·× [ L d , U d ] in X , or simply the training set { x train ,i } n train i =1 .
Revised: Choosing starting points The BFGS can be initiated from either a random sample uniformly drawn on a product of intervals [ L 1 , U 1 ] ×· · ·× [ L d , U d ] in X , or simply the training set { x train ,i } n train i =1 .

Original: The first approach will allow modes not covered by the training data to be detected, and the secondapproach can lead to faster convergence of the optimisation algorithm when the modes of Q and Poverlaps.
Revised: The ﬁrst approach will allow modes not covered by the training data to be detected, and the second approach can lead to faster convergence of the optimisation algorithm when the modes of Q and P overlaps.

Original: To combine the best of the two worlds with the same computational budget, we initialisethe points by a set of size n train , half of which is drawn randomly drawn from the training set and theother half initialised uniformly from [ L 1 , U 1 ] × · · · × [ L d , U d ] .
Revised: To combine the best of the two worlds with the same computational budget, we initialise the points by a set of size n train , half of which is drawn randomly drawn from the training set and the other half initialised uniformly from [ L 1 , U 1 ] × · · · × [ L d , U d ] .

Original: D Experimental details 
Revised: E Experimental details E.1 Multivariate Gaussian mixture: supplementary plots

Original: D.1 Multivariate Gaussian mixture: supplementary plots

Original: In correct mixing ratios in 1 dimension We repeat the same experiment in Figure 1, where,instead of varying ∆ , we fix ∆ = 6 and draw samples from the same mixture but with differentmixing weights π q .
Revised: In correct mixing ratios in 1 dimension We repeat the same experiment in Figure 1, where, instead of varying ∆ , we ﬁx ∆ = 6 and draw samples from the same mixture but with different mixing weights π q .

Original: The mixing weight for the target distribution is kept at π p = 0 .
Revised: The mixing weight for the target distribution is kept at π p = 0 .

Original: 5 as before.
Revised: 5 as before.

Original: Weobserve that the problem of low power of KSD test and KSDA GG test persists even if the samples aredrawn from both components but with a different weight; see Figure 3 in Sec. 5.
Revised: We observe that the problem of low power of KSD test and KSDA GG test persists even if the samples are drawn from both components but with a different weight; see Figure 3 in Sec. 5.

Original: Level and power experiments in 50 dimensions We include supplementary experiments wherewe study the power and level of pKSD.
Revised: Level and power experiments in 50 dimensions We include supplementary experiments where we study the power and level of pKSD.

Original: The target distribution is the same in the multivariate Gaussian mixture example in Sec.
Revised: The target distribution is the same in the multivariate Gaussian mixture example in Sec.

Original: 5 in 50 dimensions, with density p ( x ) 
Revised: 5 in 50 dimensions, with density p ( x ) ∝ π p exp

Original: [-∝ π p exp (cid:0) − 
 ∥ x ∥ 22 (cid:1) +(1 −-] π p ) exp (cid:0) − 1 2 ∥ x − ∆ e 1 ∥ 2 2 (cid:1) , where π p = 0 . 5 , ∆ = 6 , and e 1 ∈ R
Revised: [-∝ π p exp (cid:0) − 
 ∥ x ∥ 22 (cid:1) +(1 −-][Equation] π p ) exp (cid:0) − 1 2 ∥ x − ∆ e 1 ∥ 2 2 (cid:1) , where π p = 0 . 5 , ∆ = 6 , and e 1 ∈ R

Original: d is a vector with 1 in the firstcoordinate and 0 in others.
Revised: d is a vector with 1 in the ﬁrst coordinate and 0 in others.

Original: Samples are drawn either from the same distribution (level experiment), orfrom only the left component (power experiment).
Revised: Samples are drawn either from the same distribution (level experiment), or from only the left component (power experiment).

Original: The probability of rejection over 100 repetitions isplotted in Fig.
Revised: The probability of rejection over 100 repetitions is plotted in Fig.

Original: We can see that under the null hypothesis, all tests have the prescribed test levelα = 0 .
Revised: We can see that under the null hypothesis, all tests have the prescribed test level α = 0 .

Original: When samples completely miss one mode, pKSD achieves a significantly higher powerfor all sample sizes, whereas the power of KSD and KSDA GG remains close to the level. 
Revised: When samples completely miss one mode, pKSD achieves a signiﬁcantly higher power for all sample sizes, whereas the power of KSD and KSDA GG remains close to the level. [Figure] R e j [Figure]

Revised: c t i o n R a t e (

Original: D.2 Sensor network localisation
Revised: P o w e r ) KSD pKSD KSDAGG E.2 Sensor network localisation

Original: The Bayesian inference task for sensor network localisation follows the same setup in Tak et al.(2018).
Revised: The Bayesian inference task for sensor network localisation follows the same setup in Tak et al.

Original: This is a modified version of the example from Ihler et al.
Revised: This is a modiﬁed version of the example from Ihler et al.

Original: (2005), which has been usedas a benchmark for MCMC samplers designed for multimodal targets (Pompe et al., 2020; Ahnet al., 2013; Lan et al., 2014).
Revised: (2005), which has been used as a benchmark for MCMC samplers designed for multimodal targets (Pompe et al., 2020; Ahn et al., 2013; Lan et al., 2014).

Original: Here, six sensors x 1 , . . .
Revised: Here, six sensors x 1 , . . .

Original: , x 6 are located in [0 , 1] 2 , four of which haveunknown locations and the remaining two are known.
Revised: , x 6 are located in [0 , 1] 2 , four of which have unknown locations and the remaining two are known.

Original: We observe distance y ij between two sensorsx i , x j with probability exp( −∥ x i − x j ∥ 22 / (2 × 0 .
Revised: We observe distance y ij between two sensors x i , x j with probability exp( −∥ x i − x j ∥ 2 2 / (2 × 0 .

Original: If observed, the distance follows a Gaussiandistribution y ij ∼ N ( ∥ x
Revised: If observed, the distance follows a Gaussian distribution y ij ∼ N ( ∥ x

Original: i − x j ∥ , 0 . 02 2 ) .
Revised: i − x j ∥ , 0 . 02 2 ) .

Original: As in Tak et al.
Revised: As in Tak et al.

Original: (2018), we use a diffuse bivariate Gaussianprior distribution N (0 , 10 2 I 2 ) for each x i .
Revised: (2018), we use a diffuse bivariate Gaussian prior distribution N (0 , 10 2 I 2 ) for each x i .

Original: Let w ij be the binary random variable for which w ij = 1if the distance y ij is observed and 0 otherwise.
Revised: Let w ij be the binary random variable for which w ij =if the distance y ij is observed and 0 otherwise.

Original: The full posterior is
Revised: The full posterior is

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Original:  where w = { w ij } , y = { y ij } and
Revised: ) , where w = { w ij } , y = { y ij } and

Original: [Equation]
Revised: [Equation]

Original: ij
Revised: ij

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]