Original: Targeted Separation and Convergence with Kernel Discrepancies
Revised: Targeted Separation and Convergence with Kernel Discrepancies

Original: Abstract
Revised: Abstract

Original: Kernel Stein discrepancies (KSDs) are maximum mean discrepancies (MMDs) that leverage the score information of distributions, and have grown central to a wide range of applications.
Revised: Kernel Stein discrepancies (KSDs) are maximum mean discrepancies (MMDs) that leverage the score information of distributions, and have grown central to a wide range of applications.

Original: In most settings, these MMDs are required to ( i ) separate a target P from other probability measures or even ( ii ) control weak convergence to
Revised: In most settings, these MMDs are required to ( i ) separate a target P from other probability measures or even ( ii ) control weak convergence to

Original: P .
Revised: P .

Original: In this article we derive new sufﬁcient and necessary conditions that substantially broaden the known conditions for KSD separation and convergence control and develop the ﬁrst KSDs known to metrize weak convergence to P .
Revised: In this article we derive new sufﬁcient and necessary conditions that substantially broaden the known conditions for KSD separation and convergence control, and develop the ﬁrst KSDs known to metrize weak convergence to P .

Original: Along the way, we highlight the implications of our results for hypothesis testing, measuring and improving sample quality, and sampling with Stein variational gradient descent
Revised: Along the way, we highlight the implications of our results for hypothesis testing, measuring and improving sample quality, and sampling with Stein variational gradient descent

Original:  Introduction
Revised:  Introduction

Original: The Langevin kernel Stein discrepancy (KSD) is a maximum mean discrepancy (MMD) tailored to the target distribution P that relies on its score s p ≡ ∂ log p to measure the integration error between
Revised: The Langevin kernel Stein discrepancy (KSD) is a maximum mean discrepancy (MMD) tailored to the target distribution P that relies on its score s p ≡ ∂ log p to measure the integration error between

Original: P and alternative distributions.
Revised: P and alternative distributions.

Original: Kernel-based discrepancy measures are widely used for hypothesis testing [16, 21, 9], sampler selection and tuning [14], parameter estimation [4, 2, 11], generalized
Revised: Kernel-based discrepancy measures are widely used for hypothesis testing [16, 21, 9], sampler selection and tuning [14], parameter estimation [4, 2, 11], generalized

Original: Bayesian inference [8, 24, 23, 10], discrete approximation and numerical integration [7, 6, 3], control variate design [26, 25], compression [27], and bias correction [20, 17, 27].
Revised: Bayesian inference [8, 24, 23, 10], discrete approximation and numerical integration [7, 6, 3], control variate design [26, 25], compression [27], and bias correction [20, 17, 27].

Original: Each MMD uses a kernel function to measure the integration error between a pair of probability measures Q and P , and, in each setting above, their successful application relies on either P -separation , that is MMD(Q , P) > 0 whenever Q (cid:54) =
Revised: Each MMD uses a kernel function to measure the integration error between a pair of probability measures Q and P , and, in each setting above, their successful application relies on either P -separation , that is MMD(Q , P) > 0 whenever Q (cid:54) =

Original: P , or P -convergence control , namely MMD(Q n , P) →implies
Revised: P , or P -convergence control , namely MMD(Q n , P) →implies

Original: Q n → P weakly.
Revised: Q n → P weakly.

Original: Unfortunately, these properties have so far only been established under overly restrictive assumptions, e.g., for Q with continuously differentiable log densities [9, 21, 2], for
Revised: Unfortunately, these properties have so far only been established under overly restrictive assumptions, e.g., for Q with continuously differentiable log densities [9, 21, 2], for

Original: P with strongly log concave tails and Lipschitz log density gradients [14], or for bounded kernels which rarely occur in the context of score-based MMD kernels [32, 31, 30, 29].
Revised: P with strongly log concave tails and Lipschitz log density gradients [14], or for bounded kernels which rarely occur in the context of score-based MMD kernels [32, 31, 30, 29].

Original: In this work, by ﬁxing P as the target measure and allowing Q to vary, we establish new broadly applicable conditions for P -separation in Section 2, and P -convergence control in Section 3.
Revised: In this work, by ﬁxing P as the target measure and allowing Q to vary, we establish new broadly applicable conditions for P -separation in Section 2, and P -convergence control in Section 3.

Original: We begin by extending the usual notions of maximum mean discrepancy and kernel Stein discrepancy to accommodate both arbitrary probability measures Q and unbounded kernels.
Revised: We begin by extending the usual notions of maximum mean discrepancy and kernel Stein discrepancy to accommodate both arbitrary probability measures Q and unbounded kernels.

Original: Throughout we use the standard notation summarized in Appendix A.1.
Revised: Throughout we use the standard notation summarized in Appendix A.1.

Original: Kernel Stein discrepancies Consider a (reproducing) kernel k on X with reproducing kernel
Revised: Kernel Stein discrepancies Consider a (reproducing) kernel k on X with reproducing kernel

Original: Hilbert space H k [1, 28].
Revised: Hilbert space H k [1, 28].

Original: We will employ the following generalized deﬁnition of MMD, for any target measure in the set of embeddable probability measures
Revised: We will employ the following generalized deﬁnition of MMD, for any target measure in the set of embeddable probability measures

Original: P ∈ P
Revised: P ∈ P

Original: H k ≡ { Q ∈ P : H k ⊂ L 1 (Q) } :
Revised: H k ≡ { Q ∈ P [Equation] ⊂ L [Equation]

Revised: Q) } :

Original: [Equation]
Revised: [Equation]

Original: Indeed, traditionally MMD is deﬁned as the worst-case integration error across test functions in the whole RKHS unit norm ball B k [16].
Revised: Indeed, traditionally MMD is deﬁned as the worst-case integration error across test functions in the whole RKHS unit norm ball B k [16].

Original: However, the expression Q h − P h is not well deﬁned when either (i) both Q h and P h are inﬁnite or (ii) h is not integrable under Q .
Revised: However, the expression Q h − P h is not well deﬁned when either (i) both Q h and P h are inﬁnite or (ii) h is not integrable under Q .

Original: Unfortunately, both of these cases can occur when k is unbounded.
Revised: Unfortunately, both of these cases can occur when k is unbounded.

Original: Similarly we extend the integral probability metric deﬁnition of KSD associated to the score-based Langevin Stein operator S p , which corresponds to the MMD of a Stein kernel k p under no additional assumption by Theorem A.2, see Appendix A.2 for details.
Revised: Similarly we extend the integral probability metric deﬁnition of KSD associated to the score-based Langevin Stein operator S p , which corresponds to the MMD of a Stein kernel k p under no additional assumption by Theorem A.2, see Appendix A.2 for details.

Original: Deﬁnition 1.1 (Kernel Stein discrepancy (KSD)) .
Revised: Deﬁnition 1.1 (Kernel Stein discrepancy (KSD)) .

Original: For X = R d , a target P ∈ P with density p > 0 , and matrix-valued base kernel K for which S p ( H K ) exists.
Revised: For X = R d , a target P ∈ P with density p > 0 , and matrix-valued base kernel K for which S p ( H K ) exists.

Original: When S p ( H K ) ⊂
Revised: When S p ( H K ) ⊂

Original: L 1 (P) and
Revised: L 1 (P) and

Original: P( S p ( H K )) = { 0 } , we deﬁne the kernel Stein discrepancy KSD K, P : P → [0 , ∞ ] by
Revised: P( S p ( H K )) = { 0 } , we deﬁne the kernel Stein discrepancy KSD K, P : P → [0 , ∞ ] by

Original: [Equation]
Revised: [Equation]

Original:  Conditions for separating measures
Revised:  Conditions for separating measures

Original: Our ﬁrst goal is to identify when an MMD distinguishes P from other measures.
Revised: Our ﬁrst goal is to identify when an MMD distinguishes P from other measures.

Original: Given a set of probability measures M ⊂ P , we will say that k separates P from M when MMD k (Q , P) =implies that Q = P for any Q ∈ M .
Revised: Given a set of probability measures M ⊂ P , we will say that k separates P from M when MMD k (Q , P) =implies that Q = P for any Q ∈ M .

Original: When k separates P from all probability measures P we say simply that k is P -separating .
Revised: When k separates P from all probability measures P we say simply that k is P -separating .

Original: We will ﬁrst discuss restricted P -separation from distinguished subsets of measures M (cid:54)
Revised: We will ﬁrst discuss restricted P -separation from distinguished subsets of measures M (cid:54)

Original: =
Revised: =

Original: P in Sections 2.1 and 2.2, and then turn to general P -separation in Section 2.3.
Revised: P in Sections 2.1 and 2.2, and then turn to general P -separation in Section 2.3.

Original: 2.1 Bochner P -separation with MMDs
Revised: 2.1 Bochner P -separation with MMDs

Original: We ﬁrst characterize the kernels that separate P from the set P √ k of Bochner embeddable measures.
Revised: We ﬁrst characterize the kernels that separate P from the set P √ k of Bochner embeddable measures.

Original: Theorem 2.1 (Bochner P -separation with MMDs) .
Revised: Theorem 2.1 (Bochner P -separation with MMDs) .

Original: Let k be a continuous kernel over a Radon space
Revised: Let k be a continuous kernel over a Radon space

Original: X .
Revised: X .

Original: Then k separates P ∈ P √ k from P √ k iff, for any sequence (Q n ) n ⊂ P √ k ,
Revised: Then k separates P ∈ P √ k from P √ k iff, for any sequence (Q n ) n ⊂ P √ k ,

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.1 exposes an important relationship between our two goals of separation and convergence control.
Revised: Theorem 2.1 exposes an important relationship between our two goals of separation and convergence control.

Original: In particular, when k is bounded, it implies that separating P from all probability measures is equivalent to controlling tight P -convergence , i.e., having Q n → P weakly whenever
Revised: In particular, when k is bounded, it implies that separating P from all probability measures is equivalent to controlling tight P -convergence , i.e., having Q n → P weakly whenever

Original: MMD k (Q n , P) → 0 and (Q n ) n is tight.
Revised: MMD k (Q n , P) → 0 and (Q n ) n is tight.

Original: 2.2 Score P -separation with KSDs
Revised: 2.2 Score P -separation with KSDs

Original: The standard practice in the KSD literature is to identify easily-veriﬁed properties of the base kernel K , target P , and alternative measure Q that ensure separation.
Revised: The standard practice in the KSD literature is to identify easily-veriﬁed properties of the base kernel K , target P , and alternative measure Q that ensure separation.

Original: One class of KSD separation conditions applies to measures that ﬁnitely integrate the score s p but additionally requires Q to have a continuously differentiable log-density [9, 2].
Revised: One class of KSD separation conditions applies to measures that ﬁnitely integrate the score s p but additionally requires Q to have a continuously differentiable log-density [9, 2].

Original: Our next result, removes the extraneous continuity conditions and extends P -separation to all measures Q ∈ P s p under a standard separating assumption on the base kernel, D 1 L 1 ( R d ) -characteristicness [30], that covers all of the translation-invariant base kernels commonly used with KSDs including Gaussian, inverse multiquadric (IMQ), log inverse, sech, Matérn, B-spline, and Wendland’s compactly supported kernels.
Revised: Our next result, removes the extraneous continuity conditions and extends P -separation to all measures Q ∈ P s p under a standard separating assumption on the base kernel, D 1 L 1 ( R d ) -characteristicness [30], that covers all of the translation-invariant base kernels commonly used with KSDs including Gaussian, inverse multiquadric (IMQ), log inverse, sech, Matérn, B-spline, and Wendland’s compactly supported kernels.

Original: Theorem 2.2 (Score P -separation with KSDs) .
Revised: Theorem 2.2 (Score P -separation with KSDs) .

Original: Suppose a matrix-valued kernel K with H K ⊂ Cb ( R d ) is D 1 L 1 ( R d ) -characteristic. If P ∈ P K, 0 , 1 then k p separates P from P s p .
Revised: Suppose a matrix-valued kernel K with H K ⊂ C

Original: Goodness-of-ﬁt testing In goodness-of-ﬁt (GOF) testing, one uses a sequence of datapoints
Revised: Goodness-of-ﬁt testing In goodness-of-ﬁt (GOF) testing, one uses a sequence of datapoints

Original: X 1 , . . .
Revised: X 1 , . . .

Original: X n generated from a Markov chain to test whether the chain’s stationary distribution
Revised: X n generated from a Markov chain to test whether the chain’s stationary distribution

Original: Q coincides with a target distribution P .
Revised: Q coincides with a target distribution P .

Original: KSDs with D 1 L 1 -characteristic translation-invariant base kernels are commonly used as GOF test statistics, and such tests are known to consistently reject
Revised: KSDs with D 1 L 1 -characteristic translation-invariant base kernels are commonly used as GOF test statistics, and such tests are known to consistently reject

Original: Q (cid:54) =
Revised: Q (cid:54) =

Original: P whenever KSD(Q , P) > 

Revised: P whenever KSD(Q , P) >

Original: However, prior to this work, the separating condition
Revised: However, prior to this work, the separating condition

Original: KSD(Q , P) > 0 had only been established for a restricted class of alternatives (continuous Q ∈ P √ k p with differentiable log densities satisfying Q( (cid:107)
Revised: KSD(Q , P) > 0 had only been established for a restricted class of alternatives (continuous Q ∈ P √ k p with differentiable log densities satisfying Q( (cid:107)

Original: s p − s q (cid:107) ) < ∞ , [2, Prop. 1]) or a restricted class of targets ( P with Lipschitz s p and strongly log concave tails, [14, Thm. 7]).
Revised: s p − s q (cid:107) ) < ∞ , [2, Prop. 1]) or a restricted class of targets ( P with Lipschitz s p and strongly log concave tails, [14, Thm. 7]).

Original: The former restriction excludes discrete and discontinuous Q , as well as Q with tails heavier than P or non-differentiable densities.
Revised: The former restriction excludes discrete and discontinuous Q , as well as Q with tails heavier than P or non-differentiable densities.

Original: Meanwhile, the latter restriction excludes P with tails heavier than or lighter than a Gaussian.
Revised: Meanwhile, the latter restriction excludes P with tails heavier than or lighter than a Gaussian.

Original: Theorem 2.2 in the present work ensures that KSD(Q , P) > 0 for any P ∈ P K, 0 and Q ∈ P s p .
Revised: Theorem 2.2 in the present work ensures that KSD(Q , P) > 0 for any P ∈ P K, 0 and Q ∈ P s p .

Original: In particular, this accommodates discontinuous or non-smooth Q and all targets P for which the KSD
Revised: In particular, this accommodates discontinuous or non-smooth Q and all targets P for which the KSD

Original: Eq.
Revised: Eq.

Original: (2) is deﬁned.
Revised: (2) is deﬁned.

Original: Moreover, Theorem 2.2 holds for all D 1 L 1 -characteristic kernels, a strict superset of the C 10 -universal kernels [5, Def. 4.1] assumed in prior work.
Revised: Moreover, Theorem 2.2 holds for all D 1 L 1 -characteristic kernels, a strict superset of the C 10 -universal kernels [5, Def. 4.1] assumed in prior work.

Original: 2.3 General P -separation
Revised: 2.3 General P -separation

Original: The results in the preceding sections only yield general P -separation when applied to bounded kernels, and indeed this has been the standard in much of the MMD literature [32, 31, 30, 29].
Revised: The results in the preceding sections only yield general P -separation when applied to bounded kernels, and indeed this has been the standard in much of the MMD literature [32, 31, 30, 29].

Original: To accommodate the unbounded Stein kernels that often arise in KSDs, our next deﬁnition and result provide a new, convenient means to check that unbounded kernels separate P from P .
Revised: To accommodate the unbounded Stein kernels that often arise in KSDs, our next deﬁnition and result provide a new, convenient means to check that unbounded kernels separate P from P .

Original: Deﬁnition 2.3 (Bounded P -separating property) .
Revised: Deﬁnition 2.3 (Bounded P -separating property) .

Original: We say a set of functions F is bounded P -separating if L ∞ ∩ F is P -separating, i.e., if Q ∈ P and Q h
Revised: We say a set of functions F is bounded P -separating if L ∞ ∩ F is P -separating, i.e., if Q ∈ P and Q h

Original: = P h for all h ∈ L ∞
Revised: = P h for all h ∈ L ∞

Original: ∩ F then Q = P .
Revised: ∩ F then Q = P .

Original: Theorem 2.4 (Controlling tight convergence with bounded separation) .
Revised: Theorem 2.4 (Controlling tight convergence with bounded separation) .

Original: If H k is bounded Pseparating, then k is P -separating and controls tight P -convergence.
Revised: If H k is bounded Pseparating, then k is P -separating and controls tight P -convergence.

Original: Our next result applies this strategy and shows that all of the translation-invariant base kernels commonly used with KSDs, including Gaussian, IMQ, log inverse, sech, Matérn, B-spline, and
Revised: Our next result applies this strategy and shows that all of the translation-invariant base kernels commonly used with KSDs, including Gaussian, IMQ, log inverse, sech, Matérn, B-spline, and

Original: Wendland’s compactly supported kernels, contain a sub-RKHS of bounded (in fact rapidly decaying) functions that is P -separating.
Revised: Wendland’s compactly supported kernels, contain a sub-RKHS of bounded (in fact rapidly decaying) functions that is P -separating.

Original: Theorem 2.5 (Controlling tight convergence with KSDs) .
Revised: Theorem 2.5 (Controlling tight convergence with KSDs) .

Original: For any translation invariant k with a spectral density bounded away from zero on compact sets, and s.t., H k ⊂ C 1 .
Revised: For any translation invariant k with a spectral density bounded away from zero on compact sets, and s.t., H k ⊂ C 1 .

Original: Deﬁne the tilted kernel k a ( x, y )
Revised: Deﬁne the tilted kernel k a ( x, y )

Original: =
Revised: =

Original: a ( x ) k
Revised: a ( x ) k

Original: ( x, y ) a ( y ) for each strictly positive a ∈ C 1 .
Revised: ( x, y ) a ( y ) for each strictly positive a ∈ C 1 .

Original: If P ∈ P k, 0 and (cid:107) s p (cid:107) has at most root exponential growth, 2 then the Stein kernel induced by k is bounded P -separating and controls tight P -convergence.
Revised: If P ∈ P k, 0 and (cid:107) s p (cid:107) has at most root exponential growth, 2 then the Stein kernel induced by k is bounded P -separating and controls tight P -convergence.

Original: Moreover, if P ∈ P ka, 0 and a , ∂a , and a (cid:107) s p (cid:107) have at most root exponential growth, then the Stein kernel induced by k a is bounded P -separating and controls tight P -convergence.
Revised: Moreover, if P ∈ P ka, 0 and a , ∂a , and a (cid:107) s p (cid:107) have at most root exponential growth, then the Stein kernel induced by k a is bounded P -separating and controls tight P -convergence.

Original: Goodness-of-ﬁt testing, continued Theorem 2.5 extends the reach of KSD GOF testing by guaranteeing KSD(Q , P) > 0 for all alternatives Q whenever (cid:107) s p (cid:107) has at most root exponential growth.
Revised: Goodness-of-ﬁt testing, continued Theorem 2.5 extends the reach of KSD GOF testing by guaranteeing KSD(Q , P) > 0 for all alternatives Q whenever (cid:107) s p (cid:107) has at most root exponential growth.

Original: Since the Stein kernels of Theorem 2.5 are also bounded P -separating, the same consistency guarantees immediately extend to the computationally efﬁcient stochastic KSDs of [15].
Revised: Since the Stein kernels of Theorem 2.5 are also bounded P -separating, the same consistency guarantees immediately extend to the computationally efﬁcient stochastic KSDs of [15].

Original:  Conditions for convergence control
Revised:  Conditions for convergence control

Original: Having derived sufﬁcient conditions on the RKHS to separate measures and control tight convergence, we now present sufﬁcient conditions to ensure that an MMD controls weak convergence to P .
Revised: Having derived sufﬁcient conditions on the RKHS to separate measures and control tight convergence, we now present sufﬁcient conditions to ensure that an MMD controls weak convergence to P .

Original: Hereafter, we will say that k controls weak convergence to P or controls P -convergence whenever
Revised: Hereafter, we will say that k controls weak convergence to P or controls P -convergence whenever

Original: MMD k (Q n , P) → 0 implies Q n → P weakly.
Revised: MMD k (Q n , P) → 0 implies Q n → P weakly.

Original: Moreover, we will say that k enforces tightness whenever MMD k (Q n , P) → 0 implies that (Q n ) n is tight.
Revised: Moreover, we will say that k enforces tightness whenever MMD k (Q n , P) → 0 implies that (Q n ) n is tight.

Original: Enforcing tightness is central to our developments as, if k controls tight weak convergence to P and enforces tightness, then it also controls weak convergence to P .
Revised: Enforcing tightness is central to our developments as, if k controls tight weak convergence to P and enforces tightness, then it also controls weak convergence to P .

Original: We begin by introducing a new sufﬁcient condition to ensure that
Revised: We begin by introducing a new sufﬁcient condition to ensure that

Original: MMDs enforce tightness.
Revised: MMDs enforce tightness.

Original: Theorem 3.1 ( P -dominating indicators enforces tightness) .
Revised: Theorem 3.1 ( P -dominating indicators enforces tightness) .

Original: Consider a set of functions F ⊂ L 1 (P) .
Revised: Consider a set of functions F ⊂ L 1 (P) .

Original: We say that F P -dominates indicators if, for each (cid:15) > 0 , there exists a compact set S ⊂ X and a function h ∈ F that satisfy
Revised: We say that F P -dominates indicators if, for each (cid:15) > 0 , there exists a compact set S ⊂ X and a function h ∈ F that satisfy

Original: [Equation]
Revised: [Equation]

Original: If P ∈ P H k and H k P -dominates indicators then (Q n ) n is tight whenever MMD k (Q n , P) → 0 .
Revised: If P ∈ P H k and H k P -dominates indicators then (Q n ) n is tight whenever MMD k (Q n , P) → 0 .

Original: Corollary 3.2 (Controlling P -convergence with KSDs) .
Revised: Corollary 3.2 (Controlling P -convergence with KSDs) .

Original: Under the conditions of Theorem 2.2 or 2.5, if H k p P -dominates indicators, then k p controls P -convergence.
Revised: Under the conditions of Theorem 2.2 or 2.5, if H k p P -dominates indicators, then k p controls P -convergence.

Original: Note prior work relied on a stronger coercive function condition to establish that KSDs enforce tightness [14, 6, 17].
Revised: Note prior work relied on a stronger coercive function condition to establish that KSDs enforce tightness [14, 6, 17].

Original: As a ﬁrst application of Corollary 3.2, we show that KSDs with IMQ base kernels enforce tightness and control convergence whenever the dissipativity rate of the target dominates the decay rate of the kernel.
Revised: As a ﬁrst application of Corollary 3.2, we show that KSDs with IMQ base kernels enforce tightness and control convergence whenever the dissipativity rate of the target dominates the decay rate of the kernel.

Original: Theorem 3.3 (IMQ KSDs control P -convergence) .
Revised: Theorem 3.3 (IMQ KSDs control P -convergence) .

Original: Consider a target measure P ∈ P with score s p ∈ C ( R d ) ∩ L 1 (P) .
Revised: Consider a target measure P ∈ P with score s p ∈ C ( R d ) ∩ L 1 (P) .

Original: If, for some dissipativity rate u > 1 / 2 and s, s 1 , s 2 > 0 , P satisﬁes the generalized dissipativity condition
Revised: If, for some dissipativity rate u > 1 / 2 and s, s 1 , s 2 > 0 , P satisﬁes the generalized dissipativity condition

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ R d .
Revised: for all x ∈ R d .

Original: [Equation]
Revised: [Equation]

Original: If k ( x, y )
Revised: If k ( x, y )

Original: = ( c 2 + (cid:107) x − y (cid:107) 2 ) − γ for c > 0 and γ ∈ (0 , 2 u − 1) , then H k p P -dominates indicators and enforces tightness.
Revised: = ( c 2 + (cid:107) x − y (cid:107) 2 ) − γ for c > 0 and γ ∈ (0 , 2 u − 1) , then H k p P -dominates indicators and enforces tightness.

Original: If, in addition, (cid:107) s p (cid:107) has at most root exponential growth, then k p controls
Revised: If, in addition, (cid:107) s p (cid:107) has at most root exponential growth, then k p controls

Original: P -convergence.
Revised: P -convergence.

Original: Measuring and improving sample quality Because the KSD provides a computable quality measure that requires no explicit integration under P , KSDs are now commonly used to select and tune MCMC sampling algorithms [14], generate accurate discrete approximations to P [22, 6, 7, 12], compress Markov chain output [27], and correct for biased or off-target sampling [20, 17, 27].
Revised: Measuring and improving sample quality Because the KSD provides a computable quality measure that requires no explicit integration under P , KSDs are now commonly used to select and tune MCMC sampling algorithms [14], generate accurate discrete approximations to P [22, 6, 7, 12], compress Markov chain output [27], and correct for biased or off-target sampling [20, 17, 27].

Original: Each of these applications relies on KSD convergence control, but past work only established convergence control for P with Lipschitz s p and strongly log concave tails ([14, Lem. 16]; [6, Thm. 3]; [18,
Revised: Each of these applications relies on KSD convergence control, but past work only established convergence control for P with Lipschitz s p and strongly log concave tails ([14, Lem. 16]; [6, Thm. 3]; [18,

Original: Thm. 3.2]).
Revised: Thm. 3.2]).

Original: Notably, these conditions imply generalized dissipativity Eq.
Revised: Notably, these conditions imply generalized dissipativity Eq.

Original: (5) with u = 1 but exclude all P with tails lighter than a Gaussian.
Revised: (5) with u = 1 but exclude all P with tails lighter than a Gaussian.

Original: Corollary 3.2 and Theorem 3.3 signiﬁcantly relax these requirements by providing convergence control for all dissipative P with lighter-than-Laplace tails.
Revised: Corollary 3.2 and Theorem 3.3 signiﬁcantly relax these requirements by providing convergence control for all dissipative P with lighter-than-Laplace tails.

Original: Much of the difﬁculty in analyzing KSDs stems from the fact that all known convergence-controlling
Revised: Much of the difﬁculty in analyzing KSDs stems from the fact that all known convergence-controlling

Original: KSDs are based on unbounded Stein kernels k p .
Revised: KSDs are based on unbounded Stein kernels k p .

Original: As a second illustration of the power of Corollary 3.2,
Revised: As a second illustration of the power of Corollary 3.2,

Original: Theorem 3.4 develops the ﬁrst KSDs known to metrize P -convergence (i.e., KSD(Q n , P) → 0 ⇐⇒
Revised: Theorem 3.4 develops the ﬁrst KSDs known to metrize P -convergence (i.e., KSD(Q n , P) → 0 ⇐⇒

Original: Q n → P weakly), by constructing bounded convergence-controlling Stein kernels.
Revised: Q n → P weakly), by constructing bounded convergence-controlling Stein kernels.

Original: Theorem 3.4 (Metrizing P -convergence with bounded Stein kernels) .
Revised: Theorem 3.4 (Metrizing P -convergence with bounded Stein kernels) .

Original: Consider a target measure
Revised: Consider a target measure

Original: P ∈ P with score s p that, for some dissipativity rate u > 1 / 2 and s, s 1 , s 2 > 0 , satisﬁes the generalized dissipativity condition Eq.
Revised: P ∈ P with score s p that, for some dissipativity rate u > 1 / 2 and s, s 1 , s 2 > 0 , satisﬁes the generalized dissipativity condition Eq.

Original: Deﬁne the Stein kernel with base kernel K ( x, y ) =
Revised: Deﬁne the Stein kernel with base kernel K ( x, y ) =

Original: diag (cid:0) a ( (cid:107) x
Revised: diag (cid:0) a ( (cid:107) x

Original: (cid:107) )
Revised: (cid:107) )

Original: ( x i y
Revised: ( x i y

Original: i + k ( x, y ))
Revised: i + k ( x, y ))

Original: a ( (cid:107) y (cid:107) ) (cid:1) , for k characteristic to D 1 L 1 with H k ⊂ C 10 and a ( (cid:107) x (cid:107) )
Revised: a ( (cid:107) y (cid:107) ) (cid:1) , for k characteristic to D 1 L 1 with H k ⊂ C 10 and a ( (cid:107) x (cid:107) )

Original: ≡ ( c 2 + (cid:107) x (cid:107) 2 ) − γ a tilting function with c > 0 and γ ≤ u .
Revised: ≡ ( c 2 + (cid:107) x (cid:107) 2 ) − γ a tilting function with c > 0 and γ ≤ u .

Original: The following statements hold true:
Revised: The following statements hold true:

Original: If P ∈ P K, 0 , then H k p P -dominates indicators and enforces tightness.
Revised: If P ∈ P K, 0 , then H k p P -dominates indicators and enforces tightness.

Original: If P ∈ P K, 0 , γ ≥ 0 , and (cid:107) s p ( x ) (cid:107) ≤ ( c 2 + (cid:107) x (cid:107) 2 ) γ , then k p is bounded P -separating and controls P -convergence.
Revised: If P ∈ P K, 0 , γ ≥ 0 , and (cid:107) s p ( x ) (cid:107) ≤ ( c 2 + (cid:107) x (cid:107) 2 ) γ , then k p is bounded P -separating and controls P -convergence.

Original: If (cid:107) s p ( x ) (cid:107) · (cid:107) x (cid:107) ≤ ( c 2 + (cid:107) x (cid:107) 2 ) γ and s p ∈ C ,
Revised: If (cid:107) s p ( x ) (cid:107) · (cid:107) x (cid:107) ≤ ( c 2 + (cid:107) x (cid:107) 2 ) γ and s p ∈ C ,

Original: then H k p ⊂ C b and k p metrizes Pconvergence.
Revised: then H k p ⊂ C b and k p metrizes Pconvergence.

Original: Sampling with Stein Variational
Revised: Sampling with Stein Variational

Original: Gradient
Revised: Gradient

Original: Descent
Revised: Descent

Original: Stein variational gradient descent (SVGD) is a popular technique for approximating a target distribution P with a collection of n representative particles.
Revised: Stein variational gradient descent (SVGD) is a popular technique for approximating a target distribution P with a collection of n representative particles.

Original: The algorithm proceeds by iteratively updating the locations of the particles according to a simple rule determined by a user-selected KSD.
Revised: The algorithm proceeds by iteratively updating the locations of the particles according to a simple rule determined by a user-selected KSD.

Original: [19] showed that the SVGD approximation converges weakly to P as the number of particles and iterations tend to inﬁnity, provided that the chosen KSD controls P -convergence and that the Stein kernel is bounded.
Revised: [19] showed that the SVGD approximation converges weakly to P as the number of particles and iterations tend to inﬁnity, provided that the chosen KSD controls P -convergence and that the Stein kernel is bounded.

Original: However, prior to this work, no bounded convergence-controlling Stein kernels were known.
Revised: However, prior to this work, no bounded convergence-controlling Stein kernels were known.

Original: Theorem 3.4 therefore provides the ﬁrst instance of a Stein kernel satisfying the SVGD convergence assumptions of [19].
Revised: Theorem 3.4 therefore provides the ﬁrst instance of a Stein kernel satisfying the SVGD convergence assumptions of [19].

Original:  Conclusion
Revised:  Conclusion

Original: This article derived new sufﬁcient and necessary conditions for kernel discrepancies to enforce
Revised: This article derived new sufﬁcient and necessary conditions for kernel discrepancies to enforce

Original: P -separation and control P -convergence, with important consequences in many applications, as we have highlighted in the contexts of GOF, improving sample quality and Stein variational gradient descent.
Revised: P -separation and control P -convergence, with important consequences in many applications, as we have highlighted in the contexts of GOF, improving sample quality and Stein variational gradient descent.

Original: We characterized all MMDs that separate P from Bochner embeddable measures, proposed novel sufﬁcient conditions for separating all measures and enforcing tightness, strengthened all prior guarantees for KSD separation and convergence control on R d and derived the ﬁrst KSD known to exactly metrize (as opposed to strictly dominating) weak P -convergence on R d .
Revised: We characterized all MMDs that separate P from Bochner embeddable measures, proposed novel sufﬁcient conditions for separating all measures and enforcing tightness, strengthened all prior guarantees for KSD separation and convergence control on R d and derived the ﬁrst KSD known to exactly metrize (as opposed to strictly dominating) weak P -convergence on R d .

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Use unnumbered ﬁrst level headings for the acknowledgments.

Original: All acknowledgments go at the end of the paper before the list of references.

Original: Moreover, you are required to declare funding (ﬁnancial activities supporting the submitted work) and competing interests (related ﬁnancial activities outside the submitted work). More information about this disclosure can be found at: https: //neurips.cc/Conferences/2022/PaperInformation/FundingDisclosure . Do not include this section in the anonymized submission only in  the ﬁnal paper. You can use the ack environment provided in the style ﬁle to autmoatically hide this section in the anonymized submission.
Revised: AB and MG were supported by the Department of Engineering at the University of Cambridge and this material is based upon work supported by, or in part by, the U.S. Army Research Laboratory and the U.S. Army Research Ofﬁce and by the U.K. Ministry of Defence and under the EPSRC grant [EP/R018413/2]

Original: A Appendix A.1 Notation
Revised: A Appendix A.1 Notation

Original: Let P denote the set of (Borel) probability measures on a separable metric space X .
Revised: Let P denote the set of (Borel) probability measures on a separable metric space X .

Original: (cid:107)·(cid:107) and (cid:104)· , ·(cid:105) denote the Euclidean norm and inner product on R (cid:96) .
Revised: (cid:107)·(cid:107) and (cid:104)· , ·(cid:105) denote the Euclidean norm and inner product on R (cid:96) .

Original: For any function f : X → R (cid:96)  we let
Revised: For any function f : X → R (cid:96) , we let

Original: P f ≡ { Q ∈ P : (cid:107) f (cid:107) ∈
Revised: P f ≡ { Q ∈ P : (cid:107) f (cid:107) ∈

Original: L 1 (Q) } denote the set of probability measures that ﬁnitely integrate (cid:107) f (cid:107) , where L 1 (Q) denotes the seminormed space of functions with ﬁnite Q -integral.
Revised: L 1 (Q) } denote the set of probability measures that ﬁnitely integrate (cid:107) f (cid:107) , where L 1 (Q) denotes the seminormed space of functions with ﬁnite Q -integral.

Original: L ∞ will denote the normed space of bounded functions.
Revised: L ∞ will denote the normed space of bounded functions.

Original: Let C (cid:96) ( R d ) denote the space of (cid:96) -times continuously differentiable R d -valued functions on X (i.e., f ∈ C (cid:96) ( R d ) if the partial derivatives of order (cid:96) of f i exist and are continuous for i ∈
Revised: Let C (cid:96) ( R d ) denote the space of (cid:96) -times continuously differentiable R d -valued functions on X (i.e., f ∈ C (cid:96) ( R d ) if the partial derivatives of order (cid:96) of f i exist and are continuous for i ∈

Original: [ d ] ≡ { 1 , . . .
Revised: [ d ] ≡ { 1 , . . .

Original: , d } ).
Revised: , d } ).

Original: When d = 1 or (cid:96) = 0 we will use the abbreviations C (cid:96) ≡ C (cid:96) ( R 1 ) or C ( R d ) ≡ C 0 ( R d ) .
Revised: When d = 1 or (cid:96) = 0 we will use the abbreviations C (cid:96) ≡ C (cid:96) ( R 1 ) or C ( R d ) ≡ C 0 ( R d ) .

Original: We let ∂f denote the vector of partial derivatives of a function f , and, for each multi-index p , let ∂ p f denote the p -th partial derivatives of f .
Revised: We let ∂f denote the vector of partial derivatives of a function f , and, for each multi-index p , let ∂ p f denote the p -th partial derivatives of f .

Original: Decay requirements will appear as subscripts: C b ( R d ) , and C 0 ( R d ) will respectively denote the spaces of R d -valued continuous functions that are bounded, and vanishing at inﬁnity.
Revised: Decay requirements will appear as subscripts: C b ( R d ) , and C 0 ( R d ) will respectively denote the spaces of R d -valued continuous functions that are bounded, and vanishing at inﬁnity.

Original: Analogously, for each function h : X → [0 , ∞ ) , C h
Revised: Analogously, for each function h : X → [0 , ∞ ) , C h

Original: ( R d ) and C 0 ,h
Revised: ( R d ) and C 0 ,h

Original: ( R d ) respectively denote the spaces of R d -valued continuous functions f with f/ (1 + h ) bounded or vanishing at inﬁnity.
Revised: ( R d ) respectively denote the spaces of R d -valued continuous functions f with f/ (1 + h ) bounded or vanishing at inﬁnity.

Original: For any function of two arguments K ( y, x ) , we write K x ≡ K ( · , x ) .
Revised: For any function of two arguments K ( y, x ) , we write K x ≡ K ( · , x ) .

Original: For a reproducing kernel k we denote √ k : X → R the function x (cid:55)→ (cid:112) k
Revised: For a reproducing kernel k we denote √ k : X → R the function x (cid:55)→ (cid:112) k

Original: ( x, x ) .
Revised: ( x, x ) .

Original: Matrix-valued reproducing kernels are denoted K : R d × R d → R d × d , and the RKHS unit ball is denoted B K .
Revised: Matrix-valued reproducing kernels are denoted K : R d × R d → R d × d , and the RKHS unit ball is denoted B K .

Original: The
Revised: The

Original: Langevin Stein operator on R d -valued functions is denoted S p ( v ) ≡ 1p ∇ · (p v ) .
Revised: Langevin Stein operator on R d -valued functions is denoted S p ( v ) ≡ 1p ∇ · (p v ) .

Original: A.2 Extended deﬁnition of KSD
Revised: A.2 Extended deﬁnition of KSD

Original: Consider a (reproducing) kernel k on X with reproducing kernel Hilbert space H k [1, 28].
Revised: Consider a (reproducing) kernel k on X with reproducing kernel Hilbert space H k [1, 28].

Original: Traditionally, the associated kernel MMD is deﬁned as the worst-case integration error across test functions in the RKHS unit norm ball B k [16]:
Revised: Traditionally, the associated kernel MMD is deﬁned as the worst-case integration error across test functions in the RKHS unit norm ball B k [16]:

Original: [Equation]
Revised: [Equation]

Original: However, the expression Q h − P h is not well deﬁned when either (i) both Q h and P h are inﬁnite or (ii) h is not integrable under Q .
Revised: However, the expression Q h − P h is not well deﬁned when either (i) both Q h and P h are inﬁnite or (ii) h is not integrable under Q .

Original: Unfortunately, both of these cases can occur when k is unbounded as
Revised: Unfortunately, both of these cases can occur when k is unbounded as

Original: B k then necessarily contains an unbounded test function.
Revised: B k then necessarily contains an unbounded test function.

Original: Since we are interested in a ﬁxed target measure P , we address the ﬁrst issue by focusing on kernels with ﬁnitely P -integrable test functions, i.e., with B k ⊂ L 1 (P) .
Revised: Since we are interested in a ﬁxed target measure P , we address the ﬁrst issue by focusing on kernels with ﬁnitely P -integrable test functions, i.e., with B k ⊂ L 1 (P) .

Original: To address the second issue, we extend the MMD deﬁnition Eq.
Revised: To address the second issue, we extend the MMD deﬁnition Eq.

Original: (6) to all probability measures Q by taking the supremum only over the Q -integrable elements of B k , that is, h with either h + ≡ max ( h, 0)
Revised: (6) to all probability measures Q by taking the supremum only over the Q -integrable elements of B k , that is, h with either h + ≡ max ( h, 0)

Original: ∈
Revised: ∈

Original: L 1 (Q) or h − ≡ max ( − h, 0)
Revised: L 1 (Q) or h − ≡ max ( − h, 0)

Original: ∈
Revised: ∈

Original: L 1 (Q) .
Revised: L 1 (Q) .

Original: In fact, since B k is a symmetric set, considering only h with h + ∈ L 1 (Q) sufﬁces to ensure | Q h − P h | is well deﬁned and belongs to [0 , ∞ ] .
Revised: In fact, since B k is a symmetric set, considering only h with h + ∈ L 1 (Q) sufﬁces to ensure | Q h − P h | is well deﬁned and belongs to [0 , ∞ ] .

Original: Deﬁnition A.1 (Maximum mean discrepancy (MMD)) .
Revised: Deﬁnition A.1 (Maximum mean discrepancy (MMD)) .

Original: For a given kernel k , deﬁne the set of embeddable probability measures P H k ≡ { Q ∈ P :
Revised: For a given kernel k , deﬁne the set of embeddable probability measures P H k ≡ { Q ∈ P :

Original: H k ⊂ L 1 (Q) } .
Revised: H k ⊂ L 1 (Q) } .

Original: For any target measure P ∈ P H k , we deﬁne the maximum mean discrepancy MMD k ( · , P) :
Revised: For any target measure P ∈ P H k , we deﬁne the maximum mean discrepancy MMD k ( · , P) :

Original: P → [0 , ∞ ] by
Revised: P → [0 , ∞ ] by

Original: [Equation]
Revised: [Equation]

Original: Building on the Stein discrepancy formalism of [13] and the zero-mean reproducing kernel theory of [26], [9, 21, 14] concurrently developed a special class of score-based MMDs that can be computed without any explicit integration under the target P .
Revised: Building on the Stein discrepancy formalism of [13] and the zero-mean reproducing kernel theory of [26], [9, 21, 14] concurrently developed a special class of score-based MMDs that can be computed without any explicit integration under the target P .

Original: The Langevin KSD is deﬁned in terms of the
Revised: The Langevin KSD is deﬁned in terms of the

Original: Langevin Stein operator [13],
Revised: Langevin Stein operator [13],

Original: [Equation]
Revised: [Equation]

Original: as an integral probability metric over a subset of S p ( H K ) that ensures the supremum is well-deﬁned.
Revised: as an integral probability metric over a subset of S p ( H K ) that ensures the supremum is well-deﬁned.

Original: [Equation]
Revised: [Equation]

Original: However, S p ( v ) is often unbounded so that as before the expression Eq.
Revised: However, S p ( v ) is often unbounded so that as before the expression Eq.

Original: (8) need not be well deﬁned for all Q ∈ P .
Revised: (8) need not be well deﬁned for all Q ∈ P .

Original: To enable meaningful KSD evaluation for all probability measures, we follow the recipe of Deﬁnition A.1 to extend the deﬁnition of KSD to all Q ∈ P , see Deﬁnition 1.1.
Revised: To enable meaningful KSD evaluation for all probability measures, we follow the recipe of Deﬁnition A.1 to extend the deﬁnition of KSD to all Q ∈ P , see Deﬁnition 1.1.

Original: Under additional assumptions, like Bochner embeddability of P and Q and continuous differentiability of K and p , prior work showed that the KSD Eq.
Revised: Under additional assumptions, like Bochner embeddability of P and Q and continuous differentiability of K and p , prior work showed that the KSD Eq.

Original: (2) is equivalent to an MMD with a scalar Stein kernel k p and that S p ( H K ) deﬁnes a Stein RKHS H k p of scalar-valued functions [26, 9, 21, 14, 2].
Revised: (2) is equivalent to an MMD with a scalar Stein kernel k p and that S p ( H K ) deﬁnes a Stein RKHS H k p of scalar-valued functions [26, 9, 21, 14, 2].

Original: Our next result shows that no additional assumptions are necessary: KSD K, P (Q) =
Revised: Our next result shows that no additional assumptions are necessary: KSD K, P (Q) =

Original: MMD k p (Q , P) and S p ( H K )
Revised: MMD k p (Q , P) and S p ( H K )

Original: =
Revised: =

Original: H k p whenever the left-hand side quantities are well deﬁned.
Revised: H k p whenever the left-hand side quantities are well deﬁned.

Original: Theorem A.2 (KSD as MMD) .
Revised: Theorem A.2 (KSD as MMD) .

Original: Consider a target P ∈ P with density p > 0 and matrix-valued base kernel K for which S p ( H K ) exists.
Revised: Consider a target P ∈ P with density p > 0 and matrix-valued base kernel K for which S p ( H K ) exists.

Original: Then S p ( H K ) is the Stein RKHS H k p induced by the
Revised: Then S p ( H K ) is the Stein RKHS H k p induced by the

Original: Stein kernel k p ( x, y ) ≡ [Equation] [Equation]
Revised: Stein kernel k p ( x, y ) ≡ 1 p( x )p( y ) ∇ y ·

Original: [Equation]
Revised: ∇ x · (p( x ) K ( x, y )p

Original: , y )p( y )) .
Revised: ( y )) .

Original: Moreover, for target measures with zero-mean Stein RKHSes, i.e., for P in
Revised: Moreover, for target measures with zero-mean Stein RKHSes, i.e., for P in

Original: P K, 0 ≡ { Q ∈ P with q > 0:
Revised: P K, 0 ≡ { Q ∈ P with q > 0:

Original: S q ( H K ) exists,
Revised: S q ( H K ) exists,

Original: S q ( H K ) ⊂
Revised: S q ( H K ) ⊂

Original: L 1 (Q) , and Q( S q ( H K ))
Revised: L 1 (Q) , and Q( S q ( H K ))

Original: = { 0 }} , (9) the KSD matches the Stein kernel MMD: KSD K, P (Q) =
Revised: = { 0 }} , (9) the KSD matches the Stein kernel MMD: KSD K, P (Q) =

Original: MMD k p (Q , P) for all Q ∈ P .
Revised: MMD k p (Q , P) for all Q ∈ P .

Original: The zero-mean condition P ∈ P K, 0 ensures that all functions in the Stein RKHS integrate to zero under the target measure so that the KSD can be evaluated without any explicit integration under P .
Revised: The zero-mean condition P ∈ P K, 0 ensures that all functions in the Stein RKHS integrate to zero under the target measure so that the KSD can be evaluated without any explicit integration under P .

Original: Moreover, when Q embeds into the Stein RKHS, the KSD takes on its more familiar double integral form, i.e., if P ∈ P K, 0 and Q ∈ P H k p , then KSD 2 K, P (Q) =
Revised: Moreover, when Q embeds into the Stein RKHS, the KSD takes on its more familiar double integral form, i.e., if P ∈ P K, 0 and Q ∈ P H k p , then KSD 2 K, P (Q) =

Original: (cid:82)(cid:82) k p
Revised: (cid:82)(cid:82) k p

Original: ( x, y )dQ( x )dQ( y )
Revised: ( x, y )dQ( x )dQ( y )