Efficient Sampling on Riemannian Manifolds via Langevin MCMC
Abstract
We study the task of efficiently sampling from a Gibbs distribution dπ ∗ =
e − h d vol g over a Riemannian manifold M via (geometric) Langevin MCMC; this algorithm involves computing exponential maps in random Gaussian directions and is efficiently implementable in practice.
The key to our analysis of Langevin
MCMC is a bound on the discretization error of the geometric Euler-Murayama scheme, assuming ∇ h is Lipschitz and M has bounded sectional curvature.
Our error bound matches the error of Euclidean Euler-Murayama in terms of its stepsize dependence.
Combined with a contraction guarantee for the geometric Langevin
Diffusion under Kendall-Cranston coupling, we prove that the Langevin MCMC iterates lie within ε -Wasserstein distance of π ∗ after ˜ O
( ε − 2 ) steps, which matches the iteration complexity for Euclidean Langevin MCMC.
Our results apply in general settings where h can be nonconvex and M can have negative Ricci curvature.
Under additional assumptions that the Riemannian curvature tensor has bounded derivatives, and that π ∗ satisfies a CD ( · , ∞ ) condition, we analyze the stochastic gradient version of Langevin MCMC, and bound its iteration complexity by ˜ O ( ε − 2 ) as well.
Introduction
Stochastic differential equations (SDEs) offer a powerful formalism for studying diffusion processes,
Brownian motion and algorithms for sampling and optimization.
We study in particular the following geometric stochastic differential equation :
[Equation]
that evolves on a d -dimensional Riemannian manifold ( M, g ) .
Analogous to the Euclidean setting, the map β : x → T x M denotes a drift ( T x M is the tangent space at x ∈ M ), and dB gt denotes the standard Brownian motion on M [Hsu, 2002, Ch. 3].
The notation in (1.1) is a shorthand; we can define (1.1) more precisely as the unique diffusion process whose generator is the operator Lf = ⟨ ∇ f, β ⟩ + 12 ∆ f , where ∆ is the Laplace-Beltrami operator (see [Hsu, 2002, Proposition. 3.2.1]).
When the drift β ( x ) = − 12 ∇ h ( x ) (the gradient is taken w.r.t.
the manifold metric), the SDE (1.1) has an invariant distribution π ∗ with density e − h ( x ) , with respect to the Riemannian volume measure vol g ( x ) ; for M = R d with metric g ( x )
=
I , this reduces to the familiar result for Euclidean Langevin diffusion.
Using the Kendall-Cranston Coupling technique (Chapter 6.5, Hsu [2002]) together with a carefully constructed Lyapunov function by Eberle [2016], we can quantify the mixing rate for (1.1) in the 1-Wasserstein distance (w.r.t.
the manifold distance) under suitable regularity conditions.
However, the exact SDE is not implementable in practice, so we consider an MCMC algorithm using the Geometric Euler-Murayama [Piggott and Solo, 2016, Muniz et al., 2021] discretization of (1.1):
[Equation]
where Exp x denotes the exponential map for M , and ζ k is a standard Gaussian with respect to any orthonormal basis of T x k M .
It has long been known that the discrete-time process (1.2) converges to the SDE (1.1) in the limit as δ → 0 [Gangolli, 1964, Jørgensen, 1975, Hsu, 2002].
Motivation for studying the SDE (1.1)
Geometric SDEs such as (1.1) play a crucial role in the design and analysis of MCMC algorithms [Girolami and Calderhead, 2011, Patterson and Teh, 2013] that have had much success in solving
Bayesian problems on statistical manifolds.
Recently, it has been shown that the mirror Langevin
Algorithm on the Hessian manifold of a function can be significantly faster than its Euclidean counterpart [Zhang et al., 2020, Chewi et al., 2020, Li et al., 2022, Gatmiry and Vempala, 2022].
These SDEs also directly relate to the tasks of sampling and optimization on manifolds, where often a Lie group structure helps capture symmetries (e.g., the Grassmann manifold, SO ( n ) , O ( n ) , etc.)
[Moitra and Risteski, 2020, Piggott and Solo, 2016, Muniz et al., 2021].
Furthermore, in Lee and Vempala [2017, 2018], the authors propose fast algorithms for constrained sampling and volume estimation on polytopes by sampling from the Hessian manifold of a barrier function.
Overview of Main Contributions
Our first contribution in this paper is to provide a quantitative, non-asymptotic bound on the discretization error between (1.1) and (1.2).
We present this bound in Lemma 1 in Section 4.
In our bound, the expected-squared-distance between a single step of (1.2) and (1.1) over δ time is bounded by O ( δ 3 ) .
This δ 3 error scaling matches that of Euler Murayama in Euclidean space [Durmus and
Moulines, 2017].
We highlight that our bound is entirely explicit, and depends polynomially on dimension, sectional curvature of M , and the Lipschitz parameter for β – quantities which are intrisic to the manifold, and invariant to the choice of coordinate system.
Two recent papers, [Wang et al., 2020, Li et al., 2022] also look into a similar problem of bounding the discretization error of (1.2).
Wang et al.
[2020] analyze the error of (1.2), but rely on uniformly bounding certain derivatives of the densities along the sample path (see Assumption
[Wang et al., 2020]); even in Euclidean space, it is not clear whether such a bound exists, and whether it depends only polynomially on parameters such as dimension.
Li et al.
[2022] provide a quantitative bound on the bias of (1.2) on Hessian manifolds; they assume a bound on a “modified self-concordance” parameter that is not affine invariant and can be made arbitrarily large.
We discuss Lemma 1 in more detail , and sketch its proof in Section 4.
The proof relies on a careful construction of geometric Langevin Diffusion as the limit of (1.2), and may be of independent interest.
Our second contribution is to show that after ˜ O ( ε − 2 ) steps, the distribution of (1.2) is within ε 1-Wasserstein distance from the stationary distribution of (1.1).
We present this result in Theorem 1.
This ε dependence matches that of Euclidean Langevin MCMC [Durmus and Moulines, 2017].
The ˜ O in our iteration complexity hides polynomial dependency on dimension, sectional curvature,
Lipschitz-parameter of β , and 1 /α , where α can be viewed as “mixing rate of the exact SDE.”
Theorem 1 requires that β satisfy a manifold analog of the distant-dissipativity assumption (Assumptionwith m > L Ric / 2 , where − L Ric is a lower bound on the Ricci curvature of M ).
Assumption 2 is general enough to include cases when β = −∇ h for some nonconvex h or when M has negative Ricci curvature.
The catch is that in such cases, 1 /α can become very large; this is generally unavoidable, even on Euclidean space.
Distant-dissipativity assumption has often been used in the analysis of
Euclidean Langevin diffusion for non-log-concave distributions [Eberle, 2016, Bou-Rabee et al., 2020, Gorham et al., 2019, Cheng et al., 2020].
We highlight a computational sub-contribution that (1.2) only requires computing exponential maps in the direction of β plus a uniform Gaussian direction.
In many cases, exponential maps are efficiently computable, and this Theorem 1 leads to a computationally efficient sampling algorithm.
Our analysis extends naturally if one replaces (1.2) by a retraction step; if the retraction is of order 3 / or higher, the one-step error still scales as O ( δ 3 ) .
The question of how to construct good retraction maps is well studied in literature, and is somewhat orthogonal to our main objective, so we do not provide details here, and instead refer readers to [Absil and Malick, 2012, Absil et al., 2009].
Recently, Ahn and Chewi [2021], Gatmiry and Vempala [2022], Li and Erdogdu [2020] have considered a different discretization of (1.1), where one assumes that the endpoint of the geometric
Brownian motion dB gt can be sampled exactly (so that only the drift β is discretized).
Sampling the exact geometric Brownian motion can be done efficiently in special settings such as the sphere [Li and Erdogdu, 2020], but on a general manifold, this can be much more expensive than computing an exponential map; in such settings the Langevin MCMC algorithm based on (1.2) may be prefereable.
We do highlight, however, that a number of the above results provide error bounds in KL-divergence, which is tighter than the Wasserstein bound in Theorem 1.
We discuss assumptions and consequences of Theorem 1 in more detail and sketch its proof in Section 5.
The proof essentially combines our discretization analysis with a mixing result for (1.1) based on the Kendall-Cranston coupling and a Lyapunov function from [Eberle, 2016].
For our third contribution, we analyze the manifold analog of SGLD [Welling and Teh, 2011]:
[Equation]
The the difference between (2.1) and (1.2), is that at each step, β is replaced by a random vector field ˜ β k
which satisfies: (i) E
[˜ β k ] =
β ; and (ii) ∥
˜ β k ( x ) − β ( x ) ∥ ≤ σ almost surely.
We show that after ˜ O ( ε − 2 ) steps, (2.1) is within ε -2-Wasserstein distance from the stationary distribution of (1.1); thus the ε dependency does not degrade when replacing β by its stochastic estimate.
We present this result in Theorem 2.
For this analysis, we require Assumption 2, with m > L Ric / 2 and R = 0 ; this is a more restrictive condition than Theorem 1;
when β = − 12 ∇ h , this restriction is equivalent to the
CD ( · , ∞ ) condition
[Bakry et al., 2014]: ∇ 2 h + Ric ≻ 0 , where Ric is the Ricci curvature tensor 1 .
Though restrictive, the CD ( · , ∞ ) class of distributions is nonetheless interesting; it is the manifold analog of the class of log-concave densities on Euclidean space, and sampling from this class has been a topic of much recent interest.
In machine learning, the stochastic estimate of β can often be computed much more quickly than the exact β , e.g., when β
= − 2 ∇ f , where f is the empirical average of some loss over a large number of observations.
In such cases, (2.1) can be much faster than (1.2).
In Section 6 we discuss the assumptions and consequences of Theorem 2 in greater detail.
Though Theorem 2 requires more restrictive assumptions than Theorem 1, its proof differs from the proof of Theorem 1 in a significant way: we show mixing of the discrete-time process directly (instead of relying on mixing of the exact SDE).
This approach is necessary in order to take advantage of the fact that ˜ β k is an unbiased estimate β .
The key intermediate result for showing mixing under (2.1) is
Lemma 29.
We believe this lemma to be of independent interest as it quantifies the distance evolution between two arbitrary discrete-time stochastic processes (that may be unrelated to Gaussian noise and Brownian motion).
As an example, Mangoubi and Smith [2018] showed that ball walk mixes quickly on manifolds with positive sectional curvature.
Using Lemma 29, one can show the more general statement “ball walk mixes quickly on compact manifolds with positive Ricci curvature.”
Preliminaries: Key Assumptions and Notation
We state in this section the four key assumptions of this paper.
Our first assumption involves lower bounding Ricci curvature of the manifold M .
Let Ric denote the Ricci curvature tensor:
Assumption 1.
We assume that for all x ∈ M , u, u ∈ T x M , Ric( u, u ) ≥ − L Ric , for some L Ric ∈ R .
Intuitively, the more positive the Ricci curvature (i.e. the smaller the value of L Ric ), the faster geometric Brownian motion mixes.
Our second assumption is a natural generalization of the distant-dissipativity condition in the Euclidean setting.
It helps ensure that the drift traps the variable within a bounded region.
For any x, y ∈ M , let d( x, y ) denote their Riemannian distance.
Assumption 2.
We call a vector field β ( m, q, R ) - distant-dissipative if there exist constants m > 0 ,
R ≥ 0 , and q ∈ R such that, for all x, y satisfying d( x, y ) ≥ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0) =
x and γ (1) =
y , such that the inequality
[Equation]
holds, where Γ( · ; y → x ) denotes parallel transport from T y M to T x M along γ .
In addition, for all x, y satisfying d( x, y ) ≤ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0) =
x and γ (1) =
y , such that we have instead the inequality
[Equation]
For some intuition: a strictly convex function will have m > 0 , R = 0 , and q arbitrary.
Note that we do not require a unique geodesic between x, y .
For some intuition about Assumption 2: in the Euclidean setting, the first condition simplifies to ⟨ β ( y ) − β ( x ) , y − x
[Equation]
≤ − m ∥ y − x ∥ 22 , and the second condition simplifies to
[Equation]
We need our third and fourth assumptions for bounding the discretization error of (1.2).
Assumption 3 upper bounds the Lipschitz constant of β ; Assumption 4 lower bounds sectional curvature of M .
Assumption 3.
A vector field β is L ′ β
-Lipschitz if, for all x ∈ M and all v ∈ T x M , ∥∇ v β ( x ) ∥ ≤
L ′ β
∥ v ∥ .
In the (flat) Euclidean setting, Assumption 3 is equivalent to saying that β ( x ) being a L ′ β Lipschitz vector field.
Assumption 4.
Let R be the Riemannian curvature tensor of the manifold M .
We assume that there exists L R ∈ R + such that for all x ∈ M , and for all u, v, w, z ∈ T x M , ⟨ R ( u, v ) v, u ⟩ ≤
L R ∥ u
∥ 2 ∥ v ∥ 2 .
We now introduce additional notation that will be used throughout this paper.
We assume some background in Riemannian geometry, and freely use standard notation; we refer the reader to [Jost, 2008, Lee, 2006, Petersen, 2006] for an in depth treatment.
Readers may also find some works on
Riemannian optimization useful as additional context: [Bacák, 2014, Udriste, 2013, Absil et al., 2009,
Zhang and Sra, 2016, Boumal, 2022].
We use ∇ to denote the Levi Civita connection.
Given x, y ∈ M and v ∈ T x M , we use Γ( v ; x → y ) to denote parallel transport of v from x to y along their minimizing geodesic (if such a choice is not unique we will specify); we sometimes also use the more concise alternative notation Γ yx v :=
Γ( v ; x → y ) .
Given a general curve (possibly non-geodesic) γ : [0 , 1] → M , we will also use Γ γ ( t ) v to denote the parallel transport of v from γ (0) to γ (1) , along γ .
A basis F of the tangent space T x M at some point x ∈ M is an ordered tuple ( F 1 , . .
. , F d ) of vectors that span T x M .
We use Γ( F ; x → y ) :
= (Γ( F 1 ; x → y ) ...
Γ( F d ; x → y )) to denote the ordered tuple of parallel transport of the each of the basis vectors in F .
Given v ∈ R d and basis F of some T x M , we use v ◦ F as shorthand for (cid:80) di =1 v
i F i .
A distribution that we will see frequently in this paper is the one given by ξ ◦ E x , where ξ ∼ N (0 , I ) is a random vector in R d , and x ∈ M and
E x is an orthonormal basis of T x M .
We use N x (0 , I ) to denote the distribution of ξ ◦ E x .
One can verify that N x (0 , I ) does not depend on the choice of basis E x .
3.1 An illustrative example: sampling from a sphere
To give some intuition about Assumptions 1 - 4, we present a simple example for sampling from
S d − 1 , the unit d-sphere in R d , which is a positively curved Riemannian manifold.
The subsequent bounds for the S d case also generalize with minor modifications to other closely-related manifolds such as SO ( d ) .
Let U ( x ) : R d → Re be a potential function, and suppose that we wish to sample from dp ( x ) ∝ e − U ( x ) dvol g ( x ) defined over S d − 1 .
It is known that for any x ∈ S d − 1 , v ∈ T x M , Ric( v, v ) > 0 , so that Assumption 1 is satisfied with L Ric = 0 .
The sectional curvature is bounded by 1 , so that
Assumption 4 holds with L R = 1 .
For the remainder of this example, for x ∈ S d − 1 we will identify
T x M with (cid:8) v ∈ R d : v T x = 0 (cid:9) .
Since the sphere is a submanifold of R d , the metric is simply given by the Euclidean dot product, i.e. g ( u, v ) =
u T v .
Next, we will verify Assumption 3.
Let us assume that for all x on the sphere, U ( x ) satisfies ∥ ∇ U ( x ) ∥ ≤ L 1 and (cid:13)(cid:13) ∇ 2 U ( x ) (cid:13)(cid:13) 2 ≤ L 2 , where ∇ and ∇ 2 are the usual Euclidean gradient and
Hessian respectively (bolded to distinguish from ∇ , the covariant derivative).
In order to sample from dp ( x ) , β ( x ) =
( I − xx T ) ∇ U ( x ) .
It is known that a geodesic γ ( t ) corresponds to a great arc on the sphere, and that parallel transport of a vector v involves a rotation of the tangential component of v .
To be precise, for any x ∈ S d and u, v ∈ T x M such that ∥ u ∥ = 1 , we verify that, in Cartesian coordinates, ∇ u v = ( uu T ) v .
Thus ∇ β ( x ) = ( uu T )( I − xx T ) ∇ U ( x ) + ∇ 2 U ( x ) u ≤ L 1 + L 2 .
Thus Assumption 3 holds with L ′ β
=
L 1 + L 2 .
Finally, we will verify Assumption 2.
Since S d − 1 has diameter π , the first part of Assumption 2 is satisfied with arbitrary m and R =
π .
The second part of Assumption 2 is satisfied with q = L ′ β .
This is because ddt ⟨ Γ( β ( γ ( t )); γ ( t ) → γ (0))
− β ( γ (0)) , γ ′ (0) ⟩ = ddt ⟨ β ( γ ( t )) , γ ′ ( t ) ⟩ = (cid:10) ∇ γ ′ ( t ) β ( γ ( t )) , γ ′ ( t ) (cid:11) ≤ L ′ β
∥ γ ′ ( t )
∥ 2 .
Bounding the Discretization Error in Geometric Euler-Murayama
One of our main technical contributions is an error bound for the Euler-Murayama discretization.
We provide an informal statement of this error bound as Lemma 1 below, and provide the formal statement as Lemma 7 in Appendix A.3.
Lemma 1 (Informal version of Lemma 7) .
Let x ( t ) denote the solution to (1.1) initialized at some x (0) .
Let x 0 ( t ) denote one step geometric Euler Murayama discretization: x 0 ( t ) :=
Exp x (0) ( tβ ( x (0)) + √ tζ ) , where ζ ∼ N x (0) (0 , I ) .
Under Assumptions 3 and 4, for sufficiently small t , there exists a coupling between x ( t ) and x 0 ( t ) such that
[Equation]
where O () hides polynomial dependence on the Lipschitz constant L ′ β , the sectional curvature of M , and dimension d .
Proof
Sketch
for Lemma
Given a time interval T , we construct a sequence of processes { x i ( t ) } i ≥ 0 indexed by i (exact definitinion given in (4.2) below).
Marginally, each x i ( t ) corresponds to the linear interpolation of a sequence of Euler-Murayama steps with stepsize δ i
=
2 − i T .
More specifically, for k ∈ Z + ,
[Equation]
for t ∈ [ kδ i , ( k + 1)
δ i ]
Notice that for i = 0 , x 0 ( T ) is a single step of (1.2) with stepsize T .
On the other hand, x ( t ) :=
lim
i →∞ x i ( t ) is exactly the SDE in (1.1) (see Lemma 2 at the end of this section).
We will soon see that E [d( x i ( T ) , x i +1 ( T )) 2 ]
=
O ( T 2 δ i ) .
We can then bound E [d( x 0 ( T ) , x ( T )) 2 ]
=
O ( T 3 ) by summing over the pairwise distance between adjacent x i and x i +1 for i ∈
Z
+ (this diminishes geometrically).
This proves Lemma 1.
To bound the distance between x i and x i +1 , we introduce a crucial additional structure: adjacent pairs of processes ( x ik , x i +1 k ) are coupled using the manifold analog of synchronous coupling , together with the discrete-time analog of "rolling without slipping" .
(see Remark 1),
The exact formula for how x i +1 k is to be constructed from x ik is given in (4.1) below; it is a little dense, so we provide a pictorial illustration in Figure 1.
For simplicity, we assume that β
= 0 (the dominating error in Lemma 1 is due to Brownian motion).
Figure 1 takes place over a period of time from kδ i to ( k + 1)
δ i (equivalently, 2 kδ
i +1 to (2 k + 2)
δ i +1 ).
The black squiggle denotes ( B ( t ) − B ( kδ i )) ◦ E ik for t ∈ [ kδ i , ( k + 1)
δ i ] , where B ( t ) is a standard d -dimensional Brownian motion, and E ik is some basis at x ik .
Let b
ik :=
( B (( k + 1) δ i ) − B
kδ i ))
◦
E ik (black solid arrow).
Bounding the distance between x i and x i +1 consists of two steps: Step 1: Bounding d( x ik +1 , Exp
[Equation]
i
[Equation]
12 k (Γ( b ik ; x ik → x i
[Equation]
12 k )))
[Solid black vs dotted
The purple line represents the geodesic from x ik to x i +12 k ; by construction, the basis E
i +12 k is the parallel transport of E ik along the purple geodesic.
The dotted black arrow denotes the ( B (( k + 1) δ i ) − B ( kδ i )) ◦
E
i +12 k ; we verify that it is the paralle transport of b ik (black arrow) along the purple geodesic.
Using Lemma 28, we can bound the distance between x ik +
=
Exp x ik ( b ik ) (gray point) and Exp x i
+12 k
(Γ( b ik ; x ik → x i +12 k ))
(green point) as
E [d(Exp
[Equation]
ik (
[Equation]
ik ) , Exp x i
[Equation]
12 k (Γ( b ik ; x ik → x i
[Equation] [Equation]
))) 2 ] ≤ (1 + O
( L
[Equation]
dδ
[Equation]
)) E [d( x ik , x
[Equation]
Step 2:
Bounding d(Exp x
i +12 k
(Γ( b ik ; x ik → x i +12 k )) , x i +12 k +2 )
[Dotted black arrow vs blue + red arrow]
Let b i
[Equation]
B ((2 k + 1) δ i +1 )) ◦
E
i +12 k
We verify that Γ( b ik ; x ik → x i +12 k ) (black dotted arrow) is equal to b i +12 k
+ b i +12 k +
(note that δ i +
= 1 / 2 δ i ).
On the other hand, x i +12 k +2 (orange point) is obtained from taking a step in b i +12 k direction (blue arrow) followed by a step in the Γ( b i
+12 k +1 ; x i +12 k
→ x i +12 k +1 ) direction (red arrow).
Due to curvature, this is not the same as taking a single step in the b ik =
b
i +12 k
+ b i
+12 k
+1 direction (green point).
However, we can bound the distance between the orange point and the green point using Lemma 3 of Sun et al.
[2019] (restated as Lemma 38 for ease of reference):
[Equation]
Summing the bounds of Step 1 and Step 2 gives E [d( x ik +1 , x
i +12 k +2 ) 2 ] ≤ e O
( L R dδ i ) E
[d( x ik , x i +12 k ) 2 ] +
O (( δ i ) 3 ) .
Recursing over k = 0 ...
2 i , and assuming that T ≤ 1 L R d , we get E [d( x i 2 i , x i
+12 i +1 ) 2 ] =
E [d( x i ( T ) , x i +1 ( T )) 2 ] ≤ O ( T 2 δ i ) .
Summing over i = 0 ... ∞ gives E [d( x i ( T ) , x ( T )) 2 ]
=
O ( T 3 ) .
Details for Construction of Brownian Motion
To bound the discretization error, we first characterize the manifold SDE (1.1) as the limit of a family of random processes.
Let x 0 ∈ M be an initial point and E = (cid:8) E 1 , . . .
, E d (cid:9) be an orthonormal basis of T x 0 .
Let B ( t ) denote a standard Brownian Motion in R d .
Let T ∈ R + .
Define
[Equation]
For any i ∈ Z + , let δ i := 2 − i T .
We will now define points x ik ∈ M and orthonormal basis E ik of T x ik for all i and all k ∈ { 0 , . .
, T / δ i } .
Our construction is inductive: Suppose we have already defined x ik and E ik for some i and for all k ∈ { 0 , . .
, T / δ i } .
Then, we construct x i +1 k , for all
k
= { 0 , . .
, T / δ i +1 } , as follows:
[Equation]
The above display defines points x i +1 k for all
k = { 0 , . .
, T / δ i +1 } .
For parallel transport, if the minimizing geodesic is not unique, any arbitrary choice will do.
We verify that for any i , for all k , x ik is indeed of the form (1.2), i.e. x ik +
=
Exp x ik ( δ i β ( x ik ) + √ δ i ζ ik ) , where ζ ik := 1 √
δ
i ( B (( k + 1) δ i ) − B ( kδ i ) ◦
E ik ) ∼ N x ik (0 , I ) .
Finally, for any i , any k , and any t ∈ [ kδ i , ( k + 1) δ i ) , we define x i ( t ) to be the “linear interpolation” of x ik and x ik +1 , i.e., x i ( t ) :
= Exp x ik (cid:0) t − kδ i
δ i ( δ i β ( x ik )
+ ( B (( k +
[Equation]
δ i ) − B ( kδ i )
[Equation]
◦ E ik (cid:1) .
Remark 1.
The choice of basis E ik in (4.1) can be seen as a combination of “synchronous coupling” and “rolling without slipping” (see Chapter 2 of [Hsu, 2002]).
In particular, E i +12 k :=
Γ( E ik ; x ik → x i +12 k ) corresponds to “synchronous coupling”—the step from x i +12 k to x i
+12 k +1 is (roughly) parallel to the step from x ik to x ik +1 .
On the other hand, E i +12 k
+1 :
=
Γ( E i +12 k ; x i +12 k
→ x i +12 k +1 ) corresponds to “rolling without slipping”—the step from x i +12 k +1 to x i +12 k +2 is with respect to an orthonormal basis that is parallel-transported from x
i +12 k to x i +12 k +1 .
We verify in the following Lemma that the limit, i → ∞ , of x i ( t ) is the SDE (1.1):
Lemma 2.
For any T , for t ∈ [0 , T ] , let x ( t ) :
=
lim
i →∞ x i ( t ) .
This limit exists uniformly almostsurely, and x ( t ) is a diffusion process generated by the operator L whose action on any smooth function f is given by Lf = ⟨ ∇ f, β ⟩ + 12 ∆( f ) , where ∆ denotes the Laplace Beltrami operator.
Thus x ( t ) is equal to (1.1) in distribution.
Lemma 2 follows immediately from [Gangolli, 1964, Jørgensen, 1975], but we provide a proof in
Appendix A.1 for completeness.
Langevin MCMC on Riemannian Manifolds
With the one-step discretization error in Lemma 1, we can bound the iteration complexity of Langevin
MCMC (1.2):
Theorem 1 (Convergence of Langevin MCMC on Riemannian Manifold) .
Assume the manifold M satisfies Assumptions 1 and 4.
Assume in addition that there exists a constant L ′ R such that for all x ∈ M , u, v, w, z, a ∈ T x M , ⟨ ( ∇ a R )( u, v ) w, z ⟩ ≤ L ′ R ∥
u
∥∥ v ∥∥ w ∥∥
z ∥∥ a ∥ (this last assumption is for analytical convenience; L ′ R does not show up in the quantitative bounds).
Let β be a vector field satisfying Assumptions 2 and 3; assume in addition that m > L Ric / 2 and that q + L Ric / 2 ≥ 0 .
Let x ∗ be some point with β ( x ∗ ) = 0 .
Let y ( t ) denote the exact geometric SDE given in (1.1) initialized at some y (0) .
Let
K ∈ Z
+ be some iteration number and δ be some stepsize.
Let x k denote the Euler Murayama discretization of (1.1) , defined by x k +
=
Exp x k ( δβ ( x k ) + √ δζ k ) , where ζ k ∼ N x k (0 , I ) , initialized at some xsatisfying d( x 0 , x ∗ ) ≤ 2 R .
Then, there exists a constant C
= poly ( L ′ β , d, L R , R , 1 m − L Ric / 2 , log K ) , such that if δ ≤ 1 C 0 , then there is a coupling between x K and y ( Kδ ) satisfying the distance bound
E [d( y ( Kδ ) , x K )] ≤ e − αKδ +( q + L Ric / 2) R 2 / 2 E
[d( y (0) , x 0 )]
+ exp (( q + L Ric / 2) R 2 ) · ˜ O ( δ 1 / 2 ) .
˜ O hides polynomial dependence on L ′ β , d, L R , R , 1 m − L Ric / 2 , log K, log 1 δ , and α :
= min (cid:110) m − L Ric / 2, 12 R 2 (cid:111) ·
e − 12 ( q + L Ric / 2) R 2 .
We defer the proof of Theorem 1 to Appendix A.4, where we state the explicit expressions for C 0 .
Discussion of Theorem
To sample from dπ ∗ ( x )
=
e − h
( x ) d vol g ( x ) , we let β ( x ) :
= − 12 ∇ h ( x ) ; under this choice of β , π ∗ ( x ) is invariant under the SDE for y ( t ) .
Picking y (0) ∼ π ∗ ( x ) , we thus ensure that
Law( y ( t ))
=
π ∗ for all t .
Therefore, the W 1 distance between Law( x K ) and π ∗ is upper bounded by E [d( y ( Kδ ) , x K )] , which is in turn upper bounded in Theorem 1.
The distance bound consists of two terms: the first term is exponentially small in Kδ , so it goes to 0 as the number of steps tends to infinity; the second term is proportional to δ 1 / 2 .
We first pick δ = ˜
O ( ε 2 ) so that the second term is bounded by ε .
It then suffices to let
K = 1 αδ log ( E [d( y (0) ,x 0 )] ε ) + ( q + L Ric / 2) R 2 =
˜ O ( ε − 2 ) in order for the first term to be bounded by ε .
Note that by our assumptions and by Lemma 18 E [d( y (0) , x 0 )] ≤ poly ( L R , L ′ β , d, R , 1 m ) .
We now discuss a few specific cases for Assumption 2.
The easiest setting is when m > L Ric /and R =
(in this case the value of q does not matter); this occurs, for example, if β = − 12 ∇ h for some c -strongly-convex h , and M has positive Ricci curvature.
The mixing rate is α = c − L Ric, and this closely relates to a well known result by Bakry and Emery on the Log-Sobolev Inequality of
CD ( · , ∞ ) distributions:
Theorem ([Bakry et al., 2014]) Let Ric denote the Ricci curvature tensor.
If dπ ∗ ( x ) =
e − h
( x ) d vol g ( x ) , and ∇ 2 h + Ric ≻ ρI , then π ∗ satisfies the Log-Sobolev Inequality (LSI) with parameter ρ .
When π ∗ satisfies the LSI ( ρ ) , it has been shown in [Bakry et al., 2014] that for any initialization, the
KL divergence of Law( x ( t )) of (1.1) with respect to its stationary distribution converges to 0 with rate ρ .
In our example in the preceding paragraph, dπ ∗ =
e − h d vol g satisfies LSI with ρ = c − L Ric , which is, up to a constant factor, equal to our mixing rate.
The requirement q + L Ric / 2 ≥ 0 is without loss of generality.
If q + L Ric / 2 < 0 , we can take q ′ = − L Ric / 2 and verify that β satisfies Assumption 2 with ( m, q ′ , R ) .
Since q ′ + L Ric / 2 = 0 , the mixing rate is then α = min (cid:110) m − L Ric / 2, 12 R 2 (cid:111) , which corresponds to the easy “ CD (0 , ∞ ) ” setting.
A harder, but more general setting is when β = 12 ∇ h for some non-convex h , and L Ric > 0 , i.e. the manifold can have negative Ricci curvature.
We will still need to assume that ∇ h ( x ) is contractive for points which are further away than some radius R ; otherwise, the SDE (1.1) may drift off to infinity and e − h may not be integrable.
Under Assumption 3, β satisfies Assumption 2 with m, L ′ β , R .
The mixing rate α is then proportional to e − ( L ′ β + L Ric / 2) R 2 ; this can be very small if h is nonconvex and highly nonsmooth, and the manifold M has large negative Ricci curvature, leading to very slow mixing.
This is generally unavoidable, even when M is the flat Euclidean space.
Readers familiar with the Holly-Stroock perturbation may find the exp (( q + L Ric / 2) R 2 ) term familiar: Let M have diameter R and let U have 2 q -Lipschitz gradient„ so that β = 12 ∇ U satisfies
Assumption 2 with q .
Then we can decompose U
=
U
+ U 2 , where U 1 is − L Ric -strongly-convex and U 2 is a "perturbation" with magnitude (2 q + L Ric ) R 2 .
e 2 U 1 dvol g satisfies CD (0 , ∞ ) and thus e 2 U dvol g has Log-Sobolev constant scaling with exp ((2 q + L Ric ) R 2 ) .
Finally, on compact manifolds without boundary, one may take R to be the diameter of the manifold and upper bound the mixing rate α by α = 12 R 2 e − 12 ( q + L Ric / 2) R 2 .
5.1 Proof Sketch of Theorem
The proof of Theorem 1 consists of two steps.
The first step is bounding the discretization error of (1.2); we have already done this in Lemma 1 in the previous section.
The second step is showing that two paths of (1.1) converge.
To do so, we consider f (d( x ( t ) , y ( t ))) , where f is a Lyapunov function taken from [Eberle, 2016]
(its exact form is given in Definition 2 in the Appendix).
Two key property of f are that (i) 12 exp ( − ( q + L Ric / 2)
R 2 / 2) r ≤ f ( r ) ≤ r ; and (ii) f ′ ( r ) ≤ 1 , so that convergence in f (d( x ( t ) , y ( t ))) implies convergence in d( x ( t ) , y ( t )) .
Under the Kendall-Cranston Coupling 3 of x ( t ) and y ( t ) , one can show that f (d( x ( t ) , y ( t ))) contracts with rate α .
We quantify the contraction rate in Lemma 3 below.
We stress that Lemma 3 is not new; it was first presented by Eberle [2016] (with minor variation), and its proof combines Theorem 6.6.
of [Hsu, 2002] with the Lyapunov function analysis of [Eberle, 2016].
For completeness, we provide a proof of Lemma 3 in Appendix B.3.
Lemma 3.
Assume β is ( m, q, R ) -distant dissipative as per Assumption 2, and that is also satisfies Assumption 3.
Further assume that m > L Ric / 2 and q + L Ric / 2 ≥ 0 .
Let x ( t ) and y ( t ) denote solutions to (1.1) .
Then there exists a Lyapunov function f satisfying 1.
f ( r ) ≥ 12 exp ( − ( q + L Ric / 2)
R 2 / 2) r and 2.
| f ′ ( r ) | ≤ 1 , and a coupling between x ( t ) and y ( t ) , such that for all time T ,
[Equation]
where α :=
min (cid:110) m − L Ric /
[Equation]
Given these two steps , we can now sketch the proof of Theorem 1.
Consider an arbitrary step k of (1.2).
For t ∈ [ kδ, ( k + 1) δ ) , let ¯ x ( t ) denote the solution to (1.1), initialized at ¯ x ( kδ ) .
Then by Lemma 3, E [ f (d( y (( k + 1) δ ) , ¯ x (( k + 1) δ )))] ≤ e − αδ E [ f (d( y ( kδ ) , x k ))] .
On the other hand, by Lemma 1 E [d(¯ x (( k + 1) δ ) , x k +1 )] ≤ O ( δ 3 / 2 ) .
Summing these two bounds, applying triangle inequality, and using the fact that f ′ ( r ) ≤ 1 , we get
[Equation]
Applying the above recursively for k =
K , we get
[Equation]
Theorem 1 thus follows from the bounds 12 exp ( − ( q + L Ric /
[Equation]
R 2 /
[Equation]
r
[Equation]
f
[Equation]
Lastly, we briefly mention a complication in the full proof that we omitted from the above sketch: the discretization error between ¯ x (( k + 1) δ ) and x k +1 grows with ∥ β ( x k ) ∥ , which does not have a global upper bound.
In practice, we can verify that d( x k , x ∗ ) is sub-Gaussian, so that with high probability d( x k , x ∗ ) ≤ O (log( kδ )) ; this in turn allows us to bound ∥ β ( x k ) ∥ via Assumption 3.
This is the reason for the dependency on log K and log(1 /δ ) in the iteration complexity of Theorem 1.
Stochastic Gradient Langevin MCMC
Finally, we bound the iteration complexity of process (2.1), which takes the Euler-Murayama scheme (1.2), and replaces β ( x k ) at step k by a stochastic estimate ˜ β k ( x k ) .
Theorem 2 (Convergence of SGLD on Riemannian Manifold) .
Assume the manifold M satisfies
Assumptions 1 and 4.
Assume in addition that there exists a constant L ′ R such that for all x ∈ M , u, v, w, z, a ∈ T x M , ⟨ ( ∇ a R )( u, v ) w, z ⟩ ≤ L ′ R ∥
u
∥∥ v ∥∥ w ∥∥
z ∥∥ a ∥ .
Let β be a vector field satisfying
Assumptions 2 and 3; assume in addition that m > L Ric / 2 and that R = 0 .
Let y ( t ) denote the exact geometric SDE given in (1.1) .
Let x k denote the stochastic Euler Murayama discretization (2.1) , where ˜ β
k denote independent random vector fields, with E [˜ β ( x )]
= β ( x ) for all x and ∥ β ( x ) − ˜ β k ( x ) ∥ ≤ σ with probability 1.
Let
K ∈ Z
+ be some iteration number and δ be some stepsize.
There exists a constant C 1 = poly ( L ′ β , d, L R , L ′ R , 1 m , log K ) , such that if δ ≤ 1 C 1 , then there is a coupling between x K and y ( Kδ ) satisfying the distance bound
[Equation]
˜ O hides polynomial dependence on L ′ β , d, L R , L ′ R , σ,
[Equation]
We defer the proof of Theorem 2 to Appendix A.5, where we state the explicit expressions for C 1 .
Discussion of Theorem
To sample from dπ ∗ ( x )
=
e − h
( x ) d vol g ( x ) , we let β ( x ) :
= − 12 ∇ h ( x ) and let y (0) ∼ π ∗ ( x ) .
The
W 2 distance between Law( x K ) and π ∗ is upper bounded by (cid:113) E
[d( y ( Kδ ) , x K ) 2 ] .
To achieve ε error in W 2 , Theorem 2 requires ˜ O ( ε − 2 ) steps.
The reasoning is very similar to Theorem 1.
As already discussed in Section 2, the Assumption on Theorem 2 is considerably more restrictive than Theorem 1.
We also highlight that the error bound in Theorem 2 does depend on L ′ R , the derivative of the Riemannain curvature tensor .
This is in contrast to Theorem 1, where L ′ R does not appear in any of the bounds.
Though the iteration complexity for Theorem 2 can be larger than Theorem 1 as it depends on additional parameters σ and L ′ R , (2.1) may nonetheless be faster than (1.2).
For example, let h ( x ) = 1 N (cid:80) Ni
=1 h i ( x ) and assume ∥∇ h
i − ∇ h j ∥ ≤ σ for all i, j , one step of (2.1) can be performed with a single gradient computation for a single uniformly sampled h i , whereas (1.2) would require N gradient computations.
Proof Sketch of Theorem 2 and Theoretical Highlights
For any step k , let us define ¯ y k
+1 :
=
Exp y ( kδ ) ( δβ ( y ( kδ ))
+ √ δ ¯ ζ k ) , where ¯ ζ k ∼ N y k (0 , I ) .
We show, in Lemma 8, that E [d( x k +1 , ¯ y k +1 ) 2 ] ≤ (1 −
δ 4 ( m − L Ric / 2))
E
[d( x k , y ( kδ )) 2 ]
+16 δ 2 σ 2 .
On the other hand, once again applying Lemma 1, we can verify that E [d( y (( k + 1) δ ) , ¯ y k +1 ) 2 ] ≤ O ( δ 3 ) .
By Young’s inequality and triangle Inequality,
E [d
[Equation]
The bound in Theorem 2 follows immediately from applying the above recursively for k = 1 ...K .
We note that The contraction in Lemma 8 is in turn derived from Lemma 29, which may be of independent interest as it quantifies the distance evolution between two general discrete-time stochastic processes (which do not have to be diffusions or related to the Gaussian noise).
It is elucidative to compare the proof structure of Theorem 2 with that of Theorem 1 above.
On a high level, at step k , Theorem 1 first approximates the discrete MCMC step ( x k +1 ) by an exact SDE initialized at x k ( ¯ x (( k + 1) δ ) ).
It then uses two facts: 1. f
(d(¯ x ( t ) , y ( t )))
contracts under the exact
SDE , and 2.
the approximation error between x k +1 and ˜ x
(( k + 1) δ ) is small.
In contrast, at step k ,
Theorem 2 first approximates the exact SDE ( y (( k + 1) δ ) ) by a Euler-Murayama step ( ¯ y k +1 ).
It then uses two facts: 1.
d( x k +1 , ¯ y (( k + 1) δ )) contracts under the stochastic Euler-Murayama step , and 2.
the approximation error between ¯ y k +1 and y (( k + 1) δ ) is small.
The reason for this change is to make use of the fact that E [˜ β k ] = β , conditioned on the randomness up to time kδ .
Suppose we had followed the proof of Theorem 1 and defined ¯ x ( t ) to be the solution to the exact SDE, with drift ˜ β k , then showing contraction of the exact SDE becomes very tricky as E [˜ β k ] is not equal to β when conditioned on ¯ x ( t ) for any t > kδ .
Acknowledgements
Xiang Cheng acknowledge support from NSF BIGDATA grant (1741341) and NSF CCF- (TILOS AI Research Institute).
Jingzhao Zhang acknowledges support by Tsinghua University
Initiative Scientific Research Program.
Contents
of the Appendices
A Manifold SDE
A.1 SDE Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Existence of Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Discretization Error of Euler Murayama . . . . . . . . . . . . . . . . . . . . . . .
A.4 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[Table]
B.
[Table]
The Kendall Cranston Coupling . .
[Table]
[Table]
function and its smooth approximation . .
B
[Table]
Contraction of Lyapunov Function under Kendall Cranston Coupling . .
[Table]
Bounds
C.1 One-Step Distance Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 L p Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Subgaussian Bounds . .
C.4 Misc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Fundamental Manifold Results
D.1 Jacobi Field Approximations . . .
D
2 Discrete Coupling Bounds . .
E Matrix ODE
F Miscellaneous Lemmas
A Manifold SDE
An outline of this section is as follows:
In Section A.1, we prove Lemma 2, which guarantees that x i ( t ) defined in (A.2) has a limit x ( t ) that equals the solution of the exact Langevin Diffusion in (1.1).
In Section A.3, we prove Lemma 7, which bounds the distance between x 0 ( t ) from (A.2) and the limit x ( t ) .
This is equivalent to bounding the distance between the Euler Murayama discretization (1.2) and the exact Langevin Diffusion (1.1).
In Section A.4, we prove Theorem 1. 4.
In Section A.5, we prove Theorem 2.
We also list below the key lemmas which are used to prove the results above.
Theorem 1 relies on Lemma 3 (contraction of Lyapunov function under exact SDE) and Lemma 7 (bound on Euler Murayama discretization error).
(a) Lemma 7 essentially sums the bound from Lemma 4.
(b) Lemma 4 relies on Lemma 28 and Lemma 38.
(c) Lemma 3 relies on Lemma2.
Theorem 2 relies on Lemma 8 (contraction of Lyapunov function under stochastic gradient Euler-Murayama step) and Lemma 7 (bound on Euler Murayama discretization error).
(a) Lemma 8 relies on Lemma 29.
A.1 SDE Construction
In this section, we state and prove key lemmas related to our construction in Section 4, which we reproduce below for ease of reference: Let x 0 ∈ M be an initial point and E = (cid:8) E 1 , . . .
, E d (cid:9) be an orthonormal basis of T x 0 .
Let B ( t ) denote a standard Brownian Motion in R d .
Let T ∈ R + .
Define
[Equation]
For any i ∈ Z + , let δ i := 2 − i T .
We will now define points x ik ∈ M and orthonormal basis E ik of T x ik for all i and all k ∈ { 0 , . .
, T / δ i } .
Our construction is inductive: Suppose we have already defined x ik and E ik for some i and for all k ∈ { 0 , . .
, T / δ i } .
Then, we construct x i +1 k , for all
k
= { 0 , . .
, T / δ i +1 } , as follows:
[Equation]
The above display defines points x i +1 k for all
k = { 0 , . .
, T / δ i +1 } .
For any i , any k , and any t ∈ [ kδ i , ( k + 1) δ i ) , we define x i ( t ) to be the “linear interpolation” of x ik and x ik +1 , i.e.,
[Equation]
Let us define two convenient notation that we will use throughout our proofs in this Appendix.
First, let Φ( t
[Equation]
x
[Equation]
E, β, B , i )
[Equation]
denote the solution to the interpolated process in (4.2) (reproduced in (A.2)), initialized at x 0 = x , i.e.
Φ( t ; x 0 , E, β, B , i ) =
x i ( t ) as defined in (A.2).
(This notation becomes convenient later on when we need to refer to (4.2) but with different initial points, or with drift vector fields other than β , or with specific choices of B .)
We also let
[Equation]
Below, we prove Lemma 2 (stated at the end of Section 4), which guarantees that x ( t ) =
Φ( t ; x, E, β, B ) exists, and that x ( t ) is a solution to the exact Langevin diffusion SDE in (1.1).
Proof of Lemma 2.
The existence of the almost-sure, uniform limit x ( t ) is proven in Lemma 5.
For the rest of this proof, we verify that x ( t ) has the generator Lf = ⟨ ∇ f, β ⟩ + 12 ∆( f ) , where ∆ denotes the Laplace Beltrami operator.
By Proposition 3.2.1 of [Hsu, 2002], this implies that x ( t ) is the solution to (1.1).
Let F t denote the sigma field generated by B ( s ) : s ∈ [0 , t ] .
Consider any f : M → R with ∥ f ′ ∥ ≤ C ,
∥ f ′′ ∥ ≤ C , ∥ f ′′′ ∥ ≤ C globally.
Let x i ( t ) be as defined in (4.2).
We will verify that f ( x ( t ))
− f ( x (0)) − (cid:82) t
Lf ( x ( t ))
dt is a martingale.
To begin, let s, t ∈ [0 , T ] be such that s = jδ a and t =
j ′ δ a for some positive integers j ≤ j ′ , and a , (recall that δ i =
T/ 2 i ).
We will show that conditioned on x ( s ) , f ( x ( t ))
− f ( x ( s )) − (cid:82) t s Lf ( x ( t ))
dt is a martingale.
Let us define
[Equation]
so that x i ( t ) =
Exp x ik ( t − kδ i δ i u ik ) , where t ∈ [ kδ i , ( k + 1)
δ i ] .
Consider an arbitrary ℓ ≥ a .
Consider the sum t/δ i − 1 (cid:88)
[Equation]
By Taylor’s theorem,
[Equation]
≤
[Equation]
3 δ
[Equation]
3 (cid:13)(cid:13) β ( x ℓk ) (cid:13)(cid:13) 3 + 8 C 3 (cid:13)(cid:13) B
(( k + 1)
[Equation]
B ( kδ
[Equation]
) (cid:13)(cid:13)
[Equation]
where C (cid:13)(cid:13)
u ℓk (cid:13)(cid:13) 3 captures the third-and-higher order Taylor terms.
The first order Taylor term can be decomposed as
[Equation]
We now simplify the second order Taylor term.
Let v :=
( B (( k + 1) δ i ) − B
kδ i )) ◦ E ik .
We verify that E F kδℓ (cid:2) ∇ 2 f ( x ℓ ( kδ ℓ ))
[ v, v ] − δ ℓ ∆ f ( x ℓ ( kδ ℓ )) (cid:3) = 0 , because ( B (( k + 1) δ i ) − B ( kδ i )) ◦
E ik has identity covariance, and the Laplace Beltrami operator is the trace of the Hessian.
We can also bound, using Young;s inequality,
[Equation]
Finally, note that there exists a constant C ′ , which depends on T, d, L ′ β
∥ β ( x 0 ) ∥ , such that for all ℓ , for all t ∈ [0 , T ] , E (cid:104)(cid:13)(cid:13) β
( x ℓ ( t ) (cid:13)(cid:13) 6 (cid:105) ≤ C ′ .
The proof is similar to Lemma 16 and we omit it here.
Plugging into (A.5) and taking expectation conditioned on the Brownian motion B ( t ) : t ∈ [0 , s ] , we get that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
≤ poly ( C, C
[Equation]
, d )
[Equation]
[Table]
[Equation]
[Table]
[Equation]
k = s
[Equation]
δ
[Equation]
≤ poly ( C, C ′ , d, T ) δ ℓ .
where the first line is because f ( x ℓ ( t )) − f ( x ℓ ( s ))
= (cid:80) t/δ ℓ − 1 k = s/δ ℓ f ( x ℓ (( k + 1) δ ℓ ))
− f ( x ℓ ( kδ ℓ )) , noting that t, s are multiples of δ ℓ by definition, and the second line uses our first and second Taylor approximation bounds above.
Next, define g ℓk := δ ℓ (cid:10) β ( x ℓ ( kδ ℓ )) , ∇ f ( x ℓ ( kδ ℓ )) (cid:11) + δ ℓ 2 ∆ f ( x ℓ ( kδ ℓ )) .
Using the smoothness of β and f , and the fact that d( x ℓ ( t ) , x ( kδ ℓ )) ≤ δ ℓ (cid:13)(cid:13) β ( x ( kδ ℓ )) (cid:13)(cid:13) + (cid:13)(cid:13) B (( k + 1) δ ℓ ) − B ( kδ ℓ ) (cid:13)(cid:13) 2 , we verify that for any k ,
[Equation]
Putting everything together, we obtain the bound
[Equation]
By Lemma 5, sup t ∈ [0 ,T ] d( x ℓ ( t ) , x ( t )) converges to 0 almost surely as ℓ → ∞ .
By Dominated
Convergence Theorem, and by smoothness of f and β ,
[Equation]
=
[Equation]
The last equality holds because lim ℓ →∞ poly ( C, C ′ , d, T ) δ ℓ 1 / 2 = 0 .
Recall that we assumed that s and t are integral multiples of T/ 2 a for some positive integer a .
To extend to general s, t , we note that the set of dyadic points (i.e. multiples of T/ 2 a , for some integer a ) is uniformly dense on the real line.
A.2 Existence of Limit
We present below Lemma 4, which bounds the distance between two adjacent trajectories x i ( t ) and x i +1 ( t ) as defined in (4.2) (or equivalently (A.2)).
The proof of Lemma 4 works by combining
Lemma 28 (which bounds distance evolution under "synchronous coupling"), and Lemma 38 (which bounds distance evolution under "rolling without slipping"; Lemma 38 is taken from [Sun et al., 2019]).
The proof of Lemma 4 corresponds to Step 1 and Step 2 of the proof sketch of Lemma 1 in
Section 4.
Lemma 4 plays a key role bounding the Euler Murayama discretization error in Lemma 7 in Section
A.3.
The proof of Lemma 7 essentially involves summing the bound from Lemma 4, for i = 0 ... ∞ .
Another application of Lemma 4 is to verify the existence of x ( t ) =
lim
i →∞ x i ( t ) as defined in (A.4) in Lemma 5.
Lemma 4.
Let T be any positive constant.
Let x i ( t ) be the (interpolation) of the Euler Murayama discretization with stepsize δ i
=
T/ 2 i as defined in (4.2) (or equivalently (A.2) ).
Let K := 2 i so
that
T =
Kδ i .
Assume that there is are constants L β , L ′ β such that for all x, y ∈ M , ∥ β ( x ) ∥ ≤ L β and (cid:13)(cid:13) β
[Equation]
y ) (cid:13)(cid:13) ≤ L ′ β
∥ x − y ∥ .
Then
E
[Equation]
and
P
[Equation]
sup
[Equation]
Remark: Lemma 4 is usually applied with T being the step-size of a a single Euler-Murayama discretization step (i.e. δ in (1.2)).
Therefore, by taking T to be sufficiently small, the exponential term can be made small, e.g. ≤ 2 .
Proof.
Recall that x i ( t ) is the linear interpolation of x ik as defined in (4.1).
Let us define
[Equation]
Our proof breaks down the bound of d( x ik , x i +12 k +2 ) into two parts: by Young’s inequality,
[Equation]
We now bound the first term of (A.7).
From definition in (4.1) and (A.6),
[Equation]
We thus apply Lemma 28, with x := x ik , y :
=
x i +12 k , u :=
δ i β ( x ik )+( B (( k + 1) δ i ) − B ( kδ i )) ◦
E ik , v :=
δ i β ( x i +12 k )
+ ( B ((2 k + 2)
δ i +1 )
− B (2 kδ i +1 )) ◦
E i +1 k .
Let γ ( t ) , u ( t ) , v ( t ) be as defined in
Lemma 28.
Then Lemma 28 bounds d(Exp x ( u ) , Exp y ( v )) 2 ≤ (1 + 4 C
k e 4 C k )d( x, y )
+ 32 e C k ∥ v (0) − u (0) ∥ 2 + 2 ⟨ γ ′ (0) , v (0) − u (0) ⟩ (A.8)
where
C k :=
√ L R ( ∥ u ∥ + ∥ v ∥ ) ≤ 2 √ L R ( δ i
L β + (cid:13)(cid:13)
B (( k + 1) δ i ) − B ( kδ i )
(cid:13)(cid:13) 2 ) .
Some of the terms above can be simplified.
We begin by bounding the ∥ u (0) − v (0) ∥ term.
By assumption that β is Lipschitz, (cid:13)(cid:13)(cid:13) δ i β ( x ik ) − Γ x ik x i +12 k
δ i β ( x i +12 k ) (cid:13)(cid:13)(cid:13) ≤ δ i L ′ β
d( x ik , x i +12 k ) .
By definition of
E k from (4.1),
[Equation]
where the last line is because δ i =
2 δ
i +1 and because E i +12 k
[Equation]
= Γ x
[Equation]
k x ik ( E ik ) from (4.1).
Thus
[Equation]
=
We can thus bound via Young’s Inequality:
[Equation]
Finally, noting that ∥ γ ′ (0) ∥
=
i d( x ik , x i +12 k ) ,
[Equation]
Plugging into A.
[Equation]
We now bound the second term of (A.7).
Let us introduce two more convenient definitions:
[Equation]
It follows from definition that
[Equation]
We break the bound on d(Exp x
i +12 k
( a k + b k ) , x
i +12 k +2 ) into two terms:
[Equation]
To bound the first term
[Equation]
we apply Lemma 38 (from [Sun et al., 2019]) with x = x i 2 k , a = a k , y =
b k , to get
[Equation]
To bound the second term
[Equation]
we apply Lemma 28 (with x = y = z ), so that
[Equation]
[Table]
where we define C
[Equation]
Plugging everything into (A.7),
[Equation]
In fact
[Equation]
if we consider any t ∈ [ kδ i , ( k + 1) δ i )
[Equation]
and using the definition of x i ( t ) from (A.3) as the linear interpolation between x ik and x ik +
[Equation]
we can extend the bound to
[Equation]
Let us define
[Equation]
It follows from (A.9) that r k ≥ sup t ∈
[( k − 1)
δ i ,kδ i )
d( x i ( t ) , x i +1 ( t )) and that r k +1 ≥ r k with probability 1, for all k , so that sup t ≤ T d( x i ( t ) , x i +1 ( t )) ≤ r K .
We will now bound E (cid:2) r 2 K (cid:3) , and then apply
Markov’s Inequality.
Let us define F k to be the σ -field generated by B ( t ) for t ∈ [0 , kδ i ) .
Then
[Equation]
We will bound the terms above one by one.
First, note from definition that
[Equation]
Let η ik
[Equation]
=
[Equation]
k +
[Equation] [Equation]
B ( kδ i ) .
For sufficiently large i , δ i ≤ √ L R L β / 8 .
Simplifying,
[Equation]
e
[Equation]
L
[Equation]
∥
[Equation]
ik
[Equation]
(cid:105)
[Equation]
where we use
[Equation]
where we use Lemma 21, and the fact that δ i ≤ 1 32 L R d for sufficiently large i .
Next, we bound E F k (cid:104) 32 KL 2 R ( ∥ a k ∥ 6 + ∥ b k ∥ 6 ) e 2 √ L R ( ∥ a k ∥ + ∥ b k ∥ ) (cid:105) .
Note that ∥ a k ∥ ≤ δ i 2 L β + (cid:13)(cid:13)
η i +12 k
(cid:13)(cid:13) and ∥
b k ∥ ≤ δ i 2 L β + (cid:13)(cid:13) η
i +
k
+
(cid:13)(cid:13) .
By similar argument as above,
[Equation]
≤
[Equation]
Finally, note that C ′ k ≤
[Equation]
L
[Equation]
+
[Equation]
) , so that
[Equation]
Put together,
[Equation]
≤
[Equation]
Applying the above recursively and simplifying,
[Equation]
Recall that r k ≥ sup t ∈
[( k − 1)
δ i ,kδ i )
d( x i ( t ) , x i +1 ( t ))
(see (A.9)), and that r k +1 ≥ r k with probability 1, for all k , so that sup t ≤ T d( x i ( t ) , x i +1 ( t )) ≤ r K .
This proves the first claim of the lemma.
By Markov’s Inequality, and recalling that r k is w.p. 1 non-decreasing and sup t ≤ T d( x i ( t ) , x i +1 ( t )) ≤
[Equation]
K ,
[Equation]
This proves the second claim of the lemma.
Lemma 5.
Let x ∈ M be some initial point and E an orthonormal basis of T x M .
Let B ( t ) be a
Brownian motion in R d ; and β ( x ) a vector field satisfying Assumption 3.
Let T ∈ R + , for t ∈ [0 , T ] and let x i ( t ) be constructed as per (4.2) .
Then with probability 1, there is a limit x ( t ) such that for all ε , there exists an integer N such that for all i ≥ N ,
[Equation]
Proof of Lemma 5.
Let us define L 0 :=
∥ β ( x (0))
∥ .
Step 1:
Bounding the probability of deviation between x i and x i +
We would like to apply Lemma 4.
However, note that Lemma 4 assumes that ∥ β ( x ) ∥
≤ L β globally, which we do not assume here.
We must therefore approximate β by a sequence of Lipschitz vector fields.
Let us define
[Equation]
Let us denote by L β j := 2 j/ 2 .
We verify that for all x, y ∈ M , (cid:13)(cid:13)
β j ( x ) (cid:13)(cid:13) ≤ L β j and (cid:13)(cid:13) β j
[Equation]
x
[Equation]
≤
[Equation]
Finally, for any let ˜ x i,j ( t ) be as defined in (A.3), with β replaced by β j .
Lemma 4 immediately implies that, for all i ≥ C (where C is some constant depending on L R , T, d ),
[Equation]
where we use the fact that δ i L iβ 2 = T by definition.
Recalling that β j ( x ) = β ( x ) unless ∥ β ( x ) ∥ ≥ 2 j/ 2 ,
[Equation]
We can bound ∥ β ( x ) ∥ ≤ L 0 + L ′ β d( x, x 0 ) , so that
[Equation]
where we use Lemma 15, with K = 2 i , and assume that i satisfies 2 i/ 2 ≥ L 0 and 2 i ≥ T .
Using identical steps, we can also bound
[Equation]
Put together,
[Equation]
where C 2 is a constant that depends on T, L R , L ′ β , L 0 , d , but does not depend on i .
Step 2:
Apply Borel-Cantelli to show uniformly-Cauchy sequence with probability
Thus
[Equation]
By the Borel-Cantelli Lemma,
[Equation]
for infinitely many
[Equation]
Equivalently, with probability 1 , for all ε , there exists a N such that for all i ≥ N ,
[Equation]
2 − i 4 .
For any j ≥ i ≥ N , it then follows that
[Equation]
Step 3: Uniform-Cauchy sequence implies uniform convergence to limit using standard arguments Therefore, with probability 1, x i ( t ) is a uniformly Cauchy sequence.
Let x ( t ) be the point-wise limit of x i ( t ) , as i → ∞ .
It follows 4 that with probability 1, for any ε , there exists a N such that for all i ≥ N ,
[Equation]
Lemma 6.
Let β ( · ) be a vector field satisfying Assumption 3.
Assume also that there exists L β such that ∥ β ( x ) ∥
≤ L β for all x .
Consider arbitrary x 0 ∈ M and let E be an orthonormal basis of T x 0 M .
Let B be a standard Brownian motion in R d .
Let x i ( t ) =
Φ( t ; x, E, β, B , i ) and x ( t ) =
Φ( t ; x, E, β, B ) as defined in (A.3) and (A.4) respectively.
(Existence of x ( t ) follows from
Lemma 5).
Then for any non-negative integer ℓ ,
[Equation]
where δ i := 2 − i T
Proof.
Consider any fixed i , let δ i :=
T/ 2 i and let K :
= T/δ i =
2 i as in (4.1).
By the first claim of Lemma 4, we can bound
[Equation]
where we use the fact
that Kδ i =
T by definition.
By repeated application of Young’s Inequality
[Equation]
we can bound, for any ℓ and any j ≥ ℓ ,
[Equation]
Since the above holds for any j
[Equation]
we can take the limit of j → ∞ and
[Equation]
where we use the fact that d( x j 2 j , x ( T )) converges almost surely to 0 , from Lemma
[Equation]
A.3 Discretization Error of Euler Murayama
Given the results of the previous section, we are now ready to prove Lemma 7, which is informally stated as Lemma 1 in the Section 4.
The proof of Lemma 7 works by summing, for all i , the distance
[Table]
between x i ( t ) and x i +1 ( t ) (which is bounded in Lemma 4).
Extra care must be taken to ensure that iterates do not stray too far from the initial error.
The crucial analysis corresponding to Step 1 (synchronous coupling) and Step 2 (rolling without slipping) discussed in the proof sketch of Lemma 1 in Section 4 can be found in the proof of Lemma 4 in Section A.2 above.
Lemma 7.
Let M satisfy Assumption 4.
Let β ( · ) be a vector field satisfying Assumption 3.
Consider arbitrary x (0) ∈ M .
Let L 1 be any constant such that L 1 ≥
∥ β ( x (0))
∥ and let T be a step-size satisfying T ≤ min (cid:110) 1 16 L ′ β ,
[Equation]
Let x ( t ) denote the solution to (1.2) , initialized at x (0) .
Let x 0 ( t ) :
=
Exp x (0) ( tβ ( x (0)) + √ tζ ) , where ζ ∼ N x (0) (0 , I ) .
Then there exists a coupling between x 0 ( T ) and x ( T ) such that
[Equation]
Proof of Lemma 7.
Let E be an orthonormal basis of T x (0) M .
Following the definition of Φ in (A.3), we verify that x 0 ( T ) =
Φ( T ; x (0) , E, β, B , 0) , where B is some Brownian motion, and equality is in the sense of distribution.
On the other hand, by Lemma 2, x ( t ) =
Φ( t ; x (0) , E, β, B ) , where Φ is the limit of Φ as defined in (A.4).
Step 1:
Bounding the distance between x i ( T ) and x i +1 ( T )
Let us consider some fixed i .
Let δ i =
T/
i denote the stepsize, and let K := 2 i so that T
=
Kδ i .
Let x i ( t ) =
Φ( T ; x (0) , E, β, B , i ) be as defined in (A.3).
Recall that x i ( t ) is by definition the linear interpolation of x ik (defined in (4.1) or equivalently (A.2)), which are marginally Euler-Murayama sequences with stepsize δ i .
Our goal is to bound E (cid:104) d
( x i ( T ) , x i +1 ( T )) 2 (cid:105) .
Lemma 4 almost gives us what we want; the problem is that Lemma 4 assumes that for all x ∈ M , ∥ β ( x ) ∥ ≤ L β , but we do not make that assumption in this lemma.
In order to get around this issue, we use an argument based on truncating β at larger and larger norms .
Let us define L 0 :=
∥ β ( x (0))
∥ and β j ( y ) :
= (cid:40) β
( y ) for ∥ β ( y )
∥
≤ L 1 2 j/ 2+ β ( y ) · L 1 2 j/ 2+1 ∥ β ( y ) ∥ for ∥ β ( y ) ∥ > L 1 2 j/ 2+1 , (A.13) i.e. β j is the truncated version of β , so that the norm of β j is globally upper bounded by L 1 2 j/ 2+1 .
Given this definition of β j , we now define, for all j ∈ Z + , ˜ x i,j ( t ) :=
Φ( t ; x (0) , E, β j , B , i ) , which is the (interpolated) Euler-Murayama discretization from (A.3), with step-size δ i , and drift β j .
In other words, x i ( t ) and ˜ x i,j ( t ) are both Euler Murayama discretizations with stepsize δ i , but the former has drift β whereas the latter has drift β j .
We also let ˜ x · ,j ( t ) :
=
Φ( t ; x (0) , E, β i , B ) denote the limit, as i → ∞ , of ˜ x i,j ( t ) (see definition in (A.4)).
By Young’s Inequality,
[Equation] [Equation]
Before proceeding, we briefly explain the intuition behind the decomposition in (A.14).
Notice that we let j = i in ˜ x
i,i , i.e. as i increases, two this happen: δ i becomes smaller, and β i becomes truncated at a larger norm (see (A.13)), and is thus closer to the true un-truncated β .
The first and second term on the right hand side of (A.14) correspond to error due to truncating β to β i .
These two terms would equal
if sup t d( x i ( t ) , ˜ x i,i ( t )) ≤ L 1 2 i / 2 L ′ β , as that implies (via Assumption 3) that sup t (cid:13)(cid:13) β ( x i ( t ))
(cid:13)(cid:13) ≤
L 1 2 i/ 2+1 (so along the entire path, β was never large enough to require truncation).
As i increases, L 1 2 i / 2 L ′ β → ∞ , and these two truncation errors are 0 with increasing probability.
The last term in (A.14) corresponds to error due to different discretization stepsize ( δ i vs δ i +1 ).
Step 1.1:
Bounding distance of truncated-drift sequences ˜ x
i,i and ˜ x i +1 ,
i
We first bound the last term of (A.14).
By the first claim of Lemma 4, we can bound
[Equation]
where the first inequality uses the fact that δ i
= T/ 2 i ≤ T and that δ i L 1
i +
= 2 TL 1 , and second inequality is by our assumed upper bound on T .
Here, we crucially use the fact that, the effect of δ i halving with i "cancels out" the effect of (cid:13)(cid:13) β i ( x ) (cid:13)(cid:13) becoming larger with i .
Step 1.2: Bounding error due to truncation
We now bound the first two terms of (A.14).
Once again recall from the definition in (4.2) and (A.3)
that x i ( t ) (resp ˜ x
i,i ( t ) ) are linear interpolations of the discrete sequence x ik (
resp ˜ x
i,ik ) as defined in (4.1).
Under the event sup k ∈{ 0 ... 2 i } d( x ik , x (0)) ≤ 2 i/ 2 L 1 L ′ β , we verify that x ik = ˜ x
i,ik for all k ∈ (cid:8) 0 ...
2 i (cid:9) – this is because we can then bound, for all k , (cid:13) (cid:13) β
( x ik
[Equation]
which in turn implies that for all k , β ( x ik ) equals the truncated version
β i ( x ik ) , which in turn implies that d( x i ( T ) ,
˜ x i,i ( T ))
= 0 .
Therefore,
[Equation]
where the second line follows from Young’s inequality and Cauchy Schwarz.
From Lemma
[Equation]
and our assumed bound on T ,
[Equation]
Also from Lemma 16, for any k ,
[Equation]
The same upper bound also applies to E (cid:104)
d(˜ x i,i ( T ) , x (0)) 4 (cid:105) .
Plugging into (A.15),
[Equation]
Note that the bound in Lemma 16 is strictly stronger for x i +1 k compared to x ik .
Thus by exactly identical steps, we can also upper bound
[Equation] [Equation]
Plugging into (A.14), and applying Young’s Inequality, we finally have our bound between the
Euler-Murayama sequences x i ( T ) and x i +1 ( T ) (without truncation of β ).
[Equation]
where we use the assumed upper bound on T in the lemma statement.
Step 2: Summing over i
For any ℓ ∈ Z + , we can summing (A.16) for i ∈ { 0 , 1 , 2 ...ℓ } to bound
E
[Equation]
The first inequality uses triangle inequality and Young’s inequality recursively: for any i ,
[Equation]
Since (A.17) holds for all ℓ , we take the limit of ℓ → ∞ .
By dominated convergence together with
Lemma 5,
[Equation]
A.4 Proof of Theorem
Below, we provide the full proof of Theorem 1, which was sketched in Section 5.
The main results used are Lemma 3 (contraction of Lyapunov function under exact SDE) and Lemma 7 (bound on
Euler Murayama discretization error).
Proof of Theorem 1.
Step 0: Defining some Key Constants
In this step, we define a radius r , an event A k based on r , and an upper bound on δ .
[Equation]
A
[Equation]
Let A k denote the event max i ≤ k d( x i , x ∗ ) ≤ r .
The value of r and A k are chosen so that (A.23) holds in Step 1 below.
Note that r depends on log(1 /δ ) on the last line.
We now define a suitable upperbound on the stepsize δ .
To do so, we first define the constants c 1 , c 5 , c 6 , c 7 : c 1 :
[Equation]
[Equation]
For our proof, we require that δ satisfies
[Equation]
Thus C 0 from the theorem statement is explicitly C 0 =
[Equation]
6 ,c 7 } .
The motivation for this upper bound on δ is so that δ satisfies the conditions in Lemma 7 and Lemma 20 which are used in Stepand Step 3.
We provide details in Step 4 below.
Note that the upper bound on δ depends only on rbut not r , so the definition is not circular.
Step 1: Tail Bound:
We now show that with high probability, the discretization sequence x k never steps outside the ball of radius r centered at x ∗ (this is exactly the event A k that we defined in the previous step).
By Lemma 20 with σ = 0 and ˜ β
= β , for any ( δ, r ) satisfying δ ≤ min
[Equation]
, we can bound the probability of x k stepping outside the ball of radius r centered at x ∗ , for any k ≤ K , as
[Equation]
Furthermore, we can bound the fourth-moment of the distance between x K and x ∗ : by Lemmawith ξ k ( x k ) =
ζ k and σ ξ = 2 √ d , we can bound
[Equation]
Similarly, we can use Lemma 18 to bound
[Equation]
Together, we can bound the expected distance between y ( Kδ ) and x K under the low probability event A cK .
By choosing r to be sufficiently large in (A.18), we make (A.22) sufficiently small.
We then apply triangle inequality to bound d( y ( Kδ ) , x K ) ≤ d( y ( Kδ ) , x ∗ )
+ d( x K , x ∗ ) followed by
Cauchy Schwarz, and verify that
[Equation]
Step 2: Continuous Time Contraction
Having established a bound in the distance under the event A cK , we now turn our attention to the high-probability event A K .
Consider some fixed but arbitrary k ≤ K .
Let us define a continuous time SDE ¯ x k ( t ) , for t ∈ [ kδ, ( k + 1) δ ] , as the solution to the exact Langevin diffusion (1.2) initialized at ¯ x k ( kδ ) :
= x k .
The goal of this step is to bound E (cid:2) 1 { A k } d( y (( k + 1) δ ) , ¯ x k (( k + 1) δ )) (cid:3) in terms of
E (cid:2) 1 { A k } d( y ( kδ ) , ¯ x k ( kδ )) (cid:3) :
= E [ 1 { A k } d( y ( kδ ) , x k )] .
We apply Lemma 3, which guarantees that there exists a coupling between the two exact SDE processes y and ¯ x k , such that E (cid:2) f
(d( y
[Equation]
t ) , ¯ x k ( t )))
[Equation]
contracts with rate α , i.e.
[Equation]
where f is a Lyapunov function satisfying f ( r ) ≥ 12 exp ( − ( q + L Ric / 2) R 2 / 2) r and | f ′ ( r ) | ≤ 1 , and α := min (cid:110)
m − L Ric / 2, 12 R 2 (cid:111)
· exp ( − 12 ( q + L Ric / 2) R 2 ) are as defined in Lemma 3, and are consistent with the definition in our theorem statement.
Step 3: Euler Murayama Error
Next, we bound the distance between x k +1 and ¯ x k (( k + 1) δ ) .
This represents the discretization error between a single Euler-Murayama step with stepsize δ , and the exact Langevin diffusion over δ time.
We will apply Lemma 7 with L 1 = L ′ β r .
We verify that under the event A k and by Assumption 3, ∥ β ( x k ) ∥ is indeed bounded by L 1 .
Thus by Lemma 7,
[Equation]
where ˜ O hides polynomial dependency on L ′ β , d, L R , R , log K, log(1 /δ ) .
Combining (A.24) and (A.25) and using triangle inequality and the fact that | f ′ | ≤ 1 , along with the fact that A k +1 ⊂
A k , we can bound
[Equation] [Equation]
where ˜ O hides polynomial dependency on L ′ β , d, L R , R , log K, log(1 /δ )
This shows that, in one δ -time step, the Lyapunov function of the distance contracts with rate α , plus a discretization error of order δ 3 / 2 .
Applying (A.26) recursively, we can bound
[Equation]
[Equation]
Combining (A.23) and (A.27), and using the fact that f ( r ) ≤ r , gives
[Equation]
Using the fact that 12 exp ( − ( q + L Ric / 2) R 2 / 2) r ≤ f ( r ) ≤ r , we have
[Equation]
where ˜ O hides polynomial dependency on L ′ β , d, L R , R ,
[Equation]
m
[Equation]
L
[Equation]
/ 2 , log K, log 1 δ .
This concludes the proof of Theorem 1.
Step 4: Verifying Conditions on δ
In the proof above, we applied Lemma 7 and Lemma 20.
Each of these requires certain bounds on δ .
In this step, we verify that the conditions on δ for each of these lemmas is satisfied by (A.21).
Lemma 7 with L 1 = L ′ β r requires
[Equation]
We verify that this follows from (A.19).
We specifically verify that δ ≤
[Equation]
16 √ L R L ′ β r is satisfied due to the last two terms in (A.19), and Lemma 37.
Lemma 20 requires requires δ ≤ min
[Equation]
This is satisfied by (A.20) and Lemma 37.
A.5 Proof of Theorem
Below, we provide the full proof of Theorem 2, which was sketched in Section 6.
The main results used are Lemma 8 (contraction of distance under Euler Murayama step) and Lemma 7 (bound on
Euler Murayama discretization error).
Proof of
Theorem 2.
Step 0: Defining some Key Constants
In this step, we define a radius r , an event A k based on r , and an upper bound on δ .
[Equation]
Let A k denote the event max i ≤ k max { d( x i , x ∗ ) , d( y ( iδ ) , x ∗ ) } ≤ r .
The value of r and A k are chosen so that (A.36) holds in Step 1 below.
Note that r depends on log(1 /δ ) on the last line.
We now define a suitable upperbound on the stepsize δ .
To do so, we first define the constants c 1 , c 2 , c 3 , c 4 , c 5 , c 6 , c 7 : c 1 := min
[Equation]
c 2 :
[Equation]
128 L
[Equation] [Equation]
17 d
[Equation]
For our proof, we require that δ satisfies
[Equation]
{ c
[Equation]
c
[Equation]
c
[Equation]
c
[Equation]
c
[Equation]
c
[Equation]
c 7 }
[Equation]
(A.33)
Thus C 1 from the theorem statement is explicitly C 1 = 1 min { c 1 ,c 2 ,c 3 ,c 4 ,c 5 ,c 6 } .
The motivation for this upper bound on δ is so that δ satisfies the conditions in Lemma 7, Lemma 8 and Lemma 20 which are used in Steps 1-3.
We provide
Step 1: Tail Bound:
We now show that with high probability, the discretization sequence x k and the exact SDE y ( t ) never step outside the ball of radius r centered at x ∗ (this is exactly the event A k that we defined in the previous step).
By Lemma 20, for any ( δ, r ) satisfying δ ≤ min
[Equation]
, we can bound the probability of x k stepping outside the ball of radius r centered at x ∗ , for any k ≤ K , as
[Equation]
On the other hand, by Lemma 2, for T =
Kδ , there is a family of discrete sequences, y ik , corresponding to (1.2) with stepsize δ i
=
T/ 2 i , whose linear interpolation y i ( t ) converges to y ( t ) uniformly almost surely.
By Lemma 20 with ˜ β = β , and σ = 0 , and stepsize δ i , and iteration number K i :=
i
(so that T =
K i δ i ), for i sufficiently large, we can bound
P (max k ≤
[Equation]
i d
[Equation]
y ik ,
[Equation]
≤
[Equation]
exp
[Equation]
d ) .
Taking the limit of i → ∞ , we can bound
[Equation]
Combining (A.34) and (A.35), we can bound
[Equation]
Furthermore, we can bound the fourth-moment of the distance between x K and x ∗ : by Lemmawith ξ k ( x k ) = √ δ (˜ β k
( x k ) − β ( x k ))
+ ζ k and σ ξ = 2( σ + √ d ) , we can bound
[Equation]
Similarly
[Equation]
we can use Lemma 18 to bound
[Equation]
Combining the above
[Equation]
we can bound the expected squared distance between y ( Kδ ) and x K under the low probability event A cK .
By choosing r to be sufficiently large in (A.28)
[Equation]
we make (A.35) sufficiently small.
We then apply Young’s inequality to bound d( y ( Kδ ) , x K ) 2 ≤ 2d( y ( Kδ ) , x ∗ ) 2 + 2d( x K , x ∗ ) 2 followed by Cauchy Schwarz, to verify that
[Equation]
Step 2: Discrete Contraction
Having established a bound in the distance under the event A cK
[Equation]
we now turn our attention to the high-probability event A K .
Consider some fixed but arbitrary k ≤ K .
We define a useful intermediate variable, representing a single Euler-Murayama step initialized at y ( kδ ) (recall that y ( t ) corresponds to the exact SDE):
[Equation]
where ¯ ζ k ∼ N y ( kδ ) (0 , I ) .
Note that ¯ y k +1 evolves according to the Euler-Murayama step with exact drift β
( y ( kδ )) , whereas x k +1 evolves according to the Euler-Murayama step with stochastic drift ˜
β k ( x k ) .
Under the strong-convexity-like condition due to Assumption 2 with R = 0 , we can show that a single discrete Euler Murayama step leads to contraction in distance between ¯ y k +1 and x k +1 .
From
Lemma 8, there exists a coupling such that
[Equation]
Step 3: Euler Murayama Error
Next, we bound the distance between ¯ y k +1 and y (( k + 1) δ ) .
This represents the discretization error between a single Euler-Murayama step with stepsize δ , and the exact Langevin diffusion over δ time.
We will apply Lemma 7 with L 1 = L ′ β r .
We verify that under the event A k , ∥ β ( y ( kδ ))
∥ is indeed bounded by L 1 .
Thus by Lemma 7,
[Equation]
where ˜ O hides polynomial dependency on L ′ β , d, L R , R , log K, log(1 /δ )
This shows that, in one δ -time step, the distance contracts with rate m − L Ric / 2 , plus a discretization error of order δ 2 .
Combining (A.37) and (A.38) and using Young’s inequality inequality, together with the fact that
A k +1 ⊂
A k .
[Equation]
where ˜ O hides polynomial dependency on L
[Equation]
β , d, L R , R , σ, log K, log(1 /δ ) .
Applying the above recursively, we can bound
[Equation]
Combining (A.36) with (A.39) gives
[Equation]
where ˜ O hides polynomial dependency on L ′ β , d, L R , R , σ, 1 m
− L Ric / 2 , log K, log 1 δ .
This concludes the proof of Theorem 2.
Step 4: Verifying Conditions on δ
In the proof above, we applied Lemma 7, Lemma 8 and Lemma 20.
Each of these requires certain bounds on δ .
In this step, we verify that the conditions on δ for each of these lemmas is satisfied by (A.33).
Lemma 7 with L 1 = L ′ β r requires
[Equation]
We verify that this follows from (A.29).
We specifically verify that δ ≤ 1 16 √
[Equation]
R L
[Equation]
β r is satisfied due to the last two terms in (A.29), and Lemma 37.
Lemma 8 with L 1 = rL ′
β requires δ ≤ min
[Equation]
= min
[Equation]
The first 5 bounds in (A.40) follow from (A.30).
The last two bounds in (A.40) follow from (A.31) and Lemma 37.
Lemma 20 requires requires δ ≤ min
[Equation]
This is satisfied by (A.32) and Lemma 37.
The following lemma shows that, for any two initial points x and y , if x undergoes an exact Euler
Murayama step with drift β , and y undergooes a stochastic Euler Murayama step with drift ˜ β , then their expected squared distance contracts, with rate m − L Ric / 2 , plus an additional error of δ 2 σ 2 , where σ = (cid:13)(cid:13)(cid:13) ˜
β − β (cid:13)(cid:13)(cid:13) .
This lemma is somewhat analogous to Lemma 3 which shows contraction under the exact SDE, though Lemma 8 also requires a fair amount of additional discretization analysis.
The key result used in the proof of Lemma 8 is Lemma 29.
Lemma 8.
Let M satisfy Assumption 1, Assumption 4.
Assume in addition that there exists
a constant
L ′ R such that for all x ∈ M , u, v, w, z, a ∈ T x M , ⟨ ( ∇ a R )( u, v ) w, z ⟩ ≤ L ′ R ∥
u
∥∥ v ∥∥ w ∥∥
z ∥∥ a ∥ .
Let β be a deterministic vector field satisfying Assumption 3 and Assumption 2, with R = 0 .
Let x, y ∈ M be arbitrary.
Let ˜ β be a random vector field such that E (cid:104) ˜
β (cid:105) =
β .
Assume that there exists σ ∈ R + such that (cid:13)(cid:13)(cid:13) ˜
β
( y ) − β
( y )
(cid:13)(cid:13)(cid:13) ≤ σ .
Let L β :=
max {∥ β ( x ) ∥ , ∥ β ( y ) ∥} .
Let x ′ =
Exp x ( δβ ( x ) + √ δζ ) .
Let y ′
=
Exp y ( δ ˜ β
( y )
+ √ δ ˜ ζ ) , where ζ ∼ N x (0 , I ) and ˜
ζ ∼ N y (0 , I ) .
Assume that
[Equation]
13 L R L
[Equation]
β d , (
[Equation]
− L Ric /
[Equation]
2 24 L
[Equation]
R 2 d, (cid:115) m − L Ric /2 14 ( L 3 β + σ 3 ) L ′ R , m − L Ric /2 17 d 2 L 2 R (cid:41) .
Then there is a coupling (synchronous coupling) between ζ and ˜ ζ such that
[Equation]
Note: elsewhere in this paper, we have used L β do denote a Lipschitz constant for β ; the use of L β in Lemma 8 is different (but related).
Proof.
Let γ ( s ) :
[0 , 1] → M be a minimizing geodesic between x and y with γ (0)
=
x and γ (1) =
y , such that ⟨ Γ xy β ( y ) − β ( x ) , γ ′ (0) ⟩ ≤ − m d( x, y ) 2 .
(Assumption 2 guarantees the existence of such a γ .)
Step 1: Synchronous Coupling of ζ and ˜ ζ
We will now define a coupling between ζ and ˜ ζ .
Let E be an orthonormal basis at T x M , and let F be the parallel transport of E along γ , i.e. F is an orthonormal basis for T y M .
Let ζ ∼ N (0 , I ) be a standard Gaussian random variable in R d , and define ζ := ζ ◦ E , and it follows by definition that ζ so defined has distribution N x (0 , I ) .
Let ˜ ζ :=
˜ ζ ◦ F , it follows by definition that ˜ ζ has distribution
N y (0 , I ) .
Step
Applying Lemma 29 and Simplifications
We will apply Lemma 29 with u = δβ ( x ) + √ δζ and v = δ ˜ β
( y )
+ √ δ ˜ ζ .
Then d(Exp x ( u ) , Exp y ( v ))
2 − d( x, y )≤ 2 ⟨ γ ′
[Equation]
where C := √ L R ( ∥ u ∥ + ∥ v ∥ ) and C ′ :
=
L ′ R ( ∥ u ∥ + ∥ v ∥ ) 3 , and u ( t ) and v ( t ) are parallel trannsport of u and v along γ , as defined in Lemma 29.
Some notes on notation:
We will use Γ yx and Γ γ ( t ) γ ( s ) to denote parallel transport along γ . 2.
In subsequent parts, for i = 1 , 2 .... , we will use
τ i to denote terms which depend super-linearly on δ .
ξ i , to denote terms which have 0 expectation.
θ i to denote terms which depend linearly on δ , and have non-zero expectation (i
[Equation]
e. the important terms).
Step 2.1: Simplifying
By definition, v (0) − u (0) =
δ (Γ xy ˜ β
( y ) − β ( x )) , thus
[Equation]
Step 2.2: Simplifying
By similar algebra as Step 2.1,
[Equation]
Step 2.3: Simplifying
[Equation]
[Equation]
We will now bound τ 2 and τ 3 .
By Assumption 3,
[Equation]
Step 2.4: Simplifying
Since 4 has quite a few terms
[Equation]
we will bound them one by one:
[Equation]
(cid:125) :
= τ+ 18 C 4 e 2 C ∥ v (0) − u (0) ∥(cid:124) (cid:123)(cid:122) (cid:125) :
= τ+ 18 C 4 e 2 C d( x, y )(cid:124) (cid:123)(cid:122) (cid:125) :
= τ+ 4 C ′ d( x, y )(cid:124) (cid:123)(cid:122) (cid:125) :
= τ+ 4 C 2 e 2 C d( x, y )
∥ v (0) − u (0) ∥ (cid:124) (cid:123)(cid:122) (cid:125) :
= τ.
Recall that C := √ L R ( ∥ u ∥ + ∥ v ∥ ) and C ′ :
=
L ′ R ( ∥ u ∥ + ∥ v ∥ ) 3 .
Following previous calculations, we can bound ∥ v (0) − u (0) ∥
≤ δσ + δL ′
β d( x, y ) .
We can also bound ∥ u ∥ ≤ δL β + √ δ ∥ ζ ∥ and ∥ v ∥ ≤ δL β + δσ + √ δ ∥ ζ ∥ .
Thus
[Equation]
where for the third line, we use the fact that our bound on δ implies that δ ≤ min (cid:110)
[Equation]
(cid:111) .
By similar algebra, we verify that
[Equation]
β + σ 4 ))
· d( x, y ) 2 , τ= 4 C ′
d( x, y )≤ 128 L ′ R e 4 √ δ ∥ ζ ∥ · ( δ 3 / 2 ∥ ζ ∥ 3 + δ 3 ( L 3 β + σ 3 ))
· d( x, y ) 2 , τ= 4 C 2 e 2 C d( x, y )
∥ v (0) − u (0) ∥
≤ 128 L R e 4 √ δ ∥ ζ ∥ · ( δ ∥ ζ ∥ 2 + δ 2 ( L 2 β + σ 2 ))
· ( δL ′ β d( x, y ) 2 + δσ d( x, y )) ,
E (cid:2) τ 46 (cid:3) 1 / 4 ≤ 2 14 L 2 R · ( δ 2 d 2 + δ 4 ( L 4 β + σ 4 ))
( δ 2 σ 2 + δ 2 L ′ β 2 d( x, y ) 2 ) ,
E (cid:2) τ 47 (cid:3) 1 / 4 ≤ 2 14 L 2 R · ( δ 2 d 2 + δ 4 ( L 4 β + σ 4 )) ·
d( x, y ) 2 ,
E (cid:2) τ 48 (cid:3) 1 / 4 ≤ 512 L ′ R · ( δ 3 / 2 d 3 /
+ δ 3 ( L 3 β + σ 3 ))
· d( x, y ) 2 ,
E (cid:2) τ 49 (cid:3) 1 / 4 ≤ 512 L R · ( δd + δ
( L 2 β + σ 2 )) · ( δL ′ β d( x, y ) 2 + δσ d( x, y )) , (A.45) where we use Lemma 21 and the fact that our assumption on δ implies that δ ≤ 1 2 14 L R d .
Step 3: Putting Things Together
Let E [ · ] denote expectation wrt the ζ and ˜ β (which is a random function).
By Assumption 2,
E [ θ 1 ] ≤ − 2 δm d( x, y ) 2 .
By definition of Ricci curvature and by Assumption 1,
E [ θ
[Equation]
] ≤ δL Ric .
Using our assumed bound on δ , we verify that
E
[Equation]
We omit the proof for this fact, since it follows by basic but long algebra, but note that we need to apply Young’s inequality at several points.
We also verify that
[Equation]
Plugging everything into (A.41), we get
[Equation]
B Distance Contraction under Kendall Cranston Coupling
In this section, we prove Lemma 3, which is the main tool for proving mixing of manifold diffusion processes under the distant dissipativity assumption.
We note again that the proof is entirely based on existing results from [Eberle, 2016, Hsu, 2002], and is only included for completeness.
B.1 The Kendall Cranston Coupling
Lemma 9.
Let T ∈ R + be some fixed time.
Assume that there is are constants L β , L ′ β such that for all x, y ∈ M , ∥ β ( x ) ∥ ≤ L β and (cid:13)(cid:13) β ( x ) − Γ xy β
( y ) (cid:13)(cid:13) ≤ L ′ β
∥ x − y ∥ .
Let i be some integer satisfying i
[Equation]
For any x, y , let Λ( x, y ) denote the set of minimizing geodesics from x to y , i.e. for any γ ∈ Λ( x, y ) , γ (0) = x , γ (1) =
y , ∀ t, ∇ γ ′ ( t ) γ ′ ( t )
=
and d( x, y ) =
∥ γ ′ (0) ∥ .
Let κ ( r ) :
=r 2 sup d( x,y )=
r inf γ ∈ Λ( x,y ) (cid:10) Γ xy β
( y ) − β ( x ) , γ ′ (0) (cid:11) .
Let x, y ∈ M and let E x be an arbitrary orthonormal basis of T x M and let E y be an arbitrary orthonormal basis of T y .
let x i ( t ) :=
Φ( t ; x, E x , β, B x , i ) and y i ( t ) :=
Φ( t ; y, E y , β, B y , i ) where
B x and B y are standard Brownian motion in R d , and where Φ is as defined in (A.3) .
For any ε , there exists a coupling between B x and B y , and Brownian motion W i over R , such that for all k ∈ (cid:8) 0 ...
2 i (cid:9) ,
[Equation]
where τ ik satisfies
[Equation]
where C 1 is a constant depending on L R , L ′ R , d, T .
Proof.
We set up some notation: throughout this proof, consider a fixed i .
Recall that δ i :=
T/ 2 i , and assume i is large enough such that δ i ≤ 1 32 √ L R L β .
Let x ik be as defined in (4.1) so that x ik = x
i
kδ i ) .
Let us also define K := 2 i , so that T
=
Kδ i .
Step 1: defining the coupling By definition, for any k ∈ { 0 ...K } ,
[Equation]
Let γ ik : [0 , 1] → M denote a minimizing geodesic from x ik to y ik .
Let F ik be an orthonormal basis at T y ik M , obtained from the parallel transport of E ik along γ ik , i.e. for all j = 1 ...d ,
F i,jk =
Γ γ ik E i,jk
Let us define M ik ∈ R d × d as matrix whose a, b entry is
[Equation]
one can verify that M ik is an orthogonal matrix, and that for all v ∈ R d , v ◦ F ik =
Mv ◦ ˜ E
ik .
Let us define ¯ ν ik denote the unique coordinates of γ ik ′ (1) ∥ γ ik ′ (1) ∥
wrt F ik (equivalently the coordinates of γ ik ′ (0) ∥ γ
ik ′ (0) ∥
wrt E ik ).
We define ν ik := 1 (cid:8) d( x ik , y ik ) > ε (cid:9) ¯ ν ik .
We now define a coupling between B x ( t ) and B y ( t ) as follows:
[Equation]
For this to be a valid coupling, it suffices to verify that (cid:82) T 0 1 (cid:8) t ∈
[ kδ i , ( k + 1) δ i ) (cid:9) M ik
( I − 2 ν ik ν ikT )
d B x ( t ) is indeed a standard Brownian motion.
This can be done by verifying that the definition satisfies Levy’s characterization of Brownian motion.
We omit the proof, but highlight two important facts: 1.
(cid:82) T 0 1 (cid:8) t ∈
[ kδ i , ( k + 1) δ i ) (cid:9) M ik
( I − 2 ν ik ν ikT ) is adapted to the natural filtration of B x ( t ) , and 2.
M ik ( I − 2 ν ik ν ikT ) is an orthogonal matrix.
We have thus defined a coupling between B x and B y , and consequently, a coupling between x i ( t ) and y i ( t ) for all t .
Step 2: Applying Lemma
Having defined a coupling between x ik and y ik , we bound E (cid:104) d( x iK , y iK ) 2 (cid:105) for K :
=
T/δ i =
2 i by applying Lemma 29 , with x = x ik , y = y
ik , u = δ i β ( x ik ) + ( B (( k + 1) δ i ) − B ( kδ i )) ◦ E ik , v = δ i β ( y ik )
+ ( ˜ B (( k + 1)
δ i ) − ˜ B ( kδ i )) ◦
˜ E ik and γ := γ ik .
Following the notation in Lemma 29, let u ( t ) and v ( t ) be the parallel transport of u and v along γ ( t ) .
We verify that u ( s ) =
δ i Γ γ ik ( s ) x
ik
[Equation]
E ik and that
[Equation]
E ik
[Equation]
where the second equality is by definition of M ik , the third equality is by definition of F ik , the fourth equality is by definition of ν ik and the fact that γ ik is a geodesic.
It is convenient subsequently to note the following:
[Equation]
ik ′ ( s ) (cid:13)(cid:13)(cid:13) and
[Equation]
( s
[Equation]
(cid:13)(cid:13)(cid:13) γ ik
[Equation]
( s ) (cid:13)(cid:13)(cid:13)
Step 3: Reorganizing Lemma
With u, v as defined above, Lemma 29 implies that
[Equation]
where C := √ L R ( ∥ u ∥ + ∥ v ∥ ) and C ′ :
=
L ′ R ( ∥ u ∥ + ∥ v ∥ ) 3 .
Below, we bound each of the terms above
[Equation]
In the first equality above, we crucially use the fact that (cid:10) ν
ik , B (( k + 1) δ i ) − B ( kδ i ) (cid:11) γ
ik ′ ( s ) ∥ γ
ik ′ ( s ) ∥ is a scalar multiple of γ ′ k ( s ) , and the fact that ⟨ R ( u, u ) v, u ⟩ = ⟨ R ( u, v ) u, u ⟩ = 0 for all u, v by symmetry of the Riemannian curvature tensor.
Finally, we will take the remaining terms, and denote them by
[Equation]
We claim that under our assumption on i ,
[Equation]
y
[Equation]
where O () hides dependencies on L R , L ′ R , d, T .
We omit the proof for the above claim, which involves some tedious but straightforward algebra, but we note that the proof uses E (cid:104)(cid:13)(cid:13) B (( k + 1) δ i ) − B ( kδ i ) (cid:13)(cid:13)
j 2 (cid:105) =
O ( δ ij/ 2 ) (for all integer j ) and that
E (cid:2) exp ( a (cid:13)(cid:13) B (( k + 1) δ i ) − B ( kδ i ) (cid:13)(cid:13) 2 ) (cid:3) ≤ 4 exp (2 a 2 δ i d ) ≤ 8 for δ i a 2 ≤ 1 / 32 (see Lemma 21).
It is also important to use our assumption on δ i in the lemma statement.
We simplify (B.1) to d
[Equation]
ik
[Equation]
y ik
[Equation]
−
[Equation] [Equation]
Step 4: Pulling out the expectation
We will further simplify (B.2) by replacing a few terms by their expectations.
Define τ ik, 4 :=
[Equation]
By definition of Ricci Curvature and by Assumption 1,
[Equation]
where Ric denotes the Ricci curvature tensor.
By definition of
[Equation]
Let τ ik :=
τ ik, 1 + τ ik, 2 + τ ik, 3 .
We can thus further simplify (B.2) to
[Equation]
d( x ik , y ik )≤ 2 δ i κ (d( x ik , y ik ))
d( x
[Equation]
, y ik )
[Equation] [Equation]
4 δ
[Equation]
(cid:8) d( x ik , y ik ) > ε (cid:9) − δ
i
L Ric d
[Equation]
x ik
[Equation]
y ik
[Equation]
+ τ
[Equation]
≤ δ i (2 κ (d( x ik , y ik ))
+ L Ric
[Equation]
d
[Equation]
x ik
[Equation]
y
[Equation] [Equation]
x
[Equation]
y
[Equation]
+
[Equation]
ik (B.3) the conclusion follows by defining W i ( t ) :
= (cid:82) t 0 1 (cid:8) t ∈
[ kδ i , ( k + 1) δ i ] (cid:9) (cid:10) ¯ ν ik , B (( k + 1) δ i ) − B ( kδ i ) (cid:11) and verifying that it is a Brownian motion.
(Recall our definition that ν ik := 1 (cid:8) d( x ik , y ik ) > ε (cid:9) ¯ ν ik )
B.2 Lyapunov function and its smooth approximation
In this section, we consider a Lyapunov function f taken from Eberle [2016].
By analyzing how f (d( x ik , y ik ))
evolves under the dynamic in Lemma 9, one can demonstrate that the distance function contracts.
Let L , R ∈ R + .
We will see later that L and R will correspond to distant-dissipativity parameters in (2).
Let ε ∈ [0 , ∞ ) .
One should think of ε as being arbitrarily small, as eventually we are only interested in the limit as ε → 0 .
Define functions ψ ε ( r ) , Ψ ε ( r ) and ν ( r ) , all from R + to R :
[Equation]
We defined an ε -smoothed Lyapunov function as
Definition 1.
[Equation]
The case when ε = 0 (when there is no smoothing) will be of particular interest to us:
Definition 2.
[Equation]
Remark 2.
The Lyapunov function from Eberle [2016] is more general, but for the specific case of
L , R distant dissipative functions, it is equal to f as defined in (2).
Lemma 10.
Assume ε ∈
[0 , 1 / (4 √L )] , then f ε as defined in (1) satisfies
f ε ( r ) ∈
[12 exp ( − (1 + ε ) LR 2 / 2) r, r ] for all r 2 .
f ′ ε ( r ) ∈
[ 1 2 exp ( − (1 + ε ) LR 2 / 2) , 1] for all r 3 .
f ′′ ε ( r ) ∈ [ − 4 L 3 / 2 , 0] for all r 4 .
f ′′ ε ( r ) + L rf ′ ε ( r ) ≤ − exp ( − (1 + ε ) LR 2 / 2)
[Equation]
1 +
[Equation] [Equation]
R
[Equation]
f
[Equation]
r ) for r ∈ [0 , R ]
If in addition, ε > 0 , f ε satisfies
[Equation]
√L
[Equation]
for all r
Proof.
We can verify that
[Equation]
1. follows from integrating 2. 2. follows from ν ε ( r ) ∈
[1 / 2 , 1] and ψ ε ∈ [exp ( − (1 + ε ) LR 2 / 2) , 1] and the expression for f ′ ε ( r ) above.
3. follows from µ ε , ψ ε , ν ε ≥ 0 and ν ′ ε ≤ 0 , and the fact that rψ ε ( r ) ≤ 2 √L and (B.4).
is a little more involved.
First note that over r ∈ [0 , R ] , µ ε ( r ) = 1 .
This will simplify some calculations.
From the expression for f ′′ ε above, we verify
[Equation]
We can bound the denominator as
[Equation]
where the first inequality is by µ ε ( s ) ≤ 1 , and µ ε ( r )
=
for r ≥ R + ε the second inequality is by ψ ε ( r ) being monotonically decreasing, and the third inequality is by Ψ ε ( r ) ≤ r .Finally, note that Ψ ε ( r ) ≥ f ε ( r ) .
Put together,
[Equation]
We now prove the bound for 5.
It is useful to recall that ψ ε ( r ) ≤ 1 and ν ε ( r ) ≤ 1 .
[Equation]
For r ∈ [0 , R + ε ] ( ν ′′ ε = 0 outside this range),
[Equation]
√L
[Equation]
We can thus bound | f ′′′ ( r ) | as
[Equation]
Lemma 11.
Assume ε ∈ (0 , 1 / (4 √L )]
g ′ ε ( s ) =
[Equation]
g
[Equation]
s
[Equation]
s
[Equation]
g ′′′ ε ( s )
[Equation]
( s +
[Equation]
√
[Equation]
+
[Equation]
for all s
where O () notation hides dependency on L and R .
Proof.
The first 3 points follow from chain rule.
The last point follows from point 5 from Lemma 10.
[Equation]
B.3 Contraction of Lyapunov Function under Kendall Cranston Coupling
Lemma 12.
Consider the same setup as Lemma 9.
For any x, y , let Λ( x, y ) denote the set of minimizing geodesics from x to y , i.e. for any γ ∈ Λ( x, y ) , γ (0) =
x , γ (1) =
y , ∀ t, ∇ γ ′ ( t ) γ ′ ( t ) =
and d( x, y ) =
∥ γ ′ (0) ∥ .
Let κ ( r ) :=
1 r 2 sup d( x,y )=
r inf γ ∈ Λ( x,y ) (cid:10) Γ xy β
( y ) − β ( x ) , γ ′ (0) (cid:11) .
Assume there exists R ≥ 0 , q ≤ 0 such that κ ( r ) ≤ q for all r ≤ R .
Let L = q + L Ric / 2 .
Let ε ∈
(0 , 1 / (4 √L )] .
Let g ε be as defined in 1 with parameters L and R .
Let F k denote the natural filtration generated by x ik and y ik .
There exists a constant c 1 , depending on L β , L ′ β , L R , T, d , and some constant c 2 , depending on
L ′ β , L Ric , R such that for any i > c 1 and ε > c 2 , there exists a coupling between x ik and y ik such that
[Equation]
where O () hides dependency on L R , L ′ β , T, d .
Proof.
Let us define, for convenience, r k :=
d( x ik , y ik ) .
By Lemma 9, for any i and any ε , there exists a coupling satisfying
[Equation]
where τ ik satisfies
[Equation]
where O () hides dependencies on L R , L ′ R , d, T .
By third order Taylor expansion,
[Equation]
The last line uses two facts: 1.
From Lemma 13, for any j , there exists a constant C , depending on T, d, L R , L ′ β , but independent of L β , such that for all i, k , E (cid:104) d( x ik , x 0 ) 2 j (cid:105) <
C and E (cid:104) d( y ik , y 0 )
2 j (cid:105) <
C .
Roughly speaking, E (cid:104) d( x ik +1 , y ik +1 ) 2 − d( x ik , y ik ) 2 (cid:105) =
O ( δ i 3 / 2 ) .
More specifically:
[Equation]
Plugging in the definition of g ′ ε and g ′′ ε ,
[Equation]
where we use the assumption that ε ≤ 14 R 2 and ε < 1 .
On the other hand,
[Equation]
Note that r k > ε 1 / 3 implies that
[Equation]
where we use the fact that | f ′ ε | ≤ 1 .
We now bound
[Equation]
Consider three cases:
[Equation]
where we use Lemma 10 and the definition of L . 3.
r k > R : We use the fact that f ′′ ε ( r ) ≤ 0 for all r ≥ R ≥ ε .
Thus
[Equation]
Proof of Lemma 3.
Let E x be an orthonormal basis of T x (0) M , E y be an orthonormal basis of
T y (0) M , and let B x and B y denote two Brownian motions which may be coupled in a non-trivial way.
By definition of Φ in (A.4) and by Lemma 2, x ( t ) =
Φ( t ; x (0) , E x , β, B x ) and y ( t ) =
Φ( t ; y (0) , E y , β, B y ) , where equivalence is in the sense of distribution.
Lemma 12 almost gives us what we need.
However, because we assumed that β satisfies Assumption 2, the assumption that ∥ β ( x ) ∥ ≤ L β cannot possibly hold.
We thus need to approximate β by a sequence of increasingly non-Lipschitz functions.
Consider a fixed i .
Let s j be a sequence of increasing radius, such that s j → ∞ as j → ∞ .
Let β j denote the truncation of β to norm s j , i.e.
[Equation]
We verify that β j also satisfies Assumption 3 with the same L ′ β as β .
Consider some fixed j .
Let us now define the Euler Murayama discretization of x ( t ) and y ( t ) as
[Equation]
Where Φ is as defined in (A.3), and is a short-hand for the (interpolated) Euler Murayama sequence with stepsize δ i
=
T/
i , defined in (4.2) (equivalently (A.2)).
It is by definition that x ( t ) =
lim
i →∞ x i ( t ) (and similarly for y ( t ) and y i ( t ) ).
Furthermore, define, for all i, j ,
[Equation]
Note that the above definition implies a non-trivial coupling between ˜ x i,j ( t ) and x i ( t ) , via the shared
Brownian motion B x .
In words, ˜ x · ,j ( t ) denotes the exact Langevin SDE, but with drift given by β j (the truncated version of β ), and ˜ x i,j denotes the (interpolated) Euler Murayama discretization of ˜ x · ,j with stepsize δ i .
Let L 0 :=
max {∥ β ( x (0)) ∥ , ∥ β ( y (0))
∥} .
Let us define ˜ r
i,jk :=
d(˜ x i,jk , ˜ y
i,jk ) .
Let κ ( r ) be as defined in the statement of Lemma 12.
Using Assumption 2, we verify that
[Equation]
Let q
[Equation]
R be the parameters in Assumption
[Equation]
This implies that κ ( r ) ≤ q for all r ≤ R .
Let
L := q + L Ric / 2 and ε be as defined in Lemma
[Equation]
Let g ε be as defined in Definition 1 with parameters L and R .
Then
[Equation] [Equation]
where we define α ε := min (cid:110) m − L Ric / 2,
[Equation]
(cid:111)
[Equation]
−
[Equation]
LR 2 ) .
The first line follows from Lemma 12, the second line simply splits 1 { r > R} into two cases: 1 (cid:110) R <
˜
r
i,jk ≤ s j − L 0 L ′ β (cid:111) and 1 (cid:110) s j − L 0 L ′ β <
˜ r i,jk (cid:111) , and bounds κ ( r ) ≤ − m when r > R under Assumption 2.
The third inequality is by definition of α ε .
Applying (B.7) recursively for k = 0 ...
K
[Equation]
where K = T/
i
[Equation]
we get that
[Equation]
We now bound the second term of (B.8) more carefully:
[Equation]
The last line is because of the following: from Lemma 16, δ i P ( s j − L 0 L ′ β ≤ sup k ≤ K ˜ r i,jk ) 1 / 2 =
O (
L ′ β( s j − L 0 ) 2 ) =
O ( 1 s 2 j ) assuming j sufficiently large.
Also from Lemma 16,
[Equation]
Plugging (B.9) into (B.8), and recalling the definition of ˜ r , and the fact that d(˜ x iK , ˜ y iK ) :
=
d(˜ x i ( T ) , ˜ y i ( T )) ,
[Equation]
where O ( · ) hides T dependency as well.
First, by taking the limit of i to infinity (e.g. for each i , we see that for any j and any ε ,
[Equation]
Let us define ˜ x · ,j ( t ) as the almost sure limit of ˜ x i,j ( t ) , as i → ∞ , whose existence is shown in
Lemma 5 (similarly for ˜ y · ,j ( t ) ).
It follows that g ε (d(˜ x i,j ( T ) , ˜ y
i,j ( T ) 2 ) converges almost surely to g ε (d(˜ x · ,j ( T ) , ˜ y · ,j ( T )) 2 ) as i → ∞ .
By dominated convergence (Lemma 15 implies a single constant upper bounds E (cid:104)
d(˜ x i,j ( T ) ,
˜ y
i,j ( T ) 2 (cid:105) for all i ), E (cid:104) g ε (d(˜ x i,j ( T ) , ˜ y
i,j ( T )) 2 ) (cid:105) converges to
E (cid:104) g ε (d(˜ x · ,j ( T ) , ˜ y · ,j ( T )) 2 ) (cid:105) as i → ∞ .
Let Ω denote the set of all couplings between ˜ x · ,j ( t ) and ˜ y · ,j ( t ) .
Then
[Equation]
B.
[Equation]
where the first inequality uses the fact that ˜ x · ,j ( t ) (resp ˜ y · ,j ( t ) ) is the limit, as i → ∞ , of ˜ x i,j ( t ) (resp ˜
y
i,j ( t ) ).
From Lemma 16, we know that
[Equation]
where we use the fact that, by definition in (A.3), x i ( t ) are linear interpolations of x i ( k ) .
Next, notice that when sup t ∈ [0 ,T ] d( x ( t ) , x (0)) ≤ s j − L 0 L ′ β , x ( t ) =
˜ x · ,j ( t ) for all t ∈ [0 , T ] .
It thus follows that as s j → ∞ , E (cid:2) g ε (d(˜ x · ,j ( T ) , ˜ y · ,j ( T ))) (cid:3) converges to E [ g ε (d( x ( T ) , y ( T )))] almost surely.
Thus taking limit of (B.10) as j → ∞ , i.e. s j → ∞ ,
[Equation]
Finally, take the limit of ε → 0 .
Note that g ε ( r 2 ) → g 0 ( r 2 ) =
f ( r ) , where f is defined in Definition 2.
Note also that α ε → α as defined in the lemma statement.
Finally, the properties of f follows from
Lemma 10.
C Tail Bounds
In this section, we establish various probability and moment bounds for and (1.1), (1.2) and (2.1).
These bounds are used at many places in our proofs.
C.1 One-Step Distance Bounds C.1.1 Under Lipschitz Continuity
Lemma 13 (One-step distance evolution under Lipschitz Continuity) .
Let β be a vector field satisfying 3.
Let δ ∈ R + be a stepsize satisfying δ ≤ 1 16 L ′ β .
Let x 0 ∈ M be arbitrary, let x k denote the iterative process
[Equation]
where ξ k denote a random variable that possibly depends on x k .
Let γ k ( t ) :
[0 , 1] → M denote any minimizing geodesic with γ (0) =
x k and γ (1) = x 0 .
Then for any positive integer K and for any k ≤ K , we can bound,
[Equation]
Proof.
We will be using the bound from Zhang and Sra [2016]
(see Lemma 25).
Let v :=
δβ ( x k ) + √ δξ k ( x k ) .
Then Lemma 25 bounds
[Equation]
where ζ ( r ) :
= r tanh( r ) .
We will consider two cases:
[Equation]
From (C.1)
[Equation]
where the third inequality uses the definition of Case 1, and the fourth inequality is by several applications of Young’s Inequality.
[Equation]
[Equation]
Let us define z
[Equation]
I.e. z
[Equation]
t ) interpolates between x k and x k +1 .
We verify that z
[Equation]
Let us also define a family of geodesics γ t
[Equation]
where for each t , γ t is a minimizing geodesic with γ t (0) =
z ( t ) and γ t (1) = x 0 .
If such a minimizing geodesic is not unique, any choice will do.
We verify that
[Equation]
Let’s upper bound the terms one by one.
We first bound 2 , which represents the "discretization error in drift":
[Equation]
Γ z ( t ) z (0) ( δβ ( x k )) , γ ′ t (0) (cid:69) ≤ 2 (cid:13)(cid:13)(cid:13) δβ
( z ( t ))
− Γ z ( t ) z (0) ( δβ ( x k ))
(cid:13)(cid:13)(cid:13) d( z ( t ) , x 0 ) ≤ 2 δL ′
β d( z ( t ) , x k )d( z ( t ) , x 0 ) ≤ 2 δL ′
β d( z ( t ) , x k ) 2 + 2 δL ′
β d( z ( t ) , x 0 )
By definition of z ( t ) , we know that d( z ( t ) , x k ) ≤ (cid:13)(cid:13) (cid:13) δβ ( x k ) + √ δξ k ( x k ) (cid:13)(cid:13) (cid:13) ≤ δL
+ δL ′ β d( x k , x 0 ) + √ δ ∥ ξ k
( x k ) ∥ , so that 2 δL ′
β d( z ( t ) , x k ) 2 ≤ 8 δ 3 L ′ β 3 d( x k , x 0 ) 2 + 8 δ 2 L ′ β
∥ ξ k ( x k ) ∥ 2 + 8 δ 3 L ′ β
L 20 ≤ δL ′ β d( x k , x 0 ) 2 + δ ∥ ξ k
x k ) ∥
+ δ 2 L 20 , so that 2 ≤ δL ′
β d( z ( t ) , x k ) 2 + δL ′
β d( x k , x 0 ) 2 + δ ∥ ξ k ( x k ) ∥
+ δ 2 L
Next, we bound 3 , which is the most significant error term.
From the definition of Case 2, d( x k , x 0 ) >
[Equation]
where we use our assumption that δ ≤ 1 L ′ β .
Finally, we bound 1 as
[Equation]
Putting everything together, d
[Equation]
By Gronwall’s Lemma (integrating from t = 0 to t = 1 ), d( x k +1 , x 0 )=
d( z (1) , x 0 )
[Equation]
where we use the assumption that δ ≤ 18 L ′ β .
Combining Case 1 and Case 2:
[Equation]
C.1.2 Under Dissipativity
Lemma 14 (One-step distance evolution under Dissipativity) .
Let β be a vector field satisfying 3.
Let δ ∈ R + be a stepsize satisfying δ ≤ m
[Equation]
L ′ β 2 .
Let x ∗ be some point with β ( x ∗ )
= 0 .
Let x 0 ∈ M be arbitrary.
Let x k be the iterative process
[Equation]
Assume that for all x such that d( x, x ∗ ) ≥ R
[Equation]
there exists a minimizing geodesic γ : [0 , 1] → M with γ (0) =
x, γ (1) = x ∗
[Equation]
and
[Equation]
and let γ k denote such a geodesic for x = x k .
Then for any k ,
[Equation] [Equation]
Proof.
Throughout the proof, it is useful to note that by our assumptions, it must be that m ≤ L ′ β .
We will be using the bound from Zhang and Sra [2016]
(see Lemma 25).
Let v :=
δβ ( x k ) + √ δξ k ( x k ) .
Then Lemma 25 bounds
[Equation]
where ζ ( r ) :
= r tanh( r ) .
We will consider two cases:
Case 1: d
[Equation]
x
[Equation]
, x
[Equation]
) ≤
[Equation]
From (C.2): d 2 ( x k +1 , x ∗ )
[Equation]
where we use our assumptions that δ ≤ m/ (16 L ′ β 2 ) and the inequality under Case 1.
We used
Cauchy Schwarz a few times.
We can further bound
[Equation]
Thus
[Equation]
where we use the fact that e δm ≤ e m
[Equation]
Case 2: d( x k , x
[Equation]
>
[Equation]
Let us define
[Equation]
I.e. z ( t ) interpolates between x k and x k +1 .
We verify that z ′ ( t ) =
Γ z ( t )
z
[Equation]
( δβ ( x k ) + √ δξ k ( x k )) .
Let us also define a family of geodesics γ t , where for each t , γ t is a minimizing geodesic with γ t (0)
=
z ( t ) and γ t (1) =
x ∗ .
If such a minimizing geodesic is not unique, any choice will do.
We also verify that
[Equation]
Let’s upper bound the terms one by one.
We first bound 2 , which represents the "discretization error in drift":
[Equation]
By definition of z ( t ) , we know that d( z ( t ) , x k ) ≤ (cid:13)(cid:13)(cid:13) δβ ( x k ) + √ δξ k ( x k )
[Equation]
≤ δL ′ β d( x k , x ∗ ) + √ δ ∥ ξ k
x k ) ∥ , so that 4 δL
[Equation]
d
[Equation]
z
[Equation]
) , x
[Equation]
δ ∥ ξ k ( x k ) ∥ 2 , so that
[Equation]
Next, we bound 3 , which is the most significant error term.
From the definition of Case 2, d( x k , x ∗ ) >
[Equation] [Equation]
where we use our assumption that δ ≤ m 128 L
[Equation]
Finally
[Equation]
we bound 1 as
[Equation]
Putting everything together,
[Equation]
By Gronwall’s Lemma (integrating from t = 0 to t = 1 ), d
[Equation]
x k
[Equation]
, x
[Equation]
d
[Equation]
z (1) , x
[Equation] [Equation]
where we use the assumption that δ ≤ 1 128 m so that exp ( − 3 δm/ 2) ≤ 1 − 5 δm/ 4 .
Combining Case 1 and Case 2:
Combining (C.4) and (C.5),
[Equation]
C.2 L p Bounds
C.2.1 Under Lipschitz Continuity
Lemma 15 (L2 Bound and Chevyshev under Lipschitz Continuity) .
Consider the same setup as Lemma 13.
Assume in addition that there exists σ ξ ∈ R + such that for all x and for all k ,
E (cid:104) ∥
ξ k
[Equation]
x ) ∥
[Equation]
≤ σ 2 ξ .
Then for any positive integer K , and for all k ≤ K ,
[Equation]
and
[Equation]
Proof.
Let F k denote the σ -field generated by ξ 0 ...
ξ k − 1 .
To bound the first claim, take expectation of the bound from Lemma 13 wrt F k :
[Equation]
In line 3 above, γ k ( t ) is a minimizing geodesic from x k to x 0 , as defined in Lemma 13.
Applying the above recursively,
[Equation]
The above upper bound clearly also holds for E (cid:104) d( x k , x 0 ) 2 (cid:105) for all k ≤ K .
This proves our first claim.
To prove the second claim
[Equation]
let us define
[Equation]
We verify that r k as defined above is a sub-martingale.
Thus by Doob’s martingale inequality,
[Equation]
Furthermore, notice that
[Equation]
so that r k ≥ d( x k , x 0 ) with probability 1, for all k .
Thus
[Equation]
The proof for the bound on r 2 K is identical to the proof of the first claim.
We conclude our proof of the second claim
Lemma 16 (L4 Bound and Chevyshev under Lipschitz Continuity) .
Let β be a vector field satisfying 3.
Assume in addition that δ ∈ R + satisfies δ ≤ min (cid:110)
16 L ′ β ,
[Equation]
∥ β ( x
[Equation]
∥
[Equation]
Let x k be the following stochastic process:
[Equation]
Assume in addition that for all x and for all k , E (cid:104) ∥ ξ k ( x ) ∥ 4 (cid:105) ≤ 2 d 2 .
Then for any positive K ≥ 4 , and for all k ≤ K ,
[Equation]
and
[Equation]
Proof.
Let F k denote the σ -field generated by ξ 0 ...
ξ k − 1 .
We will use the following inequality from Lemma 13:
[Equation]
Squaring both sides,
[Equation]
where * has 0 -mean
[Equation]
and we used a few times Cauchy Schwarz and Young’s Inequality.
Taking expectation wrt F k ,
[Equation]
Applying the above recursively,
[Equation]
To prove the second claim, define
[Equation]
where * is the same term as (C.6).
We verify that r k as defined above is a sub-martingale.
Thus by Doob’s martingale inequality,
[Equation]
Furthermore, notice that
[Equation]
so that r k ≥ d( x k , x 0 ) with probability 1, for all k .
Thus
[Equation]
The proof for the bound on r 2 K is identical to the proof of the first claim.
We conclude our proof of the second claim
C.2.2 Under Dissipativity
Lemma 17 (L4 Bound under Dissipativity, Discretized SDE) .
Let β be a vector field satisfying 3.
Let x ∗ be some point with β ( x ∗ )
= 0 .
Assume that for all x such that d( x, x ∗ ) ≥ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0)
= x, γ (1) = x ∗ , and ⟨ β ( x ) , γ ′ (0) ⟩ ≤ − m d( x, x ∗ ).
Assume in addition that δ ∈ R + satisfies δ ≤ m 128 L ′
β
2 Let x k be the following stochastic process:
[Equation]
where ξ k is a random vector field satisfying E [ ξ k ( x )]
= 0 and E (cid:104)
∥
ξ k ∥ 4 (cid:105) ≤ ( σ ξ ) 4 .
For any k ,
[Equation]
Proof.
Let γ k denote a geodesic with γ k (0) =
x k , γ (1) =
x ∗ , and
[Equation]
From Lemma 14,
[Equation]
Squaring both sides and taking expectation wrt ξ k (and applying Young’s Inequality),
[Equation]
Applying the above recursively,
[Equation]
Lemma 18 (L4 Bound under Dissipativity, Exact SDE) .
Let β be a vector field satisfying 3.
Let x ∗ be some point with β ( x ∗ ) = 0 .
Assume that for all x such that d( x, x ∗ ) ≥ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0)
= x, γ (1) = x ∗ , and ⟨ β ( x ) , γ
[Equation]
Let x ( t ) denote the solution to (1.2) .
For any k ,
[Equation]
Proof.
For i ∈ Z + , let δ i be a sequence of stepsizes going to 0 , let K i :
= T/δ i , and let x ik be a discretization of x ( t ) with stepsize δ i of the form (1.2), i.e.
[Equation]
where ζ k ∼ N x i k (0 , I ) .
Applying Lemma 17 to x i k with ξ k ( x k ) =
ζ k and σ ξ = √ 2 d , and for i sufficiently large, gives the bound
E
[Equation]
Our conclusion follows by Lemma 2 as x iK i converges to x ( T ) almost surely as i → ∞ .
C.3 Subgaussian Bounds C.3.1 Under Dissipativity
Lemma 19 (Subgaussian Bound under Dissipativity, Discrete Time Semimartingale, Adaptive
Stepsize) .
Let β be a vector field satisfying 3.
Let x ∗ be some point with β ( x ∗ ) = 0 .
Assume that for all x such that d( x, x ∗ ) ≥ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0)
= x, γ (1) = x ∗ , and ⟨ β ( x ) , γ ′ (0) ⟩ ≤ − m d( x, x ∗ ) 2 .
Let x k be the following stochastic process:
[Equation]
where ξ k is a random vector field.
Assume that for all x , E [ ξ k ( x )] = 0 , and that for any ρ ≤ 18 ,
E (cid:104) exp ( ρ ∥ ξ k ( x ) ∥ 2 ) (cid:105) ≤ exp ( ρσ 2 ξ ) .
For each k , δ k is a positive stepsize that depends only on x k and satisfies δ k ≤ min
[Equation]
Assume that d( x 0 , x ∗ ) ≤ 2 R .
Finally, assume that there exists δ ∈ R + such that for all k , δ k ≤ δ .
Then
[Equation]
Proof.
For each k , let γ k be a minimizing geodesic with γ k (0) =
x k and γ k (1) =
x ∗ satisfying ⟨ β ( x k ) , γ ′ k (0) ⟩ ≤ − m d( x k , x ∗ ) 2 .
Using the result from Corollary 8 of Zhang and Sra [2016]
(see Lemma 25),
[Equation]
By Assumption 2 and Assumption 3, ⟨ β ( x k ) , γ ′ k (0) ⟩ ≤ − m d( x k , x ∗ ) 2 + 2 L ′ β R 2 .
Applying Cauchy
Schwarz and simplifying,
[Equation]
where we used our assumption that δ k ≤
[Equation]
Let s := m 32 σ 2 ξ We will now apply Lemma 24 with
[Equation]
We will verify Lemma 24’s condition regarding E (cid:2) exp ( √ δ k ν k ) (cid:3) .
Taking expectation conditioned on ξ 0 ( x 0 ) ...
ξ k − 1 ( x k − 1 ) ,
[Equation]
By our assumption on ξ k and Lemma
[Equation]
and our assumption that δ k ≤
[Equation]
On the other hand, by our assumption on ξ k and δ k ≤ σ
[Equation]
m
[Equation]
L R
[Equation]
Plugging both of the above into (C.8),
[Equation]
We thus verify that ( ν k , λ, µ ) satisfy the requirement for Lemma
[Equation]
which bounds
[Equation]
where the second inequality plugs in definitions from (C.7)
[Equation]
uses our assumption on x 0 .
Finally, using the fact that q k = s d( x k , x ∗ ) ,
[Equation]
Lemma 20 (Subgaussian Bound under Dissipativity, SGLD, fixed stepsize) .
Let β be a vector field satisfying Assumption 3.
Let x ∗ be some point with β ( x ∗ )
= 0 .
Assume that for all x such that d( x, x ∗ ) ≥ R , there exists a minimizing geodesic γ : [0 , 1] → M with γ (0)
= x, γ (1) = x ∗ , and ⟨ β ( x ) , γ ′ (0)
[Equation]
≤
[Equation]
m d( x, x ∗ ).
Let r ∈ R + denote an arbitrary radius, and assume that δ is a stepsize satisfying
[Equation]
Let x k be the following stochastic process:
[Equation]
where ζ k ( x k ) ∼ N x k
(0 , I ) and ˜ β k ( x ) satisfies, for all x , E (cid:104)
˜
β k ( x ) (cid:105)
=
β ( x ) and (cid:13)(cid:13)(cid:13) ˜
β k ( x ) − β ( x ) (cid:13)(cid:13)(cid:13) ≤ σ .
Assume that d( x 0 , x ∗ ) ≤ 2 R .
Then
[Equation]
Proof.
We begin by defining ξ k ( x k ) :
= ζ k ( x k ) + √ δ (˜ β ( x k ) − β ( x k )) .
We verify that
[Equation]
We verify that E [ ξ k ] = 0 and that for any λ ≤ 18 ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we use Lemma 21.
Let us define σ ξ := √ 4 d + σ 2 .
Next, let us define, for analysis purposes, the following process:
[Equation]
initialized at ˜ x 0 = x 0 and where
[Equation]
Define the event
A k := { max i ≤ k d(˜ x i , x ∗ ) ≤ r } .
Under the event A k , δ i = δ for all i ≤ k
[Equation]
and consequently, ˜ x i = x
i for all i ≤ k .
Therefore, A k = { max i ≤ k d( x i , x ∗ ) ≤ r }
[Equation]
and thus
[Equation]
where the last inequality follows from Lemma 19 with ξ k and σ ξ as defined above
[Equation]
C.4 Misc
Lemma 21.
For λ ≤ 14 and ξ ∼ N (0 , I d × d ) ,
[Equation]
Proof.
Consequence of χ 2 distribution being subexponential.
Lemma 22 (Hoeffding’s Lemma) .
Let η k be a 0 -mean random variable.
Then for all λ ,
[Equation]
Proof.
[Equation]
where ε is a Rademacher random variable.
Lemma 23 (Corollary of Doob’s maximal inequality) .
Let K be any positive integer.
For any k ≤ K , let a k , b k , c k , d k ∈ R + be arbitrary positive constants.
Assume that for all k , a k ≤ 14 and a k + c k ≤ 14 .
Let q k be a semi-martingale of the form
[Equation]
where η k are random variables.
Assume that for all k , η k satisfy
[Equation]
Assume in addition that q k ≥ 0 almost surely, for all k .
Finally, assume that (cid:80) Kk
=
a k + c k ≤ 18 .
Then
[Equation]
Proof.
Let us first define
[Equation]
i.e. r k is very similar to q k , only difference being that we replaced ≤ by = .
We first verify that for all k ≤ K , r k ≥ q k .
For k = 0 , by definition, r 0 = q 0 .
Now assume that r k ≥ q k for some k .
Then for k + 1 , r k +1 :=(
[Equation]
≥ q
[Equation]
We verify below that exp ( r k ) is a sub-martingale: conditioning on η 0 ...
η k − 1 , and taking expectation wrt η k ,
[Equation]
where the first inequality is by convexity of exp , and E [ η k ] = 0 , and Jensen’s inequality.
Let us now define s k :
=
(cid:81) k − 1 i =
(1 + a i
+ c i ) − 1 .
We can upper bound
[Equation]
where the second inequality is by Lemma 22, the third inequality is by the fact that s k ≤ 1 for all k and by Jensen’s inequality, the fourth inequality uses our assumption on η k in the Lemma statement, as well as the fact that s k ≤ 1 for all k .
The last equality is by definition of s k and because q k ≤ r k .
Applying this recursively gives
[Equation]
By Doob’s maximal inequality (recall that we e r k is a sub-martingale),
[Equation]
The first equality uses our assumption that s K = (cid:81) K − 1 k =0 (1 + a k + c k ) − 1 ≥ e − (cid:80) K − 1 k =0 4( a k + c k ) ≥
e − 14 ≥ 12 and the fact that r K ≥ q k ≥ 0 .
Lemma 24 (Uniform Bound) .
Let K be any positive integer.
For k ≤ K , let δ k ∈ R + , let λ, γ, µ ∈ R + .
Let q k be a sequence of random numbers of the form
[Equation]
where ν k are random variables.
Assume that q k and ν k are measurable wrt some filtration F k .
Assume that for all k , ν k satisfy
[Equation]
Assume that there is a constant δ such that for all k , δ k ≤ δ ≤ 18 λ .
Assume in addition that q k ≥almost surely, for all k .
Then
[Equation]
Proof.
For any s ≤ 1 ,
[Equation]
Applying the above recursively, for any k , we can bound
E
[Equation]
exp ( q k )] ≤ E [exp ((1 − δ k λ/ 2) q k − 1 + δ k
γ + µ ))] ≤
[Equation] [Equation]
Let us define t k := (cid:80) ki =
δ i .
By our assumption that δ i ≤ 14 λ , we can verify that
[Equation]
k
[Equation]
e
[Equation]
dt ≤ 4 λ .
Therefore, for all k ,
[Equation]
Let us now define N := (cid:6) 14 δλ (cid:7) ≥ 1 (inequality is because δ ≤ δ k ≤ 18 λ ).
We verify that q k +1 ≤ q k − 1 + δ k
( γ + µ ) + η k .
Let us now apply Lemma 23 with η k = √ δ k ν k , a k = 0 , b k = δ k γ, c k = λ/ 2 , d k = µ and the fact that δ k λ ≤ 1 / 4 to bound, for any k ,
P
[Equation]
max
[Equation]
t2 ) where we use the fact that N ≤ 12 δλ
Applying union bound over the events (cid:8) max i ≤ N q k
+ i ≥ t 2 (cid:9) for k = 0 , N, 2 N... , we can bound, for any positive integer M ,
[Equation]
Plugging in M = KN ≤ 8 Kδλ , it follows that for any K ,
[Equation]
Lemma 25.
Let M satisfy Assumption 4.
For any 3 points x, y, z ∈ M , let u, v ∈ T
y M be such that z
= Exp y ( v ) and x = Exp y ( u ) .
Assume in addition that ∥ u ∥ = d( x, y ) (i.e. t → Exp x ( tu ) is a minimizing geodesic).
Then
[Equation]
where ζ ( r ) :
= r tanh( r ) .
The above lemma is a restatement of Corollary 8 from Zhang and Sra [2016].
Although Zhang and
Sra [2016] required minimizizng geodesics to be unique, their proof, based on Lemma 6 of the same paper, works even if minimizing geodesics are not unique.
D Fundamental Manifold Results
In this section, we provide Taylor expansion style inequalities for the evolution of geodesics on manifold.
By making use of our bounds for matrix ODEs in Section E, we can bound the distance between two points along geodesics under various conditions.
Most of our analysis is based on some variant of the Jacobi equation
D 2 t J + R
( J, γ ′ )
γ ′ = 0 .
Notably,
Lemma 28 quantifies the distance evolution between x ( t ) =
Exp x ( tu ) and y ( t ) =
Exp y ( tv ) .
This is the key to proving Lemma 7, which bounds the distance between Euler Murayama discretization (1.2) and (1.1).
Lemma 29 is a more refined version of Lemma 28.
Lemma 29 is key to proving Lemma 8, which is in turn key to proving Theorem 2.
Lemma 29 is also used to analyze the distance evolution of two processes under the Kendall-
Cranston coupling in Lemma 3.
D.1 Jacobi Field Approximations
In the following lemma, we consider a variation field Λ( s, t ) , where for each s , Λ( s, t ) is a geodesic.
We bound the error between Λ( s, t ) , and its Taylor approximation of various orders.
This lemma is key to proving Lemma 28 and Lemma 28.
Lemma 26.
Let Λ( s, t ) : [0 , 1] ×
[0 , 1] → M be a variation field, where for each fixed s , t → Λ( s, t ) is a geodesic.
Let us define C :
= (cid:113) L R ∥
∂ t Λ( s, 0) ∥ 2 .
Then for all s, t ∈ [0 , 1] ,
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
s Λ( s
[Equation]
t
[Equation]
Λ( s
[Equation]
) Λ( s
[Equation]
s
[Equation] [Equation]
(cid:13)(cid:13)(cid:13) ≤ (cosh ( C ) − 1)
∥ ∂ s Λ( s, 0) ∥
+ sinh
[Equation]
C ) C
[Equation]
0) ∥
∥ D t ∂ s Λ( s, t ) ∥ ≤ C sinh ( C )
∥
∂ s Λ( s, 0) ∥
+ cosh ( C ) ∥
D t ∂ s Λ( s, 0) ∥ (cid:13)(cid:13)(cid:13)
[Equation]
If in addition, the derivative of the Riemannian curvature tensor is globally bounded by L ′ R , then (cid:13)(cid:13)(cid:13)
[Equation]
Proof.
For any fixed s , let E i ( s, 0) be a basis of T Λ( s, 0) M .
Let E i ( s, t ) denote an orthonormal frame along γ s ( t ) :
= Λ( s, t ) , by parallel transporting E
i
( s, 0) .
Let J ( s, t ) ∈
R
d denote the coordinates of ∂ s Λ( s, t ) wrt E
i ( s, t ) .
Let K ( s, t ) ∈
R
d denote the coordinates of D t ∂ s Λ( s, t ) wrt E
i ( s, t ) .
Let a ( s, t ) ∈
R
d denote the coordinates of ∂ t Λ( s, t ) wrt E
i ( s, t ) (this is constant for fixed s , for all t ).
Let R ( s, t ) ∈
R 4 d be such that
R ijkl =
⟨ R ( E j ( s, t ) ,
E k ( s, t ))
E l ( s, t ) , E i ( s, t ) ⟩ .
Let M ( s, t ) denote the matrix with
M i,j ( s, t ) :
= − (cid:80) k,l R ijkl ( s, t ) a k ( s, t ) a l ( s, t ) .
Notice that M i,j is symmetric, since M i,j
[Equation]
⟨ R ( E i , a ) a , E j ⟩ , where the first equality is by interchange symmetry and second equality is by skew symmetry of the Riemannian curvature tensor.
Therefore, by definition of L R in Assumption 4, it follows that
[Equation]
[Equation]
The Jacobi Equation states that D t D t ∂ s Λ( s, t ) =
− R ( ∂ s Λ( s, t ) , ∂ t Λ( s, t ))
∂ t Λ( s, t ) .
We verify that − ⟨ R ( ∂ s , ∂ t ) ∂ t , E i ⟩ = − (cid:80) j,k,l R ijkl J j a k a l =
[ M ( s, t ) J ( s, t )]
i , thus D t D t ∂ s Λ( s, t ) =
(cid:80) di
=
[ M ( s, t ) J ( s, t )]
i · E i ( s, t ) .
We verify that ddt J i ( s, t ) =
D t ⟨ ∂ s Λ( s, t ) , E i ( s, t ) ⟩ = ⟨ D t ∂ s Λ( s, t ) , E i ( s, t ) ⟩ =
K i ( s, t ) .
We also verify that ddt K i ( s, t ) =
D t ⟨ D t ∂ s Λ( s, t ) , E i ( s, t ) ⟩ = ⟨ D t D t ∂ s Λ( s, t ) , E i ( s, t ) ⟩ =
[ M ( s, t ) J ( s, t )] i .
Let us now consider a fixed s .
To simplify notation, we drop the s dependence.
The Jacobi Equation, in coordinate form, corresponds to the following second-order ODE:
[Equation]
Define L M :=
L R ∥
∂ t Λ( s, 0) ∥ 2 = C 2 .
We verify that L M ≥ max t ∈ [0 , 1] ∥ M ( t ) ∥ 2 .
Then from 34, we see that (cid:20) J ( t )
K ( t ) (cid:21) =
exp mat ( t ; M ) (cid:20) J (0)
K (0) (cid:21)
From Lemma 35, exp mat ( t ; (cid:20)
I
M ( t ) (cid:21) )
= (cid:20) A ( t )
B ( t )
C ( t )
D ( t ) (cid:21) where each block is R 2 d , and can be bounded as
[Equation]
where we use the fact that cosh( r ) , sinh( r ) and sinh( r ) r are monotonically increasing and that sinh( r ) r − 1 ≥ 0 for positive r .
It follows that
[Equation]
Thus
[Equation]
and
[Equation]
and
[Equation]
Similarly, ∥
[Equation]
and
[Equation]
To prove the last bound, let us define L ′ M :
=
L ′ R ∥
∂ t Λ( s, 0) ∥ 3 .
We verify that
L ′ M ≥ max t ∈ [0 , 1] ∥ M ( t ) − M (0) ∥ 2 .
we know that
[Equation]
where the last line follows from (D.1).
From our earlier results in this lemma,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the simplifications are from Lemma 39.
Put together,
[Equation]
Lemma 27.
Let x, y, z ∈ M , with x =
Exp z ( u ) , y =
Exp z ( v
[Equation] [Equation]
Proof.
Let us define the variational field
[Equation]
We verify that
[Equation]
Lemma 26 then immediately gives
[Equation]
D.2 Discrete Coupling Bounds
This section presents two key lemmas which play an important role in many of our proofs.
Lemma 28 analyzes the distance between Exp x ( u ) and Exp y ( v ) , as a function of x, y and u, v .
Notably,
Lemma 28 implies that when u, v are "parallel", i.e. u − Γ xy v = 0 , then the distance between d(Exp x ( u ) , Exp y ( v )) is not much larger than d( x, y ) .
The proof of Lemma 28 is based on a first-order expansion of the Jacobi equation for Riemannian manifolds.
The second key lemma is Lemma 29.
It considers a similar problem setup as Lemma 28, but is based on a second-order expansion of the Jacobi equation.
It thus requires an additional bound on the derivative of the Riemannian curvature tensor.
The more refined distance bound in Lemmais required to properly analyze the convergence of both continuous-time SDEs (Lemma 3) as well as discrete-time stochastic processes (Lemma 8).
The term (cid:82) 1 0 ⟨ R...γ ′ ( s ) ⟩ ds in the upper bound of
Lemma 29 gives rise to the Ricci curvature (as opposed to sectional curvature) dependencies in our results.
Lemma 28.
Let x, y ∈ M .
Let γ ( s ) :
[0 , 1] → M be a minimizing geodesic between x and y with γ (0)
=
x and γ (1) =
y .
Let u ∈ T x M and v ∈ T y M .
Let u ( s ) and v ( s ) be the parallel transport of u and v along γ , with u (0)
=
u and v (1) = v .
Then
[Equation]
where C := √ L R ( ∥ u ∥ + ∥ v ∥ ) .
Proof.
Let us consider the length function E ( γ )
= (cid:82) 1 0 ∥
γ ′ ( s ) ∥ 2 ds .
We define a variation of geodesics Λ( s, t ) :
[Equation]
We verify that
[Equation]
Consider a fixed t , and let γ t ( s ) :
= Λ( s, t ) (so γ ′ t ( s ) is the velocity wrt s ).
[Equation]
For any s , and for t = 0 , ∂ s Λ( s, 0) = γ ′ ( s ) and D t ∂ s Λ( s, 0)
= v ( s ) − u ( s ) .
Using the fact that norms and inner products are preserved under parallel transport, the first term can be simplified as
[Equation]
To bound the second and third term
[Equation]
we use Lemma 26:
[Equation]
∥ and
[Equation] [Equation]
≤
[Equation]
where we use the fact that (cid:113) L R ∥
∂ t Λ( s, 0) ∥ 2 ≤ C .
We can thus bound the second term of (D.2) as
[Equation]
Finally, to bound the third term of (D.2)
[Equation]
we again apply Lemma 26:
[Equation]
and
[Equation]
cosh ( C ) ∥ v (0) − u (0) ∥ for 0 ≤ t ≤ 1 , where we usse the fact that cosh( r ) and sinh( r ) r are monotonically increasing in r .
Put together, the third term of (D.2) is bounded as
[Equation]
Put together, we get
[Equation]
Integrating for t = ∈ [0 , 1] , and noting that E ( γ 0 )
=
∥ γ ′ (0) ∥ ,
[Equation]
From Lemma 39, we can upper bound
[Equation]
where we use the fact that r 2 ≤ e 2 r / 6 for all r ≥ 0 .
The conclusion follows by noting that d( x, y )
= (cid:112) E ( γ 0 ) and d(Exp x ( u ) , Exp y ( v )) ≤ (cid:112) E ( γ 1 ) .
Lemma 29.
Let x, y ∈ M .
Let γ ( s ) :
[0 , 1] → M be a minimizing geodesic between x and y with γ (0)
=
x and γ (1) =
y .
Let u ∈ T x M and v ∈ T y M .
Let u ( s ) and v ( s ) be the parallel transport of u and v along γ .
Let u
=
u 1 + u 2 and v = v
+ v 2 be a decomposition such that v 2 =
Γ yx u 2 , where the parallel transport is along γ ( s ) .
Let us define u 1 ( s ) , u 2 ( s ) , v 1 ( s ) , all mapping from [0 , 1] → T γ ( s ) M , such that they are the parallel transport of u 1 , u 2 , v 1 along γ ( s ) respectively ( u 1 (0) =
u 1 , u 2 (0)
=
u 2 , v 1 (1) = v 1 , u 2 (1) = v 2 )
Then
[Equation]
where C := √ L R ( ∥ u ∥ + ∥ v ∥ ) and C ′ :
=
L ′ R ( ∥ u ∥ + ∥ v ∥ ) 3 .
Proof.
The proof is similar to Lemma 28.
Let us consider the length function E ( γ )
= (cid:82) 1 0 ∥
γ ′ ( s ) ∥ 2 ds .
We define a variation of geodesics Λ( s, t ) :
[Equation]
We verify that
[Equation]
Consider a fixed t , and let γ t ( s ) :
= Λ( s, t ) (so γ ′ t ( s ) is the velocity wrt s ).
[Equation]
and d
[Equation]
where the second equality uses the Jacobi equation.
The Riemannian curvature tensor term can be simplified as
[Equation]
We further bound
[Equation]
where we use Lemma 26 and Lemma 39.
We also bound
[Equation]
[Equation] [Equation]
) −
[Equation]
ds
[Equation]
≤ 16 C
[Equation] [Equation]
+
[Equation]
u
[Equation] [Equation]
−
[Equation]
0) ∥where we use Lemma 26 and Lemma 39.
We finally bound (cid:90)
[Equation]
by definition of C ′ .
Combining the above bounds,
[Equation]
Our conclusion follows as d(Exp x ( u ) , Exp y ( v )) 2 ≤ E ( γ 1 ) .
Lemma 30.
Let a t , b t : t → R + satisfy
[Equation]
with initial conditions a 0 , b 0 , then for all t ,
[Equation]
Proof.
Let x t :=
[Equation]
0 cosh
[Equation]
√ Ct ) + b 0 √ C sinh ( √ Ct ) and y t := √
C
[Equation]
sinh
[Equation]
√ Ct ) + b 0 √ C cosh ( √ Ct )) .
We verify that
[Equation]
We further verify the initial conditions.
Note that sinh(0)
=
and cosh (0)
= 1 .
Thus
[Equation]
Finally, we verify that a t ≤ x t and b t ≤ y t for all t :
[Equation]
Lemma 31.
Let a t , b t : t → R + satisfy
[Equation]
with initial conditions a 0 = 0 , b 0 = 0 , then for all t ,
[Equation]
Proof.
Let x t := EC cosh( √ Ct ) + DC 3 / 2 sinh ( √ Ct ) − DC t − EC and y t :=
E √ C sinh( √ Ct ) + DC cosh ( √ Ct ) − DC .
We verify that
[Equation]
we also verify the initial conditions that x 0 = 0 and y 0 = 0 .
Lemma 32.
Let x, y ∈ M , and let E 1 ...
E d be an orthonormal basis at T x M .
Let v ∈
T x M be a random vector with E [ ⟨ E i , v ⟩ ⟨ E j , v ⟩ ]
= 1 { i = j } .
Let γ : [0 , 1] → M be any smooth path between x and y .
Let v ( t ) be the parallel transport of v along γ .
Then for any basis E ′ 1 ...E ′ d at T y M ,
[Equation]
In other words, if v has identity covariance, then the parallel transport of v has identity covariance.
Proof.
Let E i ( t ) be an orthonormal frame along γ with E i (0) =
E i .
Under parallel transport, ddt ⟨ v ( t ) , E i ( t ) ⟩ = 0 .
Thus for all t ,
[Equation]
1 { i = j }
Finally, consider any basis E
i .
Let E ′ i
= (cid:80) k
α ik E i (1) , i.e. α ik = ⟨ E
′ i , E k (1) ⟩
Then
[Equation]
Lemma 33.
Let x, y ∈ M , and let E 1 ...
E d be an orthonormal basis at T x M .
Let α denote a spherically symmetric random variable in R d , i.e. for any orthogonal matrix
G ∈ R d × d α d = Gα
Then for any x ∈ M , let E 1 ...E d and E ′ 1 ...
E ′
d be two sets of orthonormal bases of T x M .
then d (cid:88)
[Equation]
Consequently, let v ∈ T x M := (cid:80) di
=
α i E i .
Let γ : [0 , 1] → M be any differentiable path between x and y .
Let v ( t ) be the parallel transport of v along γ .
Then for any orthogonal basis E ′ 1 ...E ′ d at
T y M ,
[Equation]
Proof.
First, we verify that if α is spherically symmetric, and E 1 ..E d , E ′ 1 ...
E ′ d are two sets of orthonormal basis at some point x , then
[Equation]
To see this, notice that there exists an orthogonal matrix G , with G i,j = (cid:10) E i , E ′ j (cid:11) , such that
[Equation]
We further verify that G i,j is orthogonal.
It suffices to verify that GG T
= I .
[Equation]
Note that the inner product on the last line is dot product over R d , and the inner product on preceding lines are over T x M .
The above implies that
[Equation]
i.e. G is orthogonal.
Now consider any arbitrary function f : T x M → R , then
[Equation]
  where we defined β j := (cid:80) i
α i G i,j .
We finally verify that β d = α .
This follows from the fact that β = G T α , where G is an orthogonal matrix, and the definition of spherical symmetry for α .
Consider an arbitrary line γ ( t ) :
[0 , 1] → M , with x := γ (0) , y :
= γ (1) .
Let E i be an orthonormal basis at T x M , and E i ( t ) be an orthonormal basis at T γ ( t ) M obtained from parallel transport of E i .
This proves the first claim.
To verify the second claim, let v ∈
T x M be a random vector, given by
[Equation]
where α is some spherically random vector in R d .
Let v ( t ) be the parallel transport of v along γ .
Let α ( t ) :
= ⟨ v ( t ) , E i ( t ) ⟩ .
Then by definition of parallel transport, for all i ,
[Equation]
so that for all t ∈ [0 , 1] , α ( t ) := α the second claim then follows from the first claim.
E Matrix ODE
In this section we provide Gronwall-style inequality for matrix ODE.
The results in this section are necessary for analyzing Jacobi Equation, whose coordinates with respect to some orthonormal frame can be viewed as an ODE in R d .
In particular, Lemma 28 and Lemma 29 rely on results in this section.
Lemma 34 (Formal Matrix Exponent) .
Given M ( t ) : R + → R d × d , define exp mat ( t ; M ) : R + →
R d × d as the solution to the matrix ODE
[Equation]
Then 1.
Let x ( t ) be the solution to the ODE ddt x ( t ) =
M ( t ) x ( t ) , for some M , then
[Equation]
Let z ( t ) be the solution to ddt z ( t ) =
M ( t ) z ( t ) + v ( t ) , for some M , v , then
[Equation]
where for any s, t , N s ( t ) :
=
M ( s + t ) .
Proof of Lemma 34.
Let y t :=
exp mat ( t ; M ) x (0) .
We verify that
[Equation]
Given the same dynamics and initial conditions, we conclude that x ( t ) =
y ( t ) for all t .
To verify the second claim, note that
[Equation]
Additionally, ddt exp mat
[Equation]
t ; M )
z
[Equation]
=
M ( t ) exp mat ( t ; M ) z (0) , summing,
[Equation]
Lemma 35.
Let exp mat be as defined in Lemma 34.
Let
[Equation]
A
[Equation]
for some M ( t ) .
Assume ∥
M ( t ) ∥ 2 ≤ L M for all t .
Then for all t ,
[Equation]
and
[Equation]
M
Proof of Lemma 35.
We first verify the first part of the lemma.
Consider the ODE given by d
[Equation]
I
[Equation]
( t )
[Equation]
(cid:20) x ( t ) y
( t ) (cid:21)
By Lemma 34,
[Equation]
t
[Equation]
satisfies
[Equation]
t
[Equation]
t
[Equation]
t
[Equation]
( t ) (cid:21) (cid:20)
[Equation]
[Equation]
y (0) (cid:21)
By Cauchy Schwarz,
[Equation]
We apply Lemma
[Equation]
with a t := ∥ x ( t ) ∥ 2 and b t := ∥
y ( t ) ∥ 2 , C :=
L M .
Then
[Equation]
This immediately implies that
[Equation]
This proves the first claim of the Lemma.
We now prove the second claim.
We verify that
[Equation]
Thus
[Equation]
We verify that
[Equation]
Let us apply Lemma 31 with a t =
∥ x t − x (0) − t y (0) ∥ 2 , b t = ∥
y ( t ) − y (0) ∥ 2 , C =
L M , D =
L M ∥
y (0) ∥ 2 and E =
L M ∥ x (0) ∥∥
[Equation]
1) (E.1)
Again from Lemma 34, we know that
[Equation]
thus
[Equation]
combined with (E.1), and using the fact that the above hold for all y (0) and x (0) , we can bound
[Equation]
Finally, to prove the third claim,
[Equation]
Lemma 36.
Let exp mat be as defined in Lemma 34.
Let (cid:20)
[Equation]
t
[Equation]
:= exp mat ( t
[Equation]
for some M ( t ) .
Assume ∥
M ( t ) ∥ 2 ≤ L M for all t .
Then for all t ,
[Equation]
L M t )
Proof.
The proof is similar to Lemma 35.
Consider the ODE given by d dt (cid:20) xy (cid:21) ( t ) =
(cid:20)
I
M ( t ) (cid:21) (cid:20) x ( t ) y ( t ) (cid:21) with initial condition y (0) = 0 .
By Lemma 34, (cid:20) A ( t )
B ( t )
C ( t )
D ( t ) (cid:21) satisfies (cid:20) x
( t ) y ( t ) (cid:21) =
(cid:20) A ( t )
B ( t )
C ( t )
D ( t ) (cid:21) (cid:20) x (0) y (0) (cid:21) d dt (cid:20) x ( t ) − x (0) − t 2 2 M (0) x (0) y ( t ) − t M (0) x (0) (cid:21) =
(cid:20) y
( t ) − t M (0) x (0)
M ( t ) x ( t ) − M (0) x (0) (cid:21) =
(cid:20) y
( t ) − t M (0) x (0)
M ( t ) x ( t ) − M (0) x (0) (cid:21) =
(cid:20) y
( t ) − t M (0) x (0)
M ( t )( x ( t ) − x (0) − t 2 2 M (0)) (cid:21) + (cid:20) ( M ( t ) − M (0))
x (0) (cid:21) + (cid:20) t 2 2 M ( t ) M (0) x (0) (cid:21)
By Cauchy Schwarz, for all t ≤ 1 , d dt (cid:13) (cid:13)(cid:13)(cid:13) x
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Apply Lemma 31 with a t = (cid:13)(cid:13)(cid:13) x
( t ) − x (0) − t 2 2 M (0) x (0) (cid:13)(cid:13)(cid:13) 2 , b t = ∥
y ( t ) − t M (0) x (0) ∥ 2 , C =
L M , D = 0 and E =
( L ′ M + 12 L 2 M )
∥ x (0) ∥ 2 to get
[Equation] [Equation]
Finally, recall that
[Equation]
Since we have shown that ∥ y ( t ) − t M (0) x (0) ∥ 2 ≤ ( L ′ M + 12 L 2 M ) ∥ x (0) ∥
2 √ L M sinh( √ L M t ) for all x (0) , it follows that
[Equation]
F Miscellaneous Lemmas
Lemma 37.
Let c ∈ R + be such that c ≥ 3 .
For any x satisfying x ≥ 3 c log c , we have that x log x ≥ c
The following Lemma is taken from Sun et al.
Lemma 38.
For any x ∈ M , a, y ∈ T x M d
[Equation]
Proof.
From the proof of Lemma 3 from Sun et al.
[2019] (which is in turn a refinement of the proof from Karcher [1977])
[Equation]
where we use the fact from Lemma 39 that for all r ≥ 0 ,
[Equation]
Lemma 39.
For all r ≥ 0 ,
[Equation]
Proof.
Elementary computation from power series.