Sufficient conditions for non-asymptotic convergence of Riemannian optimization methods
Abstract
Motivated by energy based analyses for descent methods in the Euclidean setting, we investigate a generalisation of such analyses for descent methods over Riemannian manifolds.
In doing so, we find that it is possible to derive curvature-free guarantees for such descent methods. This also enables us to give the first known guarantees for a Riemannian cubic-regularised Newton algorithm over g -convex functions, which extends the guarantees by Agarwal et al. [1] for an adaptive Riemannian cubic-regularised Newton algorithm over general non-convex functions.
This analysis motivates us to study acceleration of Riemannian gradient descent in the g -convex setting, and we improve on an existing result by Alimisis et al.
[4], albeit with a curvature-dependent rate.
Finally, extending the analysis by Ahn and Sra [2], we attempt to provide some sufficient conditions for the acceleration of Riemannian descent methods in the strongly geodesically convex setting.
1. Introduction
In this paper, we are interested in the task of minimizing a function f defined over a Riemannian manifold M .
This is an interesting problem, since certain f functions that are non-convex in the
Euclidean sense have been shown to be convex in a Riemannian sense over a specific Riemannian manifold.
We refer to this notion as geodesic convexity or g -convexity and formally define this later in this paper.
This therefore motivates the study of optimisation methods over Riemannian manifolds where considerable progress has been recently made in understanding such methods and proposing better alternatives.
Zhang and Sra [16] identified that a modified triangle equality was sufficient to obtain non-asymptotic guarantees for Riemannian gradient and subgradients methods.
This triangle inequality also underscored the study of an accelerated Riemannian gradient descent algorithm [17], which also used the idea of estimate sequences [13] to achieve (local) acceleration within a ball around the minimizer of f for strongly g -convex functions.
While this study focused on guarantees for strongly g -convex functions, a recent paper by Alimisis et al.
[4] investigated acceleration of first order methods over bounded domains for a broader class of functions which include g -convex functions, and proposed an algorithm which is shown to have strictly better rate than Riemannian gradient descent, but did not achieve global acceleration.
Their analysis was motivated by a previous study on continuous-time flows to help model acceleration over Riemannian manifolds [3].
Complementary to these attempts, Hamilton and Moitra [8] and Criscitiello and Boumal [7] show that global acceleration may not be achievable over negatively curved manifolds.
However, recent papers by Mart´ınez-Rubio [10] and Kim and Yang [9] show that we can obtain global acceleration inside a bounded subset of the manifold, and the rates of convergence are affected by the size of this subset.
While the focus of the non-exhaustive review is first-order methods, second-order methods have also been proposed over Riemannian manifolds, and we refer to Boumal [6, Chapter 6] for a detailed introduction to such methods.
1.1. Background
In this subsection, we introduce key definitions and terminology necessary for this work.
A Riemannian manifold M is a smooth manifold equipped with a Riemannian metric that defines an inner product between two vectors v, w in the tangent space T x M of x for every x ∈ M .
This induces a norm given by ∥ v ∥ x = ⟨ v, v ⟩ x for all v ∈ T x M .
The gradient of a differentiable function f at a point x is a vector in T x M satisfying ⟨ grad f ( x ) , v ⟩ x = lim t
→ 0 f ( c ( t )) − f ( x ) t for curve c : [0 , 1] → M satisfying c (0) =
x and c ′ (0)
= v for every v ∈ M .
A geodesic between two points x and y on the manifold is a locally length minimizing curve starting at x and ending at y , and the distance d ( x, y ) between x and y is given by the length of this geodesic.
A subset A of M is a geodesically unique set if for any two points in A , there exists a unique geodesic connecting them.
The exponential map at a point x on the manifold maps a tangent vector v ∈ T x M to a point on the manifold Exp x ( v ) such that γ x,v (1) =
Exp x ( v ) where γ x,v : [0 , 1] → M is a geodesic satisfying γ x,v (0) =
x and γ ′ x,v (0)
= v .
The inverse of the exponential map if it exists is called the logarithmic map ( Log x : y
(cid:55)→ v ), which computes the velocity of the geodesic starting from x to reach a point y .
The exponential and logarithmic maps can be viewed as manifold analogoues to vector addition and subtraction respectively.
A µ -strongly g -convex function f : A → R satisfies for all x, y ∈ A , f ( y ) ≥ f ( x ) + ⟨ grad f
( x ) , Log x ( y ) ⟩ x + µ / 2 ∥
Log y ( x ) ∥ 2 x .
When µ = 0 , we refer to such a function as being simply g -convex.
Similarly, a function f : A → R is said to L - g -smooth when for all x, y ∈ A , f ( y ) ≤ f ( x ) + ⟨ grad f
( x ) , Log x ( y ) ⟩ x + L / 2 ∥
Log x ( y ) ∥ 2 x .
We specify other equivalent definitions of these classes of functions in Appendix A.
2. An energy based analysis of Riemannian descent methods
To study Riemannian descent methods, we first introduce an abstraction that will allow us to study a collection of such algorithms in a unified manner.
This abstraction is equivalent to 1 -descent algorithms of order p proposed in [14] in the Euclidean case.
Definition 1 ( p -descent algorithm)
An iterative algorithm A is a p -forward descent / p -backward descent algorithm in M w.r.t.
function f if the sequence of iterates { x k } k ≥ 1 satisfies
[Equation]
forward descent
[Equation]
1a
[Equation] 1b [Equation] where c is a constant independent of k and x 0 ∈ M is the initialisation given to A .
For such descent algorithms, we show that it is possible to obtain rates of convergence to the minimizer x ⋆ of f where f is g -convex function, analogous to the Euclidean setting.
This is possible since the descent property deals with vectors in the tangent space of a point on the manifold, and the tangent space is isomorphic to a Euclidean space.
Before stating this theorem, we introduce some assumptions, as made in [17] and [4] that we use in the proof
( A 1 ) A is a geodesically unique convex subset of M with bounded diameter where Exp and Log are well-defined.
( A 2 ) x ⋆ is a minimum of f , which lies inside A .
( A 3 ) All the iterates of the algorithm stay within A .
When M is a Hadamard manifold, an example of A satisfying the assumptions is a sublevel set of f with respect to the initialisation.
Hyperbolic spaces and many matrix manifolds are examples of practically relevant Hadamard manifolds.
Other examples of A when M is not a Hadamard manifold include a subset of a sphere of radius R whose diameter is strictly less than πR .
Theorem 2 (Rate for p -descent algorithms over g -convex functions)
Let f be a g -convex function, and let x 0 be the initialisation which belongs in A ⊆ M .
If { x k } are the iterates of a p -descent algorithm (forward (Eq. 1a) or backward (Eq. 1b)) then, assuming ( A 1 ), ( A 2 ) and ( A 3 ), they satisfy the following guarantee
[Equation]
where C p is a constant dependent on p .
While not as involved, it is possible to obtain rates when f is non-convex or when it is gradient dominated.
We include these results in the Appendix B, and the proof of Theorem 2 and the proofs for rates in the non-convex and gradient dominated settings in Appendix C.
Examples of Riemannian p -descent algorithms and their rates.
Theorem 2 allows us to immediately infer rates of convergence for popular Riemannian methods when used to optimize g -convex functions.
We give instances of such methods below, and include complete proofs of these propositions in Appendix C.1.
The Riemannian gradient descent algorithm which generates a sequence of iterates according to the recursion x k +
=
Exp x ( − η grad f ( x )) for k ≥ 0 , is a 2 -backward descent method when f is L -g-smooth and 0 < η < 2 / L . 2.
The Riemannian proximal descent algorithm which generates a sequence of iterates according to the recursion x k +1 = argmin
y ∈M f
( y ) + 12 η d
y, x k ) for k ≥ 0 , is a 2 -forward descent method for any η > 0 .
The Riemannian cubic-regularized Newton algorithm which generates a sequence of iterates according to the recursion x k +
=
Exp x k ( s k ) where s k satisfies
[Equation]
with m k ( s )
:= f
x k )
+
⟨ s , grad f x k ) ⟩ x
k +
12 ⟨ s
Hess f ( x k )
[ s ] x k
+ M 3 ∥ s ∥ 3 x k is 3 -forward descent algorithm when f has ρ -Lipschitz continuous Hessians and M > ρ / 2 .
Remark
The rates of convergence that we attain for p -descent methods are curvature independent , improving on the popular result for Riemannian gradient descent in [16] at the cost of some additional assumptions, and also matches the curvature independent guarantees stated in [5] for Riemannian gradient descent and proximal descent algorithms over non-negatively curved manifolds.
Remark
This theorem allows us to give the first known rates for a cubic-regularized Newton type algorithm over Riemannian manifolds for g -convex functions.
The algorithm is a simpler version of the practical algorithm by Agarwal et al.
[1], which has guarantees in the non-convex setting.
3. Accelerating descent methods for g -convex and strongly g -convex functions
The energy-based analysis provides an effective way to analyse accelerated versions of descent methods, as studied by Wilson et al.
[15] in the Euclidean case.
In this section, we study the acceleration of the simplest p -forward / backward descent algorithms – which is when p is 2 – through a Nesterov-style scheme.
The algorithm is composed of three updates defined below.
[Equation]
with y
= z 0 ∈
A ⊆ M .
τ k ∈ (0 , 1) , α k , β k > 0 for all k ≥ 0 .
G c is a mapping which ensures that for all x , f ( G c ( x )) − f ( x ) ≤ − c
∥ grad f ( x ) ∥ 2 x .
To proceed with the energy based analysis, we first define the energy function.
We use a combination of the function optimality gap and the variant of the distance to the optimum, formally defined below. [Equation]
[Equation]
This choice of the energy function was previously used by Ahn and Sra [2] to study acceleration of
Riemannian gradient descent for strongly g -convex functions.
Since the distance term of the energy is explicitly dependent on x k , the analysis is not straightforward, as we cannot directly compare ∥ Log x k ( y ) ∥ x k and ∥
Log x k +1 ( y )
∥ x k +1 for y in general.
To aid us in proceeding with the analysis, we use the notion of a valid distortion rate which was originally proposed by Ahn and Sra [2].
Definition 3 (Valid Distortion Rate Ahn and Sra [2, Definition 3.2])
δ k is a valid distortion rate at iteration k if ∥ Log x k ( z k − 1 ) − Log x k ( x ⋆ )
∥ 2 x k ≤ δ k
∥
Log x k − 1 ( z k − 1 ) − Log x k − 1 ( x ⋆ ) ∥ 2 x k − 1 .
Ahn and Sra [2] provide computable forms of δ k based on the iterates x k and z k at each iteration k for Hadamard and non-Hadamard manifolds.
Assuming the existence of such valid distortion rates, we can use Equation 3 with specific setting of parameters
A k , B k , and show that the accelerated method in Equation 2 with algorithmic parameters τ k , α k , β k has better rate guarantees than a standard 2 -forward descent / 2 -backward descent algorithm.
Theorem 4 (Guarantees for g -convex functions) Let { y k } be the sequence of y iterates generated by the algorithm described in Equation 2 when given a g -convex function f , an initialisation y
= z 0 ∈
A ⊆ M and parameters
[Equation]
Under ( A 1 ), ( A 2 ) and ( A 3 ) and assuming the existence of a valid distortion rate at every iteration k ≥ 0 , this sequence satisfies
[Equation]
for all k ≥ 0 .
Remark
Recently, Alimisis et al.
[4] proposed a slightly different version of the algorithm stated in Equation 2 involving a geodesic search step which incurred a search error.
In contrast, the algorithm in Equation 2 does not involve such a search step, and as a result, the rate guarantee derived in Theorem 4 is free of a search error.
When k ≤
[Equation]
the
[Equation]
dominates the
[Equation]
term.
Drawing from the interpretation in [4], this can be viewed as the number of steps until which we obtain an “accelerated” rate.
When δ max = 1 (for e.g., when M is Euclidean), this upper bound is ∞ , and recovers the O ( 1 / k 2 ) rate shown by Nesterov [12].
When δ max → ∞ , then we achieve the same rate as a 2 -forward / backward descent algorithm.
Due to this, this algorithm achieves a strictly better rate than a standard 2 -forward / backward algorithm as given by our Theorem 2.
While the above analysis was for (weakly) g -convex functions, we can also show that 2 -backward descent algorithms can be accelerated using the same algorithm in Equation 2 with a different set of algorithmic parameters.
This is direct consequence of Ahn and Sra [2, Theorem 3.1], which was restricted to G c ( · ) being a gradient step.
Proposition 5 (Guarantees for µ -strongly g -convex functions) Let { y k } be the sequence of y iterates generated by the algorithm described in Equation 2 when given a µ -strongly g -convex function f , an initialisation y
= z 0 ∈
A ⊆ M and parameters
[Equation]
where ξ k +1 is the solution to the equation ξ k +1 (
[Equation]
[Equation] [Equation]
in [2 µc, 1) with A 0 , B 0 , ξ 0 >and c < 1 / 2 µ .
Under ( A 1 ), ( A 2 ) and ( A 3 ) and assuming the existence of a valid distortion rate at each iteration k ≥ 0 , this sequence satisfies
[Equation]
for all k ≥ 0 .
Remark
Note that the rate of convergence directly depends on values taken by ξ j , which in turn depends on the variation of the sequence of distortion rates { δ j } .
Let δ max = max t ≤ k δ k .
When δ max = 1 (for e.g., when M is Euclidean), Ahn and Sra [2] show in their Lemma 2.1 that the sequence { ξ k } k ≥ 0 converges to √ 2 µc .
Thus choosing ξ 0 ≥ √ 2 µc , the sequence { ξ k } k ≥ 0 converges to √ 2 µc and ξ k ≥ √ 2 µc for all k , giving us the rate O (exp( −√ 2 µc · k )) .
Since µ -strongly gconvexity corresponds to ((2 µ ) − 1 , 2) gradient domination (Definition 9), we can compare this rate to the rate for 2 -backward descent algorithms over ((2 µ ) − 1 , 2) -gradient dominated functions, which is O (exp( − 2 µc · k )) .
On the other extreme, when δ max → ∞ , then ξ k → 2 µc , giving us the rate
O (exp( − 2 µc · k )) .
As noted for the g -convex case, these guarantees are better than one would expect from a non-accelerated version, which was noted in [2] but for a gradient descent step.
3.1. Some sufficient conditions for eventual full acceleration of 2 -backward descent methods over µ -strongly g -convex functions
As noted earlier, there exists a computable sequence of valid distortion rates { δ k +1 } dependent on the iterates { ( x k , z k ) } generated by the algorithm in Equation 2.
More precisely, for Hadamard manifolds with sectional curvature lower bounded by − κ < 0 , the valid distortion rate at the k th iteration is given by δ k +
=
T κ ( d ( x k , z k )) where T κ (0)
= 1 .
Therefore, it would be instructive to analyse the rate at which the sequence { d ( x k , z k ) } converges to 0 , and translate that analysis to a rate at which the sequence { ξ k } converges to √ 2 µc .
This is the technique adopted in [2] for their analysis.
In this subsection, we extend their analysis to 2 -backward descent methods.
We begin by giving the following lemma, which is a generalisation of Lemma 4.2 in [2].
Lemma 6 Let M be a Hadamard manifold and { ( x k , y k , z k ) } be the sequence of iterates obtained from Algorithm 2 with parameter settings stated in Proposition 5, where the descent constant c of G c satisfies c < min { 1 / 6 L , 1 / 2 µ } given that f has L -Lipschitz gradients.
Define D 0 = f
( y 0 )
− f ( x ⋆ ) + ξ 20 4 c ∥ Log z 0 ( x ⋆ ) ∥ 2 z 0 .
If ξ 0 ∈ (2 µc, √ 2 µc ] and the iterates satisfy d ( x k +1 , y k +1 ) ≤
C ′ L,µ,c (cid:113)(cid:81) kj =1 (1 − ξ j ) · D 0 for every k ≥ 0 , then d ( x k +1 , z k +1 ) ≤ C L,µ,c (cid:113)(cid:81) kj =1 (1 − ξ j ) · D 0 for every k ≥ 0 as well, where C L,µ,c and C ′ L,µ,c are constants only depending on L, µ, c .
Remark
The above lemma states that with any 2 -backward descent method that descends sufficiently and causes the sequence of distances { d ( x k , y k ) } to decrease at a geometric rate, then the sequence of distances { d ( x k , z k ) } decreases at the same rate.
The original analysis by Ahn and Sra [2] provides such a result when G c is a gradient descent update, along with an interesting requirement that the step size be strictly greater than 1 / L .
Recall that for a gradient update, c
= c ( γ ) :
= γ (1 − Lγ / 2 ) and argmax γ c ( γ ) = 1 / L .
Our lemma states that a small enough descent is sufficient for a similar geometric convergence property.
With the above lemma, we can provide a general convergence result due to a careful analysis of the evolution of the sequence { ξ k } by Ahn and Sra [2].
Proposition 7 (Eventual acceleration of the Algorithm in Eq. 2) Let ( x k , y k ) be the ( x, y ) iterates generated by the algorithm in Equation 2 when given a µ -strongly g -convex functions with parameter settings stated in Proposition 5, c satisfying c < min { 1 / 6 L , 1 / 2 µ } and ξ 0 ∈ (2 µc, √ 2 µc ] , where f also has L -Lipschitz gradients.
Then, when M is a Hadamard manifold with sectional curvature lower bounded by − κ < 0 , the sequence of iterates { y k } generated by this algorithm satisfies f ( y k ) − f ( x ⋆ ) ≤ (cid:16)(cid:81) kj =1 (1 − ξ j ) (cid:17) · D 0 for all k ≥ 0 .
Moreover, if d ( x k +1 , y k +1 ) ≤
C ′ L,µ,c (cid:113)(cid:81) kj =1 (1 − ξ j ) · D 0 for all k ≥ 0 , then the sequence { ξ k } satisfies | ξ k − √ 2 µc | ≤ ϵ when k ≥ C κ,L,µ,c log( 1 / ϵ ) where C κ,L,µ,c is a constant depending on κ , L , µ , c .
Remark
To achieve full acceleration, we would require ξ k = √ 2 µc for all k ≥ 0 .
This theorem states that while we might not be able to have ξ k = √ 2 µc for all k ≥ 0 , we can still get arbitrarily close as the algorithm proceeds, and eventually achieve acceleration.
We conjecture that this analysis will also extend to non-Hadamard manifolds under suitable assumptions (( A 1 ), ( A 2 ), ( A 3 )) as discussed in [2, Section D].
4. Conclusion
In this work, we presented a general analysis of Riemannian optimisation methods using an energybased analysis framework that has gained popularity in the Euclidean setting and more recently in the Riemannian setting.
Such an analysis is also conducive to a study of accelerated first order
Riemannian descent methods.
To this end, we showed that we can obtain a accelerated algorithms for first order descent methods in a straightforward manner in the g -convex and strongly g -convex setting, and present an analysis for the latter case which extends an existing analysis.
Some open questions remain: can we achieve (eventual) acceleration for a fully proximal point method, or other higher order methods such as cubic-regularized Newton even on bounded domains?
Appendix A. More definitions
Let A be a geodesically unique convex subset of the Riemannian manifold M .
The zeroth order definition of g -convexity states that a function f : A → R is g -convex if for any two points x, y in
A ,
[Equation]
for all t ∈ [0 , 1] where γ x,y : [0 , 1]
→ A is the geodesic with end points given by x and y .
When f is differentiable, we obtain an equivalent definition of g -convex in terms of the gradient of f , which is often used to prove the results in this paper; this result is formally stated in [6, Theorem 11.21].
When a differentiable function f is g -convex, then for all x, y in A
[Equation]
Analogously, a differentiable function f is µ -strongly g -convex satisfies for all x, y in A
[Equation]
With the notion of curves and geodesics, one can transport vectors in a tangent space at one point to the tangent space at another point.
This is made possible via the concept of parallel transports.
The parallel transport between T x M and T y M for x, y ∈ M along curve c is denoted by Γ( c ) yx : T x M → T y M .
When c is a geodesic between x and y we omit the c in the notation and use Γ yx to simplify the notation.
A key property of parallel transports is that it is norm-preserving: for any v ∈ T x M , ∥ v ∥ x = ∥
Γ yx v ∥ y where Γ yx v ∈ T y M per the definition of Γ yx .
We use the parallel transport to hence define the property of L -Lipschitz gradients.
A function f : A → R is said to have L -Lipschitz gradients when it satisfies for all x
, y ∈ A ,
[Equation]
Such a function is also L - g -smooth i.e., for all x, y ∈
A [6, Corollary 10.54]
[Equation]
A twice differentiable function f : M → R is said to have ρ -Lipschitz continuous Riemannian
Hessians, when for all x, s in the domain of the exponential map,
[Equation]
Equivalently from [6, Corollary 10.56],
[Equation]
Appendix B. Rates for p -descent algorithms for non-convex functions and gradient-dominated functions
Theorem 8 (Rate for p -descent over non-convex functions) Let f a non-convex function, and let x 0 ∈ M be the initialisation.
Then, a p -descent algorithm (forward (Eq. 1a) or backward (Eq. 1b)) satisfies the following guarantee
[Equation]
Definition 9 ( ( τ, p ) -gradient dominated functions)
A differentiable function f : M → R is said to be ( τ, p ) -gradient dominated if x ⋆ is the global minimizer of f and for all x
[Equation]
Theorem 10 (Rate for p -descent algorithms over ( τ, p ) -gradient dominated functions) Let f be a ( τ, p ) -gradient dominated function, and let x 0 ∈ M be the initialisation.
Then, a p -descent algorithm satisfies the following guarantees
[Equation]
Appendix C. Proofs for the rate theorems in Section 2 and B
Proof [Proof of Theorem 2] We begin by noting that under the assumptions, the exponential map and its inverse exists at every v ∈ T x M for every x ∈ A .
Consider an energy function
E k
=
A k ( f
[Equation]
x [Equation] − f ( x ⋆ [Equation]
Here, { A k } k ≥ 1 is a sequence satisfying A k
+
=
A k + a k and x ⋆ is the minimizer of f .
The difference between E k +1 and E k is
[Equation]
Since f is g -convex
[Equation]
and
[Equation]
If A is a p -forward-descent algorithm w.r
[Equation]
t. f [Equation] we can use Equation 1a and bound the difference in energies
[Equation]
If A is a p -backward-descent algorithm w.r
[Equation]
t. f , we can use Equation 1b and bound the difference in energies
[Equation]
To bound the quantity inside the brackets in Equations 5 and
[Equation]
we use Lemma 11.
Specifically [Equation] we invoke the lemma with q = p / p − 1 and
• α = − a k cA k p − 1 p for Equation
, • α = − a k c
( A k + a k ) p − 1 p for Equation
to get
[Equation]
where c ′ p =
c 1 − p p (cid:16) p − 1 p (cid:17) p − 1 .
By definition of the exponential map, d ( x, z ) =
∥
Exp − 1 x ( z ) ∥ x for all x, z ∈ A .
Also, by ( A 2 ), x ⋆ ∈ A .
Therefore, d ( z, x ⋆ ) =
∥
Exp − 1 z ( x ⋆ ) ∥
z ≤ diam( A ) for any z ∈ A .
This further bounds of the difference in energy as
[Equation]
Choose
A k = k
( k +1) ...
( k + p − 1) p ! .
This gives a k =
A k +1 − A k
=
( k +1) ...
( k + p − 1) ( p − 1)! .
Furthermore,
[Equation]
We finally have
[Equation]
Summing both sides from k = 0 to k = T − 1 , we get
E
[Equation]
⇒ E T ≤ c ′′ p, fwd · diam( A ) p · T + E 0 ⇒
[Equation]
T ) −
[Equation]
x ⋆ ) ≤ c
[Equation]
p, fwd · T A T · diam( A ) p , and
[Equation]
⇒ E
[Equation]
≤ c
[Equation]
p
[Equation]
bwd
[Equation]
A
[Equation]
⇒
[Equation]
−
[Equation]
x
[Equation]
≤
[Equation]
( A ) p .
Since A T ≥ T p / p ! , T / A T ≤ p ! /
T p − 1 .
Consequently, p -fwd-descent ⇒ f ( x T ) − f ( x ⋆ ) ≤ c
[Equation] [Equation]
T p
[Equation]
p -bwd-descent ⇒ f
( x T ) − f ( x [Equation] c [Equation]
p p
[Equation]
p
[Equation]
[Equation]
[Equation]
T
[Equation]
Proof [Proof of Theorem 8]
If A is a p -forward descent algorithm w.r
[Equation]
t. f , we can use Equation 1a to get
[Equation]
If A is a p -backward descent algorithm w.r
[Equation]
t. f , we can use Equation 1b to get
[Equation]
Proof [Proof of Theorem 10] Consider the energy function
[Equation]
Then, we obtain
[Equation]
If A is a p -forward descent algorithm w.r
[Equation]
t. f , then using Eq.
1a
[Equation]
As a result
[Equation]
If A is a p -backward descent algorithm w.r
[Equation]
t. f , then using Eq.
1b
[Equation]
As a result
[Equation]
C.1. Proofs for the examples of descent methods
Proof [Rate for Riemannian gradient descent] Using the assumptions, we have for k ≥ 0 that
[Equation]
When η = 1 / L , we get a simplified bound as
[Equation]
Using Theorem 2 in the backward case with c = 1 / 2 L , p = 2 , we directly obtain
[Equation]
Proof [Rate for Riemannian proximal descent] Using the assumptions, we have for k ≥ 0 that
[Equation]
The proximal update also satisfies Log x k +1 ( x k ) =
η grad f ( x k +1 ) leading to
[Equation]
Using Theorem 2 in the forward case with c = η 2 , p = 2 , we directly obtain
[Equation]
Proof [Rate for Cubic-regularized Newton] For convenience, we will denote Γ Exp x ( s ) x by P s when operationalising the property of function with ρ -Lipschitz continuous Hessians when the choice of x is obvious Under our assumptions, the domain of the exponential map when restricted to x ∈ A is the tangent space at every point.
At iteration k , the update velocity satisfies
[Equation]
Using the fact that f has ρ -Lipschitz continuous Riemannian Hessians, we get
[Equation]
From Agarwal et al.
[1, Theorem 3], the gradient of m k at s k can be computed as
[Equation]
In the last step, we have added and subtracted P − 1 s k grad f ( x k +1 ) . This leads us to,
[Equation]
In the penultimate step [Equation] we have used the alternative characterisation of ρ -Hessian Lipschitz functions. Therefore,
[Equation]
Combining this with the descent statement previously
[Equation]
we get
[Equation]
When θ = ρ / [Equation] M = ρ
[Equation]
we get a concise inequality
[Equation]
Using Theorem 2 in the forward case with c = [Equation] √ [Equation]
√ ρ , p = 3 , we directly obtain
[Equation]
Appendix D. Proof for results in Section
In the proof that follow, we denote ∥ Log x ( w ) − Log x ( v ) ∥ x by d x ( w, v ) for convenience.
With this notation, the δ k is a valid distortion rate at iteration k if
[Equation]
D.1. Proof for convergence guarantees of Algorithm
Proof [Proof of Theorem 4] We analyse the difference in energy functions at iterations k and k + 1 .
We assume that f is µ -strongly g -convex, and at the end make the substitution µ =
[Equation]
We begin by simplifying ∆ E Dk .
First, using the fact that δ k +1 is a valid distortion rate, we get:
[Equation]
Next, since the tangent space T w M is Euclidean for w ∈ M
[Equation]
we have the canonical three-term lemma
[Equation]
which states
[Equation]
Using this with w = x k +
[Equation]
a = x ⋆
[Equation]
b =
z k +1 and c
= z k we get the bound
[Equation]
Due to the update step 2c
[Equation]
We use this to obtain the simplification
[Equation]
Applying the three-term lemma again with w = x k +1 , a = x k +
[Equation]
b =
z k +1 and c =
x ⋆
[Equation]
we obtain
[Equation]
Since the squared projected distance is effectively the squared norm of the distance between two vectors in a Euclidean space, we can use the fact that
[Equation]
This is due to the convexity of the function f a ( x ) = 12 ∥ x − a ∥ 2 .
Using the inequality over T x k +1 M with a = Log x k +
( z k +1 ) , b =
Log x k +1 ( x k +1 ) , c = Log x k +
( z k ) , λ =
α k +1 β
k +
+ α k +1 and w k +
=
λb + (1 − λ ) c for some w k +1 ∈ T x k +1 M , we get
[Equation]
The final inequality is due to the facts that
[Equation]
and that f is µ -strongly g -convex
[Equation]
Next
[Equation]
note that the choice of α k +1 and β k +1 satisfies
B k
=
B
k
α k +
δ k +1 β
k +
and hence T 1 = 0 .
[Equation]
Due to the form of the update in Eq.
2a [Equation] Log x k +
( y k ) =
τ k +1 1 − τ k
+1 Log x k +
( z k ) .
[Equation]
Finally, by definition of the constants
[Equation]
we have
[Equation]
Next
[Equation]
we look at ∆ E Fk .
[Equation]
As a result of these computations:
[Equation]
The choice of τ k +1 ensure that the coefficient of the f ( x k +1 ) − f ( y k ) term is 0 .
This leads to, along with µ = 0 ,
[Equation]
⋆
[Equation]
When, G c is a 2 -backward descent method, f ( y k +1 ) − f ( x k +1 ) ≤ − c
∥ grad f ( x k +1 )
∥ 2 x k +1 .
This gives
[Equation]
Choose B k +1 = 4 c and A k
+1 =
( k +1)( k +2).
Note that for this choice A k
=
A k +1 − A k
=
( k + 1) and therefore, cA k +1 > cA 2 k 2 .
Due to this,
[Equation]
This gives us a rate
[Equation]
Proof [Proof of Proposition 5] The proof of this proposition is directly given by Ahn and Sra [2,
Theorem 3.1] where we make the substitution ∆ γ → c .
By definition 2 µ ∆ γ = 2 µ · γ (1 − Lγ / 2 ) is strictly less than 0 under the preconditions of their theorem, whereas due to our generality, we will have to enforce it as a property of G c .
We also find that their theorem holds more generally when
Exp and Log is well-defined at every x ∈ A , hence the additional assumptions ( A 1 ), ( A 2 ) and ( A 3 ).
As noted earlier, when M is a Hadamard manifold, these assumptions hold.
D.2. Proofs for the sufficient conditions
Proof [Proof of Lemma 6]
We carefully follow the proof of Lemma 4.2 in Ahn and Sra [2].
By our assumption, ξ 0 ≤ √ 2 µc and 2 µc < 1 .
For convenience, we use λ k +
= β
[Equation]
and η k +1 = 1 β k[-+
 + α k +
-]
[-and η k +
 = 
 β k-] +
+ α k +1 .
[Equation]
Step ( i ) holds since f has L -Lipschitz continuous gradients.
Next, step ( ii ) holds due to the triangle inequality over M .
Finally, step ( iii ) holds due to the fact that x k +1 lies between y k and z k through Eq.
2a.
To get a bound on d ( x k +1 , z k +1 ) [Equation] we need to have a bound on d ( y k , z k ) and d ( y k , x ⋆ ) .
We can use the energy inequality from Prop. 5 with µ -strong g-convexity to get the following statements
[Equation]
With these we also have
[Equation]
∵ △
[Equation]
for Hadamard manifolds
[Equation]
However, this doesn’t quite help us yet, since d x k ( y k , z k ) ≤ d ( y k , z k ) [Equation] and we need the quantity on the RHS for the upper bound on d ( x k +1 , z k +1 ) .
Following the proof of Ahn and Sra [2, Prop.
C.7], we will analyse the quantity d x k +
( y k +1 , z k +1 ) .
[Equation]
Step ( i ) and ( iv ) use the fact that x k +1 lies between y k and z k by Eq.
2a.
Step ( ii ) uses the fact that f has L -Lipschitz continuous gradients.
Step ( iii ) applies the triangle inequality over M .
This gives us
[Equation]
We make note of the fact that ξ k +1 ≤ 1 and use the bound from Eq.
11 and Eq.
The final piece is to bound d ( y k
+1 , x k +1 ) and to show that λ k +1 (1 − τ k +1 )
− η k +1 Lτ
k +1 can be bounded in terms of a constant involving L, µ, c alone.
The first part is given by the statement of the lemma, which states
[Equation]
For the second part, we make use of global properties of the recurrence relation governing the sequence { ξ k } .
From Ahn and Sra [2, Proposition C.9], we have that if ξ 0 ≤ √ a , then ξ k ≤ √ a for all k ≥ 0 , where
[Equation]
for any δ ≥ 1 and a ∈ (0 , 1) .
We use this statement with a = 2 µc , and δ being the valid distortion rate at iteration k which is ≥ 1 .
Therefore,
[Equation]
The quantity 1 − √ 2 µc −
Lc strictly positive when c < 1 / 6 L .
Therefore, we have the bound on d ( y k , z k ) as
[Equation]
Using this to bound d ( x k +1 , z k +1 ) we obtain
[Equation]
Proof [Proof of Proposition 7] This proposition can be proven using the analysis of the recurrence relation as presented in Ahn and Sra [2, Section C.7].
The key tool of the analysis is the distance shrinking lemma, which we have proven for 2 -backward descent methods in general when c is sufficiently small.
Appendix E. Auxiliary lemmas
Lemma 11 (Conjugate lemma [11]) Let s, u be vectors in Euclidean space, and α be a scalar.
Then
[Equation]
Proof Let s ⋆ be the maximizer of the LHS (taken with respect to s ).
By the first order optimality, we have
[Equation]
Consequently,
[Equation]
Also,
[Equation]
Hence,
[Equation]