Accelerated Riemannian Optimization: Handling Constraints to Bound Geometric Penalties
author names withheld
Under Review for OPT
Abstract
We propose a globally-accelerated, ﬁrst-order method for the optimization of smooth and (strongly or not) geodesically-convex functions in Hadamard manifolds.
Our algorithm enjoys the same convergence rates as Nesterov’s accelerated gradient descent, up to a multiplicative geometric penalty and log factors.
Crucially, we can enforce our method to stay within a compact set we deﬁne.
Prior fully accelerated works resort to assuming that the iterates of their algorithms stay in some pre-speciﬁed compact set, except for two previous methods, whose applicability is limited to local optimization and to spaces of constant curvature, respectively.
Achieving global and general
Riemannian acceleration without iterates assumptively staying in the feasible set was asked as an open question in [50], which we solve for Hadamard manifolds.
In our solution, we show that we can use a linearly convergent algorithm for constrained strongly g-convex smooth problems to implement a Riemannian inexact proximal point operator that we use as a subroutine, which is of independent interest.
1. Introduction
Riemannian optimization concerns the optimization of a function deﬁned over a Riemannian manifold.
It is motivated by constrained problems that can be naturally expressed on Riemannian manifolds allowing to exploit the geometric structure of the problem and effectively transforming it into an unconstrained one.
Moreover, there are problems that are not convex in the Euclidean setting, but that when posed as problems over a manifold with the right metric, are convex when restricted to every geodesic and this allows for fast optimization [9, 14, 20, 27].
That is, they are geodesically convex (g-convex) problems, cf.
Deﬁnition 1.
Some applications of Riemannian optimization in machine learning include dictionary learning [22, 66], robust covariance estimation in Gaussian distributions [80], Gaussian mixture models [37], operator scaling [9], computation of BrascampLieb constants [13], Karcher mean [83], Wasserstein Barycenters [76], low-rank matrix completion [17, 35, 59, 68, 72], optimization under orthogonality constraints [31, 53], and sparse principal component analysis [33, 41, 44].
The ﬁrst seven problems are deﬁned over Hadamard manifolds, which we consider in this work.
In fact, the optimization in these cases is over symmetric spaces, which satisfy a property that one instance of our algorithm requires, cf.
Theorem 6.
Riemannian optimization, whether under g-convexity or not, is an extensive and active area of research, for which one aspires to develop Riemannian optimization algorithms that share analogous properties to the more broadly studied Euclidean methods, such as the following kinds of Riemannian ﬁrst-order methods: deterministic [16, 78, 81], adaptive [47], projection-free [76, 77], saddle-point-
© .
escaping [24, 25, 67, 85], stochastic [38, 49, 70], variance-reduced [63, 64, 83], and min-max methods [84], among others.
Riemannian generalizations to accelerated convex optimization are appealing due to their better convergence rates with respect to unaccelerated methods, specially in ill-conditioned problems.
Acceleration in Euclidean convex optimization is a concept that has been broadly explored and has provided many different fast algorithms.
A paradigmatic example is Nesterov’s Accelerated
Gradient Descent (AGD), cf.
[60], which is considered the ﬁrst general accelerated method, where the conjugate gradients method can be seen as an accelerated predecessor in a more limited scope [58].
There have been recent efforts to better understand this phenomenon in the Euclidean case [8, 29, 30, 45, 65, 79], which have yielded some fruitful techniques for the general development of methods and analyses.
These techniques have allowed for a considerable number of new results going beyond the standard oracle model, convexity, or beyond ﬁrst-order, in a wide variety of settings [5–7, 12, 18, 23, 28, 32, 36, 42, 46, 71, 73], among many others.
There have been some efforts to achieve acceleration for Riemannian algorithms as generalizations of AGD, cf.
Section 1.2.
These works try to answer the following fundamental question:
Can a Riemannian ﬁrst-order method enjoy the same rates of convergence as Euclidean AGD?
The question is posed under (possibly strongly) geodesic convexity and smoothness of the function to be optimized.
And due to the lower bound in [26], we know the optimization must be under bounded geodesic curvature of the Riemannian manifold, and we might have to optimize over a bounded domain.
Main result
In this work, we study the question above in the case of Hadamard manifolds ℳ of bounded sectional curvature and provide an instance of our framework for a wide class of Hadamard manifolds.
For a differentiable 𝑓 : ℳ → (cid:82) with a global minimizer at 𝑥 * , let 𝑥 0 ∈ ℳ be an initial point and 𝑅 be an upper bound on the distance 𝑑 ( 𝑥 0 , 𝑥 * ) .
If 𝑓 is 𝐿 -smooth and (possibly 𝜇 -strongly) g-convex in a closed ball of center 𝑥 * and radius 𝑂 ( 𝑅 ) , our algorithms obtain the same rates of convergence as AGD, up to logarithmic factors and up to a geometric penalty factor, cf.
Theorem 6.
See Table 1 for a succint comparison among accelerated algorithm and their rates.
This algorithm is a consequence of the general framework we design:
General accelerated scheme Riemacon .
Given a not necessarily accelerated, linearly-convergent subroutine for strongly g-convex smooth problems, constrained to a geodesically convex set 𝒳 , we design ﬁrst-order algorithms that enjoy the same rates as AGD when approximating min 𝑥 ∈𝒳 𝑓 ( 𝑥 ) , up to logarithmic factors and up to a geometric penalty factor, where 𝑓 : 𝒩 ⊂ ℳ → (cid:82) is a differentiable function that is smooth and g-convex (or strongly g-convex) in 𝒳 ⊂ 𝒩 , cf.
Theorem 4.
Importantly, our algorithms obtain acceleration without an undesirable assumption that most previous works had to make: that the iterates of the algorithm stay inside of a pre-speciﬁed compact set without any mechanism for enforcing or guarateeing this condition.
To the best of our knowledge only two previous methods are able to deal with some form of constraints, and they apply to the limited settings of local optimization [26] and constant sectional curvature manifolds [58], respectively.
Techniques in the rest of papers resort to assuming that the iterates of their algorithms are always feasible.
Removing this condition in general, global, and fully accelerated methods was posed as an open question in [50], that we solve for the case of Hadamard manifolds.
The difﬁculty of constraining problems in order to bound geometric penalties as well as the necessity of achieving this goal in order to provide full optimization guarantees with bounded geometric penalties is something that has also been noted in other kinds of Riemannian algorithms, cf.
We develop new techniques on inexact proximal methods in Riemannian manifolds and show that with access to a (not necessarily accelerated) constrained linear subroutine for strongly g-convex and smooth problems, we can inexactly solve a proximal subproblem to enough accuracy so it can be used in our accelerated outer loop, in the spirit of other Euclidean algorithms like Catalyst [54].
After building this machinery, we show that we are able to implement an inexact ball optimization oracle, cf.
[19], as an instance of our solution.
Crucially, the diameter 𝐷 of this ball depends on 𝑅 and the geometry only, so in particular it is independent on the condition number of 𝑓 .
We can use the linearly convergent algorithm in [26] for the implementation of the prox subroutine and we show that iterating the application of the ball optimization oracle leads to global accelerated convergence.
See Appendix A for a review of the geometric concepts and deﬁnitions used in this work and see
Appendix B for a format statement of our theorems.
1.1. Notation
Let ℳ be a uniquely geodesic 𝑛 -dimensional Riemannian manifold.
Given points 𝑥, 𝑦, 𝑧 ∈ ℳ , we abuse the notation and write 𝑦 in non-ambiguous and well-deﬁned contexts in which we should write Log 𝑥 ( 𝑦 ) .
For example, for 𝑣 ∈ 𝑇 𝑥 ℳ we have ⟨ 𝑣, 𝑦 − 𝑥 ⟩ =
−⟨ 𝑣, 𝑥 − 𝑦 ⟩ = ⟨ 𝑣, Log 𝑥 ( 𝑦 ) −
Log 𝑥 ( 𝑥 ) ⟩ = ⟨ 𝑣, Log 𝑥 ( 𝑦 ) ⟩ ; ‖ 𝑣 − 𝑦 ‖
=
𝑣 − Log 𝑥 ( 𝑦 ) ‖ ; ‖ 𝑧 − 𝑦
‖ 𝑥 =
‖ Log 𝑥 ( 𝑧 ) − Log 𝑥
( 𝑦 )
‖ ; and ‖ 𝑦 − 𝑥
‖ 𝑥 =
‖ Log 𝑥
( 𝑦 ) ‖ = 𝑑 ( 𝑦, 𝑥 ) .
We denote by 𝒳 a compact, uniquely geodesic g-convex set of diameter 𝐷 contained in an open set 𝒩 ⊂ ℳ and we use 𝐼 𝒳 for the indicator function of 𝒳 , which is 0 at points in 𝒳 and + ∞ otherwise.
For a vector 𝑣 ∈ 𝑇 𝑦 ℳ , we use Γ 𝑥𝑦 ( 𝑣 ) ∈
𝑇 𝑥 ℳ to denote the parallel transport of 𝑣 from 𝑇 𝑦 ℳ to 𝑇 𝑥 ℳ along the unique geodesic that connects 𝑦 to 𝑥 .
We call 𝑓 : 𝒩 ⊂ ℳ → (cid:82) a differentiable 𝐿 -smooth g-convex function we want to optimize.
We use 𝜀 to denote the approximation accuracy parameter, 𝑥 0 ∈ 𝒳 for the initial point of our algorithms, and ¯ 𝑅 def = 𝑑 ( 𝑥 0 , ¯ 𝑥 * ) for the initial distance to an arbitrary constrained minimizer ¯ 𝑥 * ∈ arg min 𝑥 ∈𝒳 𝑓 ( 𝑥 ) .
We use 𝑅 for an upper bound on the initial distance 𝑑 ( 𝑥 0 , 𝑥 * ) to an unconstrained minimizer 𝑥 * , if it exists.
The big- 𝑂 notation ̃︀ 𝑂 ( · ) omits log factors.
Note that in the setting of Hadamard manifolds, the bounds on the sectional curvature are 𝜅 min ≤ 𝜅 max ≤ 0 .
Hence for notational convenience, we deﬁne ¯ 𝜁 def = 𝜁 𝐷 =
𝐷 √︀ | 𝜅 min | coth( 𝐷 √︀ | 𝜅 min | ) ≥ 1 , ¯ 𝛿 def = 1 , and similarly 𝜁 def = 𝜁 𝑅 and 𝛿
def = 𝛿 𝑅 = 1 .
If 𝑣 ∈ 𝑇 𝑥 ℳ , we use Π ¯ 𝐵 (0 ,𝑟 ) ( 𝑣 ) ∈
𝑇 𝑥 ℳ for the projection of 𝑣 onto the closed ball with center atand radius 𝑟 .
1.2. Our results and comparisons with related work
In this work, we optimize functions deﬁned over Hadamard manifolds ℳ of ﬁnite dimension 𝑛 and of sectional curvature bounded lower bounded by 𝜅 min .
As all previous related works discussed in the sequel, we assume that we can compute the exponential and inverse exponential maps, and parallel transport of vectors for our manifold.
The differentiable function 𝑓 to be optimized is deﬁned over an open set 𝒩 ⊂ ℳ that contains a compact g-convex set 𝒳 of ﬁnite diameter 𝐷 .
Our function 𝑓 is 𝐿 -smooth and g-convex (or 𝜇 -strongly g-convex) in 𝒳 and we have access to it via a gradient oracle that can be queried at points in 𝒳 .
For this setting, we show in Theorem 4 that with access to a (possibly unaccelerated) linearly convergent subroutine for g-strongly smooth problems in 𝒳 , the algorithms we propose ﬁnd a point 𝑦 𝑇 ∈ 𝒳 such that 𝑓 ( 𝑦 𝑇 ) − min 𝑥 ∈𝒳 𝑓 ( 𝑥 ) ≤ 𝜀 after calling the gradient oracle and the subroutine the following number of times: ̃︀ 𝑂 (¯ 𝜁 √︀ 𝐿 ¯ 𝑅 2 /𝜀 ) for the g-convex case and ̃︀ 𝑂 (¯ 𝜁 √︀ 𝐿/𝜇 log( 𝜇 ¯ 𝑅 2 /𝜀 )) for the 𝜇 -strongly g-convex case, where ¯ 𝑅 def = 𝑑 ( 𝑥 0 , ¯ 𝑥 * ) and 𝑥 0 ∈ 𝒳 is an initial point.
Then in Theorem 6, we Method
[Table]
[61, AGD]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
𝒲 )
[Table] [Table]
[Equation]
𝜁
[Equation]
[Table]
Theorem
[Equation]
[Table] (cid:51) (cid:51) instantiate our algorithm with the method in [26] as subroutine and boost the convergence by implementing and sequentially applying an inexact ball optimization oracle and we obtain the rates ̃︀ 𝑂 ( 𝜁 2 √︀ 𝜁 + 𝐿𝑅 2 /𝜀 ) and ̃︀ 𝑂 ( 𝜁 2 √︀ 𝐿/𝜇 log( 𝜇𝑅 2 /𝜀 )) where 𝑅 is a bound on the initial distance 𝑑 ( 𝑥 0 , 𝑥 * ) to an unconstrained minimizer 𝑥 * .
In sum, the algorithms enjoy the same rates as AGD in the Euclidean space up to a factor of 𝜁 2 = 𝑅 2 𝜅 2min coth 2 ( 𝑅 √︀ | 𝜅 min | ) ≤ (1 + 𝑅 · | 𝜅 min | ) 2 (our geometric penalty) and up to universal constants and log factors.
Note that as the minimum curvature 𝜅 min approaches
we have 𝜁 → 1 .
We have summarized the comparison with related works in Table 1.
There are some works on
Riemannian acceleration that focus on empirical evaluation or that work under strong assumptions [3, 4, 40, 55, 56], see [57] for instance for a discussion on these works.
We compare the most related work with guarantees in Table 1.
There, column K? refers to the supported values of the sectional curvature, G? to whether the algorithm is global (any initial distance to a minimizer is allowed).
Here L and L ′ mean they are local algorithms that require initial distance 𝑂 (( 𝐿/𝜇 ) − 3 / 4 ) and 𝑂 (( 𝐿/𝜇 ) − 1 / 2 ) , respectively.
Column F? refers to whether there is full acceleration, meaning dependence on 𝐿 , 𝜇 , and 𝜀 like AGD up to possibly log factors.
Column C? refers to whether the method can enforce some constraints.
All methods require their iterates to be in some speciﬁed compact set, but the works with (cid:55) just assume the iterates will remain within the constraints.
We use 𝒲 def = √︁ 𝐿𝜇 log( 𝐿𝑅 2 𝜀 ) .
See Section 1.2 for a more thorough discussion on related work.
2. Algorithmic Framework and Pseudocode
In this section, we present our Riema nnian ac celerated algorithm for con strained g-convex optimization, or Riemacon.
This is a general framework that we later instantiate to provide a full algorithm.
See the pseudocode in Appendix B.
We start with an interpretation of our algorithm that helps understanding its high-level ideas.
The following intends to be a qualitative explanation, and we refer to the pseudocode and the supplementary material for the exact descriptions and analysis.
Euclidean accelerated algorithms can be interpreted, cf.
[8], as a combination of a gradient descent (GD) algorithm and an online learning algorithm with losses being the afﬁne lower bounds 𝑓
( 𝑥 𝑘 ) + ⟨∇ 𝑓 ( 𝑥 𝑘 ) , · − 𝑥 𝑘 ⟩ we obtain on 𝑓 ( · ) by applying convexity at some points 𝑥
𝑘 .
That is, the latter builds a lower bound estimation on 𝑓 .
By selecting the next query to the gradient oracle as a cleverly picked convex combination of the predictions given by these two algorithms, one can show that the instantaneous regret of the online learning algorithm can be compensated by the local progress GD makes, which leads to accelerated convergence.
In Riemannian optimization, there are two main obstacles.
Firstly, the ﬁrst-order approximations of 𝑓 at points 𝑥 𝑘 yield functions that are afﬁne but only with respect to their respective 𝑇 𝑥 𝑘 ℳ , and so combining these lower bounds that are only simple in their tangent spaces makes obtaining good global estimations not simple.
Secondly, when one obtains such global estimations, then one naturally incurs an instantaneous regret that is worse by a factor than is usual in Euclidean acceleration.
This factor is a geometric constant depending on the diameter 𝐷 of a set 𝒳 where the iterates and a (possibly constrained) minimizer lie.
As a consequence, the learning rate of GD would need to be multiplicatively increased by such a constant with respect to the one of the online learning algorithm in order for the regret to still be compensated with the local progress of
GD (and the rates worsen by this constant).
But if we ﬁx some 𝒳 of ﬁnite diameter, because GD’s learning rate is now larger, it is not clear how to keep the iterates in 𝒳 .
And if we do not have the iterates in one such set 𝒳 , then our geometric penalties could grow arbitrarily.
We ﬁnd the answer in implicit methods.
An implicit Euclidean (sub)gradient descent step is one that computes, from a point 𝑥 𝑘 ∈ 𝒳 , another point 𝑦
* 𝑘 = 𝑥 𝑘 − 𝜆𝑣
𝑘 ∈ 𝒳 , where
𝑣 𝑘 ∈ 𝜕 ( 𝑓 + 𝐼 𝒳 )( 𝑦 * 𝑘 ) , is a subgradient of 𝑓 + 𝐼 𝒳 at 𝑦 * 𝑘 .
Intuitively, if we could implement a Riemannian version of an implicit GD step then it should be possible to still compensate the regret of the other algorithm and keep all the iterates in the set 𝒳 .
Computing such an implicit step is computationally hard in general, but we show that approximating the proximal objective ℎ 𝑘 ( 𝑦 ) def = 𝑓 ( 𝑦 )
+ 12 𝜆 𝑑
[Equation]
𝑥 𝑘 , 𝑦 )with enough accuracy yields an approximate subgradient that can be used to obtain an accelerated algorithm as well.
In particular, we provide an accelerated scheme for which we show that the error incurred by the approximation of the subgradient can be bounded by some terms we can control, cf.
Lemma 8, namely a small term that appears in our Lyapunov function and also a term proportional to the squared norm of the approximated subgradient, which only increases the ﬁnal convergence rates by a constant.
This proximal approach works by exploiting the fact that the Riemannian
Moreau envelop is convex in Hadamard manifolds [10] and that the subproblem ℎ 𝑘 , deﬁned with our 𝜆 = 𝜁 2 𝐷 /𝐿 , is strongly g-convex and smooth with a condition number that only depends on the geometry.
For this reason, a local algorithm like the one in [26] can be implemented in balls whose radius is independent on the condition number of 𝑓 .
Besides these steps, we use a coupling of the approximate implicit RGD and of a mirror descent (MD) algorithm, along with a technique in [50] to move dual points to the right tangent spaces without incurring extra geometric penalties, that we adapt to work with dual projections, cf.
Lemma 9.
Importantly, the MD algorithm keeps the dual point close to the set 𝒳 by using the projection in Line 12, which implies that the point 𝑥 𝑘 is close to 𝒳 as well, and this is crucial to keep low geometric penalties.
This MD approach is a mix between follow-the-regularized-leader algorithms, that do not project the dual variable, and pure mirror descent algorithms that always project the dual variable.
In the analysis, we note that partial projection also works, meaning that deﬁning a new dual point that is closer to all of the points in the feasible set but without being a full projection leads to the same guarantees.
Because we use the mirror descent lemma over 𝑇 𝑦 𝑘 ℳ , what we described translates to: we can project the dual 𝑧 𝑦 𝑘 𝑘 onto a ball deﬁned on 𝑇 𝑦 𝑘 ℳ that contains the pulled-back set Log 𝑦 𝑘 ( 𝒳 ) and by means of that trick we can keep the iterates 𝑥 𝑘 close to 𝒳 .
And at the same time, the point for which we prove guarantees, namely 𝑦 𝑘 , is always in 𝒳 .
Finally, we instantiate our subroutine with the algorithm in [26], in balls of radius independent on the condition number of 𝑓 and show in Theorem 6 that if we iterate this approximate implementation of a ball optimization oracle, we obtain convergence at a globally accelerated rate.
We note [81, Thm. 15] also provided a claimed linearly convergent algorithm for constrained strongly g-convex smooth problems, and thus in principle it could be used for our subroutine.
Unfortunately, we noticed that the proof is ﬂawed when the optimization is constrained.
The ﬁrst inequality in their proof only holds in general for unconstrained problems and not for projected Riemannian gradient descent, not even for the Euclidean constrained case.
3. Conclusion and future directions
In this work, we pursued an approach that, by designing and making use of inexact Riemannian proximal methods, yielded accelerated optimization algorithms.
Consequently we were able to work without an undesirable assumption that most previous methods required, whose potential satisﬁability is not clear: that the iterates stay in certain speciﬁed geodesically-convex set without enforcing them to be in the set.
A future direction of research is the study of whether there are algorithms like ours that incur even lower geometric penalties or that do not incur log(1 /𝜀 ) factors.
Another interesting direction consists of studying generalizations of our approach to more general manifolds, namely the full Hadamard case, and manifolds of non-negative or even of bounded sectional cuvature.
Appendix A. Geometric preliminaries
We provide deﬁnitions of Riemannian geometry concepts that we use in this work.
The interested reader can refer to [11, 62] for an in-depth review of this topic, but for this work the following notions will be enough.
A Riemannian manifold ( ℳ , g ) is a real 𝐶 ∞ manifold ℳ equipped with a metric g , which is a smoothly varying, i.e., 𝐶 ∞ , inner product.
For 𝑥 ∈ ℳ , denote by 𝑇 𝑥 ℳ the tangent space of ℳ at 𝑥 .
For vectors 𝑣, 𝑤 ∈ 𝑇 𝑥 ℳ , we denote the inner product of the metric by ⟨ 𝑣, 𝑤 ⟩ 𝑥 and the norm it induces by ‖ 𝑣 ‖ 𝑥 def = √︀ ⟨ 𝑣, 𝑣 ⟩ 𝑥 .
Most of the time, the point 𝑥 is known from context, in which case we write ⟨ 𝑣, 𝑤 ⟩ or ‖ 𝑣 ‖ .
A geodesic of length ℓ is a curve 𝛾 : [0 , ℓ ] → ℳ of unit speed that is locally distance minimizing.
A uniquely geodesic space is a space such that for every two points there is one and only one geodesic that joins them.
In such a case the exponential map Exp 𝑥 : 𝑇 𝑥 ℳ → ℳ and the inverse exponential map Log 𝑥 : ℳ → 𝑇 𝑥 ℳ are well deﬁned for every pair of points, and are as follows.
Given 𝑥, 𝑦 ∈ ℳ , 𝑣 ∈ 𝑇 𝑥 ℳ , and a geodesic 𝛾 of length ‖ 𝑣 ‖ such that 𝛾 (0)
=
𝑥 , 𝛾 ( ‖ 𝑣 ‖ ) =
𝑦 , 𝛾 ′ (0) =
𝑣/
‖ 𝑣 ‖ , we have that Exp 𝑥 ( 𝑣 )
= 𝑦 and Log 𝑥 ( 𝑦 )
=
𝑣 .
We denote by 𝑑 ( 𝑥, 𝑦 ) the distance between 𝑥 and 𝑦 , and note that it takes the same value as ‖ Log 𝑥 ( 𝑦 ) ‖ .
The manifold ℳ comes with a natural parallel transport of vectors between tangent spaces, that formally is deﬁned from a way of identifying nearby tangent spaces, known as the Levi-Civita connection ∇ [51].
We use this parallel transport throughout this work.
Given a 2 -dimensional subspace 𝑉 ⊆
𝑇 𝑥 ℳ of the tangent space of a point 𝑥 , the sectional curvature at 𝑥 with respect to 𝑉 is deﬁned as the Gauss curvature, for the surface Exp 𝑥 ( 𝑉 ) at 𝑥 .
The
Gauss curvature at a point 𝑥 can be deﬁned as the product of the maximum and minimum curvatures of the curves resulting from intersecting the surface with planes that are normal to the surface at 𝑥 .
A Hadamard manifold is a complete simply connected Riemannian manifold whose sectional curvature is non-positive, like the hyperbolic space or the space of 𝑛 × 𝑛 symmetric positive deﬁnite matrices with the metric ⟨ 𝑋, 𝑌 ⟩ 𝐴 def =
Tr( 𝐴 − 1 𝑋𝐴 − 1 𝑌 ) where 𝑋, 𝑌 are in the tangent space of 𝐴 .
Hadamard manifolds are uniquely geodesic.
Note that in a general manifold Exp 𝑥 ( · ) might not be deﬁned for each 𝑣 ∈ 𝑇 𝑥 ℳ , but in a Hadamard manifold of dimension 𝑛 , the exponential map at any point is a global diffeomorphism between 𝑇 𝑥 ℳ ∼ =
(cid:82) 𝑛 and the manifold, and
so the exponential map is deﬁned everywhere.
We now proceed to deﬁne the main properties that will be assumed on our model for the function to be minimized and on the feasible set 𝒳 .
Deﬁnition 1 Let 𝑓 : 𝒩 ⊂ ℳ → (cid:82) be a differentiable function deﬁned on an open set 𝒩 contained in a Riemannian manifold ℳ .
Given 𝐿 ≥ 𝜇 > 0 , we say that 𝑓 is 𝐿 -smooth , and respectively 𝜇 -strongly g-convex , in a set 𝒳 ⊆ 𝒩 if for any two points 𝑥, 𝑦 ∈ 𝒳 , 𝑓 satisﬁes
[Equation]
If the previous inequality is satisﬁed with 𝜇 = 0 , we say the function is g-convex in 𝒳 .
We present the following fact about the squared-distance function, when one of the arguments is ﬁxed.
The constants 𝜁 𝐷 , 𝛿 𝐷 below appear everywhere in Riemannian optimization because, among other things, Fact 2 yields Riemannian inequalities that are analogous to the equality in the Euclidean cosine law of a triangle, cf.
Corollary 14, and these inequalities have wide applicability in the analyses of Riemannian methods.
Fact 2 (Local information of the squared-distance) Let ℳ be a Riemannian manifold of sectional curvature bounded by [ 𝜅 min , 𝜅 max ] that contains a uniquely g-convex set 𝒳 ⊂ ℳ of diameter 𝐷 < ∞ .
Then, given 𝑥, 𝑦 ∈ 𝒳 we have the following for the function Φ 𝑥 : ℳ → (cid:82) , 𝑦 ↦→
12 𝑑 ( 𝑥, 𝑦 ) 2 : ∇ Φ 𝑥 ( 𝑦 ) =
− Log 𝑦 ( 𝑥 ) and 𝛿 𝐷
[Equation]
where
[Equation]
[Equation]
and
[Equation]
Consequently, Φ 𝑥 is 𝛿 𝐷 -strongly g-convex and 𝜁 𝐷 -smooth in 𝒳 .
See [52] for a proof.
In particular, for Hadamard manifolds, Φ 𝑥 is 1 -strongly g-convex and sublevel sets of g-convex functions are g-convex sets, so balls are g-convex in these manifolds [11].
Appendix B. Algorithms’ Pseudocode and formal statements of our theoretical results
Recall our abuse of notation for points 𝑝 ∈ ℳ to mean Log 𝑞 ( 𝑝 ) in contexts in which one should place a vector in 𝑇 𝑞 ℳ and note that in our algorithm 𝑥 𝑘 and 𝑦 𝑘 are points in ℳ whereas 𝑧 𝑥 𝑘 𝑘 ∈ 𝑇 𝑥 𝑘 ℳ , 𝑧 𝑦 𝑘 𝑘 , ¯ 𝑧 𝑦 𝑘 𝑘 ∈ 𝑇 𝑦 𝑘 ℳ .
We present our algorithmic framework in Algorithm 1.
Using the insights explained in Section 2, we show the following inequality on 𝜓 𝑘 , deﬁned below, that will be used as a Lyapunov function to prove the convergence rates of our Riema nnian ac celerated algorithm for con strained g-convex optimization, or Riemacon 1 (Algorithm 1).
Proposition 3 [ ↓ ] By using the notation of Algorithm 1, let
[Equation]
Then, for all 𝑘 ≥ 1 , we have (1 − ∆ 𝑘 ) 𝜓 𝑘 ≤ 𝜓 𝑘 − 1 .
Finally, we can state our theorem for the optimization of 𝐿 -smooth and g-convex functions.
Input: Feasible set 𝒳 .
Initial point 𝑥 0 ∈ 𝒳 ⊂ 𝒩 .
Diff.
function 𝑓 : 𝒩 ⊂ ℳ → (cid:82) for a Hadamard manifold ℳ that is 𝐿 -smooth and g-convex in 𝒳 .
Optionally: ﬁnal iteration 𝑇 or accuracy 𝜀 .
If 𝜀 is provided, compute the corresponding 𝑇 , cf.
Theorem 4.
Parameters:
• Geometric penalty 𝜉 def = 4 𝜁 2 𝐷 − 3 ≤ 8¯ 𝜁 − 3 = 𝑂
(¯ 𝜁 ) .
• Implicit Gradient Descent learning rate 𝜆 def = 𝜁 2 𝐷 /𝐿 .
• Mirror Descent learning rates 𝜂 𝑘 def =
𝑎 𝑘 /𝜉 .
• Proportionality constant in the proximal subproblem accuracies: ∆ 𝑘 def =
Deﬁnition: (computation of this value is not needed)
• Prox.
accuracies: 𝜎 𝑘 def = Δ
𝑘 𝑑 ( 𝑥 𝑘 ,𝑦 * 𝑘 ) 2 78 𝜆
where 𝑦 * 𝑘 def = arg min 𝑦 ∈𝒳 { 𝑓 ( 𝑦 ) + 12 𝜆 𝑑 ( 𝑥 𝑘 , 𝑦 ) 2 } .
1: 𝑦 0 ← 𝑥 0 ; 𝐴 0 ← 200 𝜆𝜉 2: 𝑧 𝑥 0 0 ← 0 ∈ 𝑇 𝑥 0 ℳ ; ¯ 𝑧 𝑦 0 0 ← 𝑧 𝑦 0 0 ← 0 ∈ 𝑇 𝑦 0 ℳ 3: for 𝑘 = 1 to 𝑇 do 4: 𝑎 𝑘 ← 2 𝜆 𝑘 +32 𝜉5: 𝐴 𝑘 ← 𝑎 𝑘 /𝜉 + 𝐴 𝑘 − 1 = ∑︀ 𝑘𝑖 =1 𝑎 𝑖 /𝜉 + 𝐴 0 = 𝜆 (︁ 𝑘 ( 𝑘 +1+64 𝜉 ) 5 𝜉 + 200 𝜉 )︁ 6: 𝑥 𝑘 ←
Exp 𝑦 𝑘 − 1 ( 𝑎 𝑘 𝐴 𝑘 − 1 + 𝑎 𝑘 ¯ 𝑧 𝑦 𝑘 − 1 𝑘 − 1 + 𝐴 𝑘 − 1 𝐴 𝑘 − 1 + 𝑎 𝑘 𝑦 𝑘 − 1 )
=
Exp 𝑦 𝑘 − 1 ( 𝑎 𝑘 𝐴 𝑘 − 1 + 𝑎 𝑘 ¯ 𝑧 𝑦 𝑘 − 1 𝑘 − 1 ) ◇ Coupling 7: 𝑧 𝑥 𝑘 𝑘 − 1 ← Γ 𝑥 𝑘 𝑦 𝑘 − 1 (¯ 𝑧 𝑦 𝑘 − 1 𝑘 − 1 )
+ Log 𝑥 𝑘 ( 𝑦 𝑘 − 1 )
= Log 𝑥 𝑘
(Exp 𝑦 𝑘 (¯ 𝑧 𝑦 𝑘 − 1 𝑘 − 1 )) 8: 𝑦 𝑘 ← 𝜎 𝑘 -minimizer of the proximal problem min 𝑦 ∈𝒳 { 𝑓 ( 𝑦 )
+ 12 𝜆 𝑑 ( 𝑥 𝑘 , 𝑦 ) 2 } 9: 𝑣
𝑥𝑘 ← − Log 𝑥 𝑘
( 𝑦 𝑘 ) /𝜆
◇ Approximate subgradient 10: 𝑧 𝑥 𝑘 𝑘 ← 𝑧 𝑥 𝑘 𝑘 − 1 − 𝜂 𝑘 𝑣 𝑥𝑘 ◇ Mirror Descent step 11: 𝑧 𝑦 𝑘 𝑘 ← Γ 𝑦 𝑘 𝑥 𝑘 ( 𝑧 𝑥 𝑘 𝑘 )
+ Log 𝑦 𝑘 ( 𝑥 𝑘 )
◇ Moving the dual point to 𝑇 𝑦 𝑘 ℳ 12: ¯ 𝑧 𝑦 𝑘 𝑘 ← Π ¯ 𝐵 (0 ,𝐷 ) ( 𝑧 𝑦
𝑘 𝑘 ) ∈ 𝑇
𝑦 𝑘 ℳ ◇ Easy projection done so the dual point is not very far 13: end for 14: return 𝑦 𝑇 .
Theorem 4 [ ↓ ] Let ℳ be a ﬁnite-dimensional Hadamard manifold of bounded sectional curvature, and consider 𝑓 : 𝒩 ⊂ ℳ → (cid:82) be an 𝐿 -smooth and g-convex differentiable function in a compact g-convex set 𝒳 ⊂ 𝒩 of diameter 𝐷 ,
¯ 𝑥 * ∈ arg min 𝑥 ∈𝒳
𝑓
( 𝑥 ) , and ¯ 𝑅 def =
𝑑 ( 𝑥 0 , ¯ 𝑥 * ) .
For any 𝜀 > 0 , Algorithm 1 yields an 𝜀 -minimizer 𝑦 𝑇 ∈ 𝒳 after 𝑇 = 𝑂 (¯ 𝜁 √︁ 𝐿 ¯ 𝑅 2 𝜀 ) iterations. If the function is 𝜇 -strongly convex then, via a sequence of restarts, we converge in 𝑂 (¯ 𝜁 √︁ 𝐿𝜇 log( 𝜇 ¯ 𝑅 2 𝜀 )) iterations.
We note that a straightforward corollary from our results is that if we can compute the exact
Riemannian proximal point operator and we use it as the implicit gradient descent step in Lineof Algorithm 1, then the method is an accelerated proximal point method.
One such Riemannian algorithm was previously unknown in the literature as well.
Finally, we instantiate Algorithm 1 to implement approximate ball optimization oracles in an accelerated way.
We show that applying these oracles sequentially leads to global accelerated convergence.
Moreover, we show that the iterates do not get farther than 2 𝑅 from 𝑥 * , which ultimately leads to the geometric penalty being a function of ◇ [26] as subroutine Algorithm Theorem 𝜁 and not on the condition number of 𝑓 .
For the subroutine in Line 8 of Algorithm 1, we use the algorithm in [26, Section 6], and for that we require the following.
Assumption 5 Let R be the curvature tensor of a Riemannian manifold ℳ .
Its covariant derivative is ∇ R = 0 .
We note that locally symmetric manifolds, like SO( 𝑛 ) , the SPD matrix manifold, the Grasmannian manifold, manifolds of constant sectional curvature are all manifolds such that ∇ R = 0 .
We argue that this assumption is mild, since in particular these manifolds cover all of the applications in
Section 1.
Theorem 6 [ ↓ ] Let ℳ be a ﬁnite-dimensional Hadamard manifold of bounded sectional curvature satisfying Assumption 5.
Consider 𝑓 : 𝒩 ⊂ ℳ → (cid:82) be an 𝐿 -smooth and 𝜇 -strongly g-convex differentiable function in ¯ 𝐵 ( 𝑥 * , 3 𝑅 ) , where 𝑥 * is its global minimizer and where 𝑅 ≥ 𝑑 ( 𝑥 0 , 𝑥 * ) for an initial point 𝑥 0 .
For any 𝜀 > 0 , Algorithm 2 yields an 𝜀 -minimizer after ̃︀ 𝑂
( 𝜁 2 √︀ 𝐿/𝜇 log( 𝐿𝑅 2 /𝜀 )) calls to the gradient oracle of 𝑓 .
By using regularization, this algorithm 𝜀 -minimizes the g-convex case ( 𝜇 = 0 ) after ̃︀ 𝑂 ( 𝜁 2 √︀ 𝜁 + 𝐿𝑅 2 /𝜀 ) gradient oracle calls.
Appendix C. More related work
We comment on the related work in Table 1 and discuss more related work.
[82] obtain an algorithm that, up to constants, achieves the same rates as AGD in the Euclidean space, for 𝐿 -smooth and 𝜇 -strongly g-convex functions but only locally , namely when the initial point starts in a small neighborhood 𝑁 of the minimizer 𝑥 * : a ball of radius 𝑂 (( 𝜇/𝐿 ) 3 / 4 ) around it.
[2] generalize the previous algorithm and, by using similar ideas as in [82] for estimating a lower bound on 𝑓 , they adapt the algorithm to work globally, proving that it eventually decreases the objective as fast as AGD.
However, as [57] noted, it takes as many iterations as the ones needed by Riemannian gradient descent (RGD) to reach the neighborhood of the previous algorithm.
The latter work also noted that in fact
RGD and the algorithm in [82] can be run in parallel and combined to obtain the same convergence rates as in [2], which suggested that for this technique, full acceleration with the rates of AGD only happens over the small neighborhood 𝑁 in [82].
Note however that [2] show that their algorithm will decrease the function value faster than RGD, but this is not quantiﬁed.
[43] developed a different framework, arising from [2] but with the same guarantees for accelerated ﬁrst-order methods.
We do not feature it in the table.
[26] showed, under mild assumptions, that in a ball of center 𝑥 ∈ ℳ and radius 𝑂 (( 𝜇/𝐿 ) 1 / 2 ) containing 𝑥 * , the pullback function
𝑓 ∘ Exp 𝑥 : 𝑇 𝑥 ℳ → (cid:82) is Euclidean, strongly convex, and smooth with condition number 𝑂 ( 𝐿/𝜇 ) , so AGD yields local acceleration as well.
In short, acceleration is possible in a small neighborhood because there the manifold is almost
Euclidean and the geometric deformations are small in comparison to the curvature of the objective.
These techniques fail for the g-convex case since the neighborhood becomes a point ( 𝜇/𝐿 = 0 ).
Finding fully accelerated algorithms that are global presents a harder challenge.
By a fully accelerated algorithm we mean one with rates with same dependence as AGD on 𝐿 , 𝜀 , and if it applies, on 𝜇 .
[57] provided such algorithms for g-convex functions, strongly or not, deﬁned over manifolds of constant sectional curvature and constrained to a ball of radius 𝑅 .
The convergence rates initially had large constants with respect to 𝑅 but were later improved, cf.
Table 1.
Kim and Yang [50] designed global algorithms with the same rates as AGD up to universal constants and a factor of ¯ 𝜁 , their geometric penalty.
However, they need to assume that the iterates of their algorithm remain in their feasible set 𝒳 and they point out on the necessity of removing such an assumption, which they leave as an open question.
Our work solves this question for the case of Hadamard manifolds.
In their technique, they show that they can use the structure of the accelerated scheme to move lower bound estimations on 𝑓 ( 𝑥 * ) from one particular tangent space to another without incurring extra errors, when the right Lyapunov function is used.
By moving lower bounds here we mean ﬁnding suitable lower bounds that are simple (a quadratic in their case), if pulled-back to one tangent space, if we start with a similar bound that is simple when pulled-back to another tangent space.
Lower bounds.
In this paragraph, we omit constants depending on the curvature bounds in the big- 𝑂 notations for simplicity.
[34] proved an optimization lower bound showing that acceleration in
Riemannian manifolds is harder than in the Euclidean space.
[26] largely generalized their results.
They essentially show that for a large family of Hadamard manifolds, there is a function that is smooth and strongly g-convex in a ball of radius 𝑅 that contains the minimizer 𝑥 * , and for which ﬁnding a point that is 𝑅/ 5 close to 𝑥 * requires ̃︀ Ω( 𝑅 ) calls to the gradient oracle.
Note that these results do not preclude the existence of a fully accelerated algorithm with rates ̃︀ 𝑂 ( 𝑅 )+ AGD rates, for instance.
A similar hardness statement is provided for smooth and only g-convex functions.
Also, reductions as in [57] evince this hardness is also present in this case.
Handling constraints to bound geometric penalties.
In our algorithm and in all other known fully accelerated algorithms, learning rates depend on the diameter of the feasible set.
This is natural: estimation errors due to geometric deformations depend on the diameter via the constants 𝜁 𝐷 , 𝛿 𝐷 , the cosine-law inequalities Corollary 14, or other analogous inequalities, and the algorithms take these errors into account.
All other previous works are not able to deal with any constraints and hence they simply assume that the iterates of their algorithms stay within one such speciﬁed set, except for [57] and [26] that enforce a ball constraint, as we explained above.
However, these two works have their applicability limited to spaces of constant curvature and to local optimization, respectively.
Note that even if one could show that given a choice of learning rate, convergence implies that the iterates will remain in some compact set, then because the learning rates depend on the diameter of the set, and the diameter of the set would depend on the learning rates, one cannot conclude from this argument that the assumption these works make is going to be satisﬁed.
In contrast, in this work, we design a general accelerated framework and an instance of it that keep the iterates bounded, effectively bounding geometric penalties while we do not need to resort to any other extra assumptions, solving the open question in [50].
Riemannian proximal methods
There have been some works that study proximal methods in
Riemannian manifolds, but most of them focus on asymptotic results or assume the proximal operator can be computed exactly [15, 16, 21, 48, 74].
The rest of these works study proximal point methods under different inexact versions of the proximal operator as ours and they do not show how to implement their inexact version in applications, like in our case of smooth and g-convex optimization.
In contrast, we implement the inexact proximal operator with a ﬁrst-order method [1] provide a convergence analysis of an inexact proximal point method but when applied to optimization they assume the computation of the proximal operator is exact.
[69] uses a different inexact condition and proves linear convergence, under a growth condition on 𝑓 .
[75] obtains linear convergence of an inexact proximal point method under a different growth assumption on 𝑓 and under an absolute error condition on the proximal function.
Appendix D. Proof of Theorem 4, Analysis of Algorithm
We start by noting a property that our parameters satisfy.
Lemma
For the parameter choices of 𝑎 𝑘 and 𝐴 𝑘 − 1 in Algorithm 1 we have, for all 𝑘 ≥ 1 :
[Equation]
Proof It is a simple computation to check that 𝑎 𝑘 and 𝐴 𝑘 − 1 satisfy such inequality.
The inequalities are equivalent to the following, which trivially holds:
[Equation]
We now prove Proposition 3, which will allow us to use 𝜓 𝑘 as a Lyapunov function to show the ﬁnal convergence rates.
The proof will use Lemma 8 and Lemma 9, that we state and prove afterwards.
Proof [Proposition 3]Inequality (1 − ∆ 𝑘 ) 𝜓 𝑘 ≤ 𝜓 𝑘 − 1 is equivalent to
[Equation]
which, by bounding (1 − ∆ 𝑘 )( 𝑓 ( 𝑦 𝑘 ) − 𝑓 (¯ 𝑥 * )) ≤ 𝑓 ( 𝑦 𝑘 ) − 𝑓 (¯ 𝑥 * ) and reorganizing, is implied by the following:
[Equation]
We have that due to the projection in Line 12, then 𝑥 𝑘 is not very far from any 𝑝 ∈ 𝒳 :
[Equation]
where 1 holds by the deﬁnition of 𝑥 𝑘
and the fact 𝑦 𝑘 − 1 , 𝑝 ∈ 𝒳 , and 2 is due to the projection deﬁning ¯
𝑧 𝑦 𝑘 − 1 𝑘 − 1 .
Now we use the ﬁrst part of Lemma 8 with both 𝑥 ← 𝑦 𝑘 − 1 and 𝑥 ← ¯ 𝑥 *
and we bound the resulting errors 𝜀 𝑘 ( · ) by using the second part of Lemma 8.
We also use Lemma 9, so it is enough to prove the following:
[Equation]
Note that thanks to Lemma 9 now we have the potentials on the right hand side as expressions in the tangent space of 𝑥 𝑘 .
Also, note that we have canceled some potentials proportional to ∆ 𝑘 coming from the bound on the error 𝜀 𝑘 ( · ) .
Now we use that by deﬁnition of 𝑥 𝑘 we have, for all 𝑣 ∈ 𝑇 𝑥 𝑘 ℳ , 𝐴 𝑘 − 1 ⟨ 𝑣, 𝑥 𝑘 − 𝑦 𝑘 − 1 ⟩ = − 𝑎 𝑘 ⟨ 𝑣, 𝑥 𝑘 − 𝑧 𝑥 𝑘 𝑘 − 1 ⟩ , so we use this fact for 𝑣 = 𝑣 𝑥𝑘 and use the following identity, that holds by the deﬁnion of 𝑧 𝑥 𝑘 𝑘 def = 𝑧 𝑥 𝑘 𝑘 − 1 − 𝜂 𝑘 𝑣 𝑥𝑘 : 𝑎 𝑘 /𝜉
[Equation]
so that, after canceling terms, it is enough to prove:
[Equation]
Now we show that in the previous inequality (2), the ﬁrst line cancels with the last line.
Note that ( 𝑎 𝑘 [Equation] − [Equation]
𝜉
[Equation]
= 𝜉 − 1 .
Thus, by using again the deﬁnition of 𝑧 𝑥 𝑘 𝑘 , we have: 𝑎 𝑘
[Equation]
Finally, it only remains to prove:
[Equation]
which holds by Lemma 7.
We now show the two auxiliary lemmas that we used in the previous proof.
Lemma 8 Let ℎ 𝑘 ( 𝑥 ) def = 𝑓 ( 𝑥 ) + 12 𝜆 𝑑 ( 𝑥 𝑘 , 𝑥 ) 2 be the strongly g-convex function used at step 𝑘 , and let 𝑦 * 𝑘 = arg min 𝑦 ∈𝒳 ℎ 𝑘 ( 𝑦 ) .
Then, for 𝑦 𝑘 ∈ 𝒳 , if we let 𝑣 𝑥𝑘 def = − Log 𝑥 𝑘
( 𝑦 𝑘 ) /𝜆 , then the following holds, for all 𝑥 ∈ 𝒳 :
[Equation]
where 𝜀 𝑘 ( 𝑥 ) def = − 1 𝜆 ⟨ 𝑦 𝑘 − 𝑦
* 𝑘 , 𝑥 − 𝑥 𝑘 ⟩ 𝑥
𝑘
+ ( ℎ 𝑘 ( 𝑦 𝑘 ) − ℎ 𝑘 ( 𝑦 * 𝑘 )) .
Moreover, if 𝑦 𝑘 satisﬁes
[Equation]
then we have
[Equation]
Proof The function ℎ 𝑘 is 1 𝜆 -strongly g-convex because by Fact 2 the function 12 𝑑 ( 𝑥 𝑘 , 𝑥 ) 2 is 1strongly g-convex in a Hadamard manifold.
By the ﬁrst-order optimality condition of ℎ 𝑘 at 𝑦 * 𝑘 we have that ˜ 𝑣 𝑦𝑘 def = 𝜆 − 1 Log 𝑦 * 𝑘 ( 𝑥 𝑘 ) ∈ 𝜕 ( 𝑓 + 𝐼 𝒳 )( 𝑦 * 𝑘 ) is a subgradient of 𝑓 + 𝐼 𝒳 at 𝑦 * 𝑘 .
Thus, we have, for all 𝑥 ∈ 𝒳 and for ˜ 𝑣 𝑥𝑘 def =
Γ 𝑥 𝑘 𝑦
𝑘
(˜ 𝑣 𝑦𝑘 ) : 𝑓
[Equation]
𝑥 )
[Equation]
where 1 holds because ˜ 𝑣 𝑦𝑘 ∈ 𝜕 ( 𝑓 + 𝐼 𝒳 )( 𝑦 * 𝑘 ) and 𝑥, 𝑦 * 𝑘 ∈ 𝒳 .
In 2 , we used the ﬁrst part of
Lemma 16 along with ¯ 𝛿 = 1 .
We just added and subtracted some terms in 3 , and in 4 , we dropped 𝜆 2 ‖ ˜ 𝑣 𝑥𝑘 ‖ 2 , and we used the deﬁnitions of ℎ 𝑘 , ˜ 𝑣 𝑥𝑘 , and 𝑣 𝑥𝑘 = − Log 𝑥 𝑘
( 𝑦 𝑘 ) /𝜆 .
Now we proceed to prove the second part.
The following holds:
[Equation]
where 1 groups some terms and uses Cauchy-Schwartz.
In inequality 2 , for the ﬁrst term we bounded the distance between 𝑦 * 𝑘 and 𝑦 𝑘 estimated from 𝑇 𝑥 𝑘 ℳ by the actual distance, which is a property that holds in Hadamard manifolds and it holds by the ﬁrst part of Corollary 13 with ¯ 𝛿 = 1 , 𝑝 ← 𝑦 * [Equation] 𝑧 𝑦 [Equation]
The second term is substituted by a term of equal value by using Euclidean trigonometry in 𝑇 𝑥 𝑘 ℳ , as in the following.
Let 𝑤 def =
[Equation]
𝐴 𝑘 − 1 Log 𝑥 𝑘
( 𝑦 𝑘 − 1 )) and let 𝑢 ∈ 𝑇
𝑥 𝑘 be the point in the line containing Log 𝑥 𝑘
( 𝑦 𝑘 − 1 ) and 0 =
Log 𝑥 𝑘 (
𝑥 𝑘 ) ∈
𝑇 𝑥 𝑘 such that the triangle with vertices 0 , Log 𝑥 𝑘 ( 𝑦 𝑘 − 1 ) and 𝑤 and the triangle with vertices 𝑢 , Log 𝑥 𝑘 ( 𝑦 𝑘 − 1 ) and Log 𝑥 𝑘 (¯ 𝑥 * ) are similar triangles, and so
[Equation]
We used the triangle similarity in 5 and in 6 we used the deﬁnition of 𝑤 as a convex combination of Log 𝑥 𝑘 (¯ 𝑥 * ) and Log 𝑥 𝑘 ( 𝑦 𝑘 − 1 ) .
It is enough to show 𝑢 = 𝜉𝑧
𝑥 𝑘 𝑘 − 1 as in such a case what we applied in 2 is equivalent to the equality (5) above.
By the deﬁnition of 𝑥 𝑘 , we have 7 below and by triangle similarity we have 8 below:
[Equation]
as desired.
In the next inequality 3 , we used that by (1 /𝜆 )
-strong g-convexity of ℎ 𝑘 and by optimality of 𝑦 * 𝑘 , we have 12 𝜆 𝑑 ( · , 𝑦 * 𝑘 ) 2 ≤ ℎ 𝑘 ( · ) − ℎ 𝑘 ( 𝑦 * 𝑘 ) .
For the second term, we used that for vectors 𝑏, 𝑐 ∈ (cid:82) 𝑛 and 𝜔 ∈ (cid:82) ≥ 0 , we have, by Young’s inequality, ‖ 𝑏 + 𝑤𝑐 ‖ = √︀
‖ 𝑏 ‖ 2 + 𝜔 [Equation] 𝑐 ‖ 2 +
[Equation]
𝜔𝑏, √ 𝜔𝑐 ⟩ ≤ √︀ (1 + 𝜔 )( ‖ 𝑏 ‖ 2 + 𝜔 ‖ 𝑐 ‖ 2 ) .
In 4 we used Young’s inequality.
Before we conclude, we note that
[Equation]
which is implied by the following, where we use the same as in 3 above, the assumption on 𝑦 𝑘 and ∆ 𝑘 ≤ 1 :
[Equation]
Finally [Equation] we can make use of (4) and (6) to obtain the claim in the second part of the lemma:
[Equation]
where 1 holds by the assumption on 𝑦 𝑘 , ∆ 𝑘 ≤ 1 , and (6).
Inequality 2 uses the upper bound on 𝑎 2 𝑘 in Lemma 7, and 3 uses the deﬁnition 𝑣 𝑥𝑘 def = − Log 𝑥 𝑘 ( 𝑦 𝑘 ) /𝜆 .
The following lemma allows to move the regularized lower bounds on the objective without incurring extra geometric penalties.
Lemma 9 (Translating Potentials with no Geometric Penalty)
Using the variables in Algorithm 1, for any ∆ 𝑘 ∈ [0 , 1) , we have
[Equation]
Proof Firstly, by the projection step in Line 12, we have
[Equation]
and
[Equation]
since the operation is a simple Euclidean projection onto the closed ball ¯ 𝐵 (0 , 𝐷 ) in 𝑇 𝑦 𝑘 ℳ .
By the second part of Corollary 13, 𝑦 = 𝑥 𝑘 and 𝑥 =
𝑦 𝑘 − 1 and by (1), we have 1 below
[Equation]
and 2 uses the deﬁnition of 𝑥 𝑘 .
In 3 , we used the deﬁnition of 𝜉 = 4 𝜁 2 𝐷 − 3 that implies 𝜉 − 𝜁 2 𝐷 ≥ 34 ( 𝜉 − 1) and for 𝜏 𝑘 def =
𝑎 𝑘 / ( 𝑎 𝑘 + 𝐴 𝑘 − 1 ) we have that (1+ 𝑎 𝑘 𝐴 𝑘 − 1 ) 2 − 1 ≥ 2 𝑎 𝑘 𝐴 𝑘 − 1 = 2( 1 1 − 𝜏 𝑘 − 1) .
Now, using the second part of Lemma 12 with 𝑦 = 𝑦
𝑘 , 𝑥 = 𝑥 𝑘 𝑧 𝑥 =
− 𝜂 𝑘 𝑣 𝑥𝑘 , 𝑎 𝑥
= 𝑧 𝑥 𝑘 𝑘 − 1 , so that 𝑧 𝑥 + 𝑎 𝑥
=
𝑧 𝑥 𝑘 𝑘 and 𝑧 𝑦 + 𝑎 𝑦
=
𝑧 𝑦
𝑘 𝑘 and
[Equation]
Note that by the choice of parameters and the fact that 𝑟 < 1 , the assumptions in Lemma 12 are satisﬁed.
Thus, the following holds
[Equation]
Hence, combining (7), (8) and (10) we obtain that it is enough to prove
[Equation]
The proof will be ﬁnished if we prove the result for ∆ 𝑘 = 0 .
If we use this last inequality, and the fact that for 𝑟 ≤ 5 / 6 , we have 𝑟 1 − 𝑟 ≤ 3 (︁ 1 1 − [Equation] / 4 − [Equation]
︁ , we deduce that it sufﬁces to show 𝜏 𝑘 ≥ 34 𝑟 to conclude
[Equation]
Such inequality, namely 𝜏 𝑘 ≥ 34 𝑟 , is equivalent to 𝑎 2 𝑘 𝜆 ≥ 3 𝜉 4 ( 𝑎 𝑘 + 𝐴 𝑘 − 1 ) and it holds by Lemma 7.
Finally, we use Proposition 3 to show the ﬁnal convergence rates.
Proof [Theorem 4]Given the inequality (1 − ∆ 𝑘 ) 𝜓 𝑘 ≤ 𝜓 𝑘 − 1 , proven in Proposition 3, we can use 𝜓 𝑘 as a Lyapunov function in order to prove convergence rates of Algorithm 1.
It follows straightforwardly by deﬁnition of 𝜓 𝑘 , in the following way
[Equation]
In 1 , we used ∏︀ 𝑘𝑖 =1 (1 − ∆ 𝑘 ) = ∏︀
𝑘𝑖 =1 𝑖 ( 𝑖 +2)
( 𝑖 +1)
= 𝑘 +2 2( 𝑘 +1) ≥ 12 .
We used smoothness in 2 .
Note 𝜉 − 12 ‖ 𝑦 0 − 𝑧 𝑦 0
‖ 𝑦
= 0 and ‖ 𝑧 𝑦 0
− ¯ 𝑥
* ‖ 2 𝑦
= ¯ 𝑅 2 .
In 3 , we used 𝜉 = 𝑂 (¯ 𝜁 ) and we dropped some terms in the denominator.
This means that the number of iterations is 𝑂 (¯ 𝜁 √︁ 𝐿 ¯ 𝑅 2 𝜀 ) if we want the right hand side to be bounded by 𝜀 .
The algorithm and analysis for strongly g-convex and smooth functions follows directly by applying the reduction in [57, Theorem 7] to Algorithm 1.
We denote this algorithm by RiemaconSC( 𝒳 , 𝑥 0 , 𝑓, 𝜀 ) , where 𝒳 is the feasible set, 𝑥 0 is the initial point, 𝑓 is the function to optimize and 𝜀 is an optional parameter specifying the desired accuracy.
Although the statement of the reduction in this paper assumes a function 𝑓 : ℳ → (cid:82) to be optimized has a global minimizer in an unconstrained problem, the same proof of this theorem works if we have a 𝜇 -strongly g-convex and 𝐿 -smooth function 𝑓 deﬁned over an open set containing a closed geodesically convex set 𝒳 and a minimizer ¯ 𝑥 * of this function restricted to 𝒳 .
The algorithm runs the algorithm for g-convex smooth minimization for Time ns ( 𝐿, 𝜇, 𝑅 ) , where this is deﬁned as the number of iterations needed by the non-strongly g-convex algorithm to reach accuracy 𝜇𝑅 2 / 8 if the initial distance is upper bounded by 𝑅 .
In such a case it guaranteed that the distance to the minimizer is reduced by half, and we restart the algorithm and run it again with the initial distance parameter equal to 𝑅/ 2 , and so on.
This happens 𝑂 (log( 𝜇 ¯ 𝑅 2 𝜀 )) times if we want to achieve accuracy 𝜀 from an initial distance ¯ 𝑅 .
Thus, the total complexity in number of iterations can be bounded by 𝑂 (Time ns ( 𝐿, 𝜇, ¯ 𝑅 ) log( 𝜇 ¯ 𝑅 2 /𝜀 )) , since all initial distances are ≤ ¯ 𝑅 .
In our case, since we optimize over the set 𝒳 with diameter 𝐷 , so it is
Time ns ( 𝐿, 𝜇, 𝑅 ) =
𝑂 (¯ 𝜁 √︀ 𝐿/𝜇 ) , and the total number of iterations is 𝑂 (¯ 𝜁 √︀ 𝐿/𝜇 log( 𝜇 ¯ 𝑅 2 /𝜀 )) .
We note that the reverse reduction in [57] yields extra geometric penalties but this one does not.
Appendix E. Proofs of Proposition 3, Theorem 6, analysis of Algorithm
We start by showing that the iterates of Algorithm 2 stay reasonably bounded, which is crucial in order to bound geometric penalties.
Proposition 10 The iterates 𝑥 𝑘 of Algorithm 2 satisfy 𝑑 ( 𝑥 𝑘 , 𝑥 * ) ≤ 2 𝑅 .
Proof
We ﬁrst show that the optimizer 𝑥 * 𝑘 in the ball 𝒳 𝑘 is no farther than the center of 𝒳 𝑘 to 𝑥 * , that is, 𝑑 ( 𝑥 * 𝑘 , 𝑥 * ) ≤ 𝑑 ( 𝑥 𝑘 − 1 , 𝑥 * ) .
We assume 𝑥 * is not in the ball because otherwise the property holds trivially.
The geodesic segment joining 𝑥
* 𝑘 and 𝑥 * does not contain any other point of the ball, since otherwise by strong convexity we would have that the function value of one such point would be lower than 𝑓 ( 𝑥 * 𝑘 ) .
This fact implies that the angle between Log 𝑥 * 𝑘 ( 𝑥 * ) and Log 𝑥
* 𝑘 ( 𝑥 𝑘 − 1 ) is obtuse, and so 1 holds below and by using Corollary 14 we conclude 𝑑 ( 𝑥 * 𝑘 , 𝑥 * ) ≤ 𝑑 ( 𝑥 𝑘 − 1 , 𝑥 * ) :
[Equation]
If instead of optimizing exactly in the ball we obtain a close approximation, the iterates will not get very far from 𝑥 * .
Indeed, by 𝜇 -strong convexity, if 𝑥 𝑘 is an 𝜀 ′ -minimizer of 𝑓 in 𝒳 𝑘 , we have that 𝑑 [Equation] 𝑅𝑇 , where we used the deﬁnition of 𝜀 [Equation]
= min { 𝐷𝜀 8 𝑅 [Equation] 𝜇𝑅 2 2 𝑇 2 } in the last inequality.
Consequently, applying the non-expansiveness and this last inequality recursively, we obtain
[Equation]
Before we prove Theorem 6, let’s discuss about the initialization of 𝐷 .
As we explain in
Appendix E.1, we can apply the subroutine in [26, Section 6] for any value of 𝐷 that satisﬁes (notice 𝐷 is twice the radius of the ball): [Equation]
[Equation]
If 𝐷 = 2 𝑅 satisﬁes the inequality, then the algorithm uses this value.
If it is not satisﬁed, then for any value 𝐷 ≥ 0 that satisﬁes the inequality it must be 𝐷 < 2 𝑅 , so we assume that this inequality holds for the rest of the argument.
Indeed, it is a consequence of the function 𝑥 2 coth( 𝑥 ) being monotonously increasing for 𝑥 ≥
and that given the deﬁnition of 𝜁 𝐷 = 𝐷 √︀ | 𝜅 min | coth( 𝐷 √︀ | 𝜅 min | ) , we have that inequality (11) is equivalent to 𝐷 2 | 𝜅 min | coth( 𝐷 √︀ | 𝜅 min | ) ≤ (46 𝑅 √︀ | 𝜅 min | ) − 1 .
In this case, the larger 𝐷 is, the faster the algorithm runs.
So one could solve the 1 -dimensional problem 𝐷 =
(46 𝑅 | 𝜅 min | 𝜁 𝐷 ) − 1 on 𝐷 in order to obtain the best guarantee.
On the other hand, we can provide the simple bound on this 1 -dimensional problem 𝐷 = 1 / (70 𝑅 | 𝜅 min | ) which would only lose a constant in the ﬁnal convergence rates.
We show now how this is indeed a bound.
Let 𝑥 be 𝐷 √︀ | 𝜅 min | , for some 𝐷 satisfying inequality (11) and let 𝑆 be the set of all such 𝑥 ≥ 0 .
Because we want 𝑥 2 coth( 𝑥 ) ≤ (46 𝑅 √︀ | 𝜅 min | ) − 1 and the right hand side is upper bounded by ≤ 1 / (23 𝑥 ) , then by monotonicity it must be 𝑆 ⊂ [0 , 1 / 4] .
It holds that for this interval the fourth derivative of 𝑥 2 coth( 𝑥 ) ≤ 0 , which along with its third order Taylor expansion yields 1 below, so the points satisfying 3 below are in 𝑆 and we can use 𝐷 = [Equation] √ | 𝜅 min | =
[Equation]
70 𝑅 | 𝜅 min [Equation] [Equation] 𝑅 | 𝜅 min | as our simple-to-compute bound:
[Equation]
where in 2 we used 𝑥 < 1 for all 𝑥 ∈ 𝑆 .
Now, we can proceed to prove the theorem.
Proof [Theorem 6]
If 𝐷 = 2 𝑅 , which is the case in which the condition in Line 1 of Algorithm 2 is satisﬁed, then we just need to call Algorithm 1 once in the corresponding ball ¯ 𝐵 ( 𝑥 0 , 𝑅 ) and we obtain rates ̃︀ 𝑂
( 𝜁 2 √︁ 𝐿𝜇 ) .
So from now on we assume 𝐷 < 2 𝑅 .
Let 𝑇 =
⌈ 4 𝑅𝐷 ln( 𝐿𝑅 2 𝜀 ) ⌉ and let 𝜀 ′ =
min { 𝐷𝜀 8 𝑅 , 𝜇𝑅 2 2 𝑇 2 } .
Since every time we call Algorithm 1 we do it over a ball of diameter 𝐷 , we still use the notation ¯ 𝜁 def =
𝜁 𝐷 to refer to the geometric constant associated to the sets 𝒳 𝑘 , for every 𝑘 ≥ 1 .
Recall that we use 𝜁 def = 𝜁 𝑅 = 𝑅 √︀ | 𝜅 min | coth( 𝑅 √︀ | 𝜅 min | ) ∈
[ 𝑅 √︀ | 𝜅 min | , 𝑅 √︀ | 𝜅 min | + 1] .
By deﬁnition, it is 𝐷 ≤ (46 𝑅 | 𝜅 min | ¯ 𝜁 ) − 1 .
Using 2 𝑅 > 𝐷 and
¯ 𝜁 ∈
[ 𝐷 √︀ | 𝜅 min | , 𝐷 √︀ | 𝜅 min | + 1] , we conclude 𝐷 ≤ 1 / 3 √ 46 √︀ | 𝜅 min | ≤ 1 / √︀ | 𝜅 min | and ¯ 𝜁 ≤ 𝐷 √︀ | 𝜅 min | + 1 ≤ 2 .
Since ¯ 𝜁 = 𝑂 (1) , the subroutine in Line 8 takes ̃︀ 𝑂 (1) gradient oracle calls by the analysis in Appendix E.1 and thus, Lineof Algorithm 2 takes ̃︀ 𝑂 ( √︁ 𝐿𝜇 log( 1 𝜀 ′ ))
gradient oracle calls to optimize in the ball 𝒳 𝑘 of diameter 𝐷 , for any 𝑘 .
Recall that we denote the global optimizer of 𝑓 by 𝑥 * .
Deﬁne the g-convex combination ˜
[Equation]
Since 𝒳 𝑘 is a ball of radius 𝐷/ 2 and by Proposition 10, it is 𝑑 ( 𝑥 𝑘 , 𝑥 * ) ≤ 2 𝑅 , we have ˜ 𝑥 𝑘 ∈ 𝒳 𝑘 .
Consequently, we have
[Equation]
where 1 is due to the guarantees of the optimization in the ball and the fact that ˜ 𝑥 𝑘 ∈ 𝒳 𝑘 , 2 holds due to g-convexity.
Subtracting 𝑓
( 𝑥 * ) in both sides and rearranging, we obtain
[Equation]
Applying this inequality recursively, we obtain
[Equation]
Above, we used 1 − 𝑥 ≤ exp( − 𝑥 ) , we used smoothness to bound 𝑓 ( 𝑥 0 ) − 𝑓
( 𝑥 * )
≤ 𝐿𝑑 ( 𝑥 0 ,𝑥 * ) 2, we bounded ∑︀ 𝑇 − 1 𝑖 =
(1 − 𝐷 4 𝑅 ) 𝑖 ≤ ∑︀ ∞
𝑖
=0 (1 − 𝐷 4 𝑅 )
𝑖 = 4 𝑅𝐷
and we used the values of 𝜀 ′ and 𝑇 .
Finally, we compute the complexity of this algorithm.
We have 𝑇 iterations taking ̃︀ 𝑂 ( √︁ 𝐿𝜇 ) gradient oracle queries each.
Using the value of 𝑇 and 𝐷 , we obtain that in total, we call the gradient oracle ̃︀ 𝑂 ( 𝑅𝐷 √︁ 𝐿𝜇 )
=
̃︀ 𝑂 ( 𝑅 2 | 𝜅 min | √︁ 𝐿𝜇 ) =
̃︀ 𝑂 ( 𝜁 2 √︁ 𝐿𝜇 ) times, in both of the suggested initializations for 𝐷 , cf.
Algorithm 2.
We conclude by studying the case in which 𝑓 is not strongly convex.
Assume there is a global optimizer 𝑥 * and as before let 𝑅 ≥ 𝑑 ( 𝑥 0 , 𝑥 * ) .
Given 𝜀 > 0 , we use the regularizer 𝑟 ( 𝑥 ) =
[Equation] 𝑥 ) =
𝜀 2 𝑅
𝑑
( 𝑥 0 , 𝑥 ) 2 .
Let 𝑥 * 𝜀 be the minimizer of 𝑓 + 𝑟 .
By [57, Lemma 21], we have 𝑑 ( 𝑥 0 , 𝑥 * 𝜀 ) ≤ 𝑑 ( 𝑥 0 , 𝑥 * ) ≤ 𝑅 .
We run Algorithm 2 on 𝑓 + 𝑟 , which satisﬁes that the iterates of the algorithm and the subroutine go no farther than 2 𝑅 + 𝐷/ 2 < 3 𝑅 from 𝑥 * 𝜀 .
Indeed, the centers of the balls 𝒳 𝑘 are at a distance at most 2 𝑅 from 𝑥 * 𝜀 by Proposition 10 and each ball has radius 𝐷/ 2 .
Recall that we are still optimizing over a Hadamard manifold.
So in ¯ 𝐵 ( 𝑥 * 𝜀 , 3 𝑅 ) , we have that 𝑓 + 𝑟 is ( 𝜀𝑅 2 ) -strongly
convex and its smoothness constant is 𝜁 3 𝑅 ·
𝜀𝑅 2 + 𝐿 = 𝑂
( 𝜁 · 𝜀𝑅 2 + 𝐿 ) , by Fact 2.
Hence, the algorithm ﬁnds an 𝜀/ 2 minimizer 𝑥 𝑇 ′ of 𝑓 + 𝑟 after 𝑇 ′
= ̃︀ 𝑂 ( 𝜁 2 √︁ 𝜁 + 𝐿𝑅 2 𝜀 ) queries to the gradient oracle.
By deﬁnition, it is 𝑑 ( 𝑥 0 , 𝑥 * ) ≤ 𝑅
so 𝑟 ( 𝑥 * ) ≤ 𝜀 2 𝑅 2 · 𝑅 2 =
𝜀 2 and thus 𝑥 𝑇 ′ is an 𝜀 -minimizer of 𝑓 :
[Equation]
E.1. Details of the subroutine chosen by Algorithm 2 for Line 8 of Algorithm
Given a constant 𝐹 such that ‖∇ R ‖ ≤ 𝐹 , in their Proposition 6.1, Criscitiello and Boumal [26] argue that given an 𝐿 ′ -smooth and 𝜇 ′ -strongly
g-convex function in a ball of radius 𝑟 ≤ min {
[Equation]
| [Equation] 𝐹 } 2 , the retraction of the function in the ball to the Euclidean space ˆ ℎ ( · ) def = ℎ ∘ Exp 𝑥 𝑘 ( · ) is strongly convex and smooth with condition number 3 𝐿 ′ 𝜇 ′ .
Here, 1 𝐹 is interpreted as + ∞ .
They assume that the global minimizer is in this ball, but this fact is only used in order to use 𝐿 ′ -smoothness to bound the Lipschitz constant of the function by 2 𝑟𝐿 ′ .
In our case, the global optimizer is at a distance at most 3 𝑅 from any point in any of our balls 𝒳 𝑘 , as argued in the previous section.
However, we can bound the Lipschitz constant by other means.
The functions we will apply this subroutine to have the form ℎ ( 𝑦 ) def = 𝑓 ( 𝑦 )
+ 1 𝜆 𝑑 ( 𝑥, 𝑦 ) 2 , where 𝑥 is a point such that 𝑑 ( 𝑥, 𝑦 ) ≤ 2 𝐷 for all 𝑦 ∈ 𝒳 𝑘 , cf.
Line 8 in Algorithm 1 and (1).
Here 𝐷 = 2 𝑟 is the diameter of 𝒳 𝑘 .
Using the value of 𝜆 , we have that the smoothness of 𝑔 : 𝑦 ↦→ 1 𝜆 𝑑 ( 𝑥, 𝑦 ) 2 in the ball 𝒳 𝑘 is 𝐿 and the global minimizer of this function is at most a distance 2 𝐷 = 4 𝑟 .
So we can estimate the Lipschitz constant of such an ℎ as
[Equation]
where the last inequality uses 𝑟 ≤ 𝑅 which holds by construction of Algorithm 2.
Now, it is enough to satisfy the following inequality in Proposition 6.1 in [26] in order to have that the Euclidean pulled-back function has condition number of the same order as ℎ , which is 𝑂 (¯ 𝜁 ) for 𝒳 𝑘 :
[Equation]
Since 𝑟 ≤ 𝑅 , 𝜁 2 𝐷 ≤ 2 𝜁 𝐷 , 𝐿 ′ = 2 𝐿 and 𝜇 ≥ 0 , it is enough to have 23 𝐿𝑟𝑅 | 𝜅 min | ≤ 𝐿/ (4 𝜁 𝐷 ) .
Note that in Algorithm 2, we ensure 𝑟 ≤ (92 𝑅 | 𝜅 min | 𝜁 2 𝑟 ) − 1 which satisﬁes the previous inequality and also the initial requirement in Proposition 6.1 in [26].
After this result, we can use Euclidean machinery on ˆ ℎ : Log 𝑥 𝑘 − 1 ( 𝒳 𝑘 ) → (cid:82) , namely AGD [61] with a warm start in order to satisfy the condition in Line 8 of Algorithm 1.
The algorithm requires projecting into the feasible set, and we note that in our case it is a Euclidean ball so the operation is very simple.
Indeed, let ˆ 𝒳 𝑘 def =
Log 𝑥 𝑘 − 1 ( 𝒳 𝑘 ) and let ˆ 𝑥 def = Log 𝑥 𝑘 − 1 ( 𝑥 ) , where 𝑥 is the center of the prox deﬁning 𝑔 above.
By [54, Proposition 15] we have that ˆ 𝑥 ′ def
= Π ^ 𝒳
[Equation]
[Equation]
))) is a point that satisﬁes
[Equation]
where ˆ 𝑦 * ℎ def = arg min ^ 𝑦 ∈ ^ 𝒳
𝑘 ˆ ℎ ( 𝑦 ) is the minimizer of ˆ ℎ , that is, the exact prox.
By [61], the convergence rate of AGD with ˆ 𝑥 ′ as initial point is 𝑂 ( √︁ 𝐿 ′ 𝜇 ′ log( ^ ℎ (^ 𝑥 ′ )
− ℎ (^ 𝑦 * ℎ ) ^ 𝜀 )) , where ˆ 𝜀 def = ∆ 𝑘 ′
‖ ˆ 𝑦 * ℎ − ˆ 𝑥
‖ 2 / (78 𝜆 ) is the accuracy we will require, which is less than the accuracy required by Algorithm 1:
∆ 𝑘 ′ 𝑑
(ˆ 𝑥, ˆ 𝑦 * ℎ ) 2 / (78 𝜆 ) .
Here 𝑘 ′ is the internal counter for Algorithm 1 and we used the reasoning above yielding that the condition number of ˆ ℎ is 𝑂 ( 𝐿 ′ 𝜇 ′ )
=
𝑂 (¯ 𝜁 ) .
Using (12), we conclude that it is enough to run AGD for 𝑂 (¯ 𝜁 12 log( 78 𝜆𝐿 ′ 2Δ 𝑘 ′ ))
=
̃︀ 𝑂 (¯ 𝜁 12 ) gradient oracle queries.
Remark 11 We can make Algorithm 2 work under a weaker assumption than Assumption 5 after a minor modiﬁcation on the algorithm.
Because the algorithm in [26] can work with bounded ‖∇ R ‖ ≤ 𝐹 for a constant 𝐹 , we can use it as a subroutine in this more generic case.
In such a case, the diameter of the balls 𝒳 𝑘 must be 𝐷 ≤ | 𝜅 min | 2 𝐹 , and it is enough to change the condition in Line 1 to 2 𝑅 ≤ min { (46 𝑅 | 𝜅 min | 𝜁 2 𝑅 ) − 1 , | 𝜅 min | / (2 𝐹 ) } and if this condition is not satisﬁed, then after computing 𝐷 in Line 2, we further update 𝐷 ← min { 𝐷, | 𝜅 min | / (2 𝐹 ) } .
In this way, the condition is satisﬁed and the geometric penalty is ̃︀ 𝑂 ( 𝑅𝐷 )
=
̃︀ 𝑂 ( 𝜁 2 + 𝑅𝐹 | 𝜅 min | ) instead of ̃︀ 𝑂 ( 𝜁 2 ) .
Appendix F. Geometric lemmas
In this section, we state and prove Lemma 16, which is used in the proof of Theorem 4 to show that the lower bound given by 𝑓 ( 𝑦 * 𝑘 ) + ⟨ ˜ 𝑣 𝑦𝑘 , 𝑥 − 𝑦
* 𝑘 ⟩ that is afﬁne if pulled back to 𝑇 𝑦
* 𝑘 can be bounded by another function, that is afﬁne if pulled back to 𝑇 𝑥 𝑘 .
We also include and prove, with some generalizations, some known Riemannian inequalities that are used in Riemannian optimization methods and that we also use.
The second part of the following lemma appeared in [50].
Similarly with the second part of the corollary that follows.
In this section, unless otherwise speciﬁed, ℳ is an 𝑛 -dimensional Riemannian manifold of bounded sectional curvature.
Lemma 12 Let 𝑥, 𝑦, 𝑝 ∈ ℳ be the vertices of a uniquely geodesic triangle 𝒯 of diameter 𝐷 , and let 𝑧 𝑥 ∈ 𝑇 𝑥 ℳ , 𝑧 𝑦 def =
Γ 𝑦𝑥 ( 𝑧 𝑥 ) + Log 𝑦 ( 𝑥 ) , such that 𝑦 =
Exp 𝑥 ( 𝑟𝑧 𝑥 ) for some 𝑟 ∈ [0 , 1) .
If we take vectors 𝑎 𝑦 ∈ 𝑇 𝑦 ℳ , 𝑎 𝑥 def =
Γ 𝑥𝑦 ( 𝑎 𝑦 ) ∈
𝑇 𝑥 ℳ , then we have the following, for all 𝜉 ≥ 𝜁 𝐷 :
[Equation]
and
[Equation]
Proof Let 𝛾 be the unique geodesic in 𝒯 such that 𝛾 (0)
=
𝑥 and 𝛾 ( 𝑟 )
=
𝑦 .
We have 𝛾 ′ (0)
=
𝑧 𝑥 .
Along 𝛾 , we deﬁne the vector ﬁeld 𝑉 ( 𝑡 ) =
Γ 𝑡 0 ( 𝛾 )( 𝑧 𝑥 − 𝑡𝛾 ′ (0)) .
Then, it is 𝑉 ′ ( 𝑡 ) = − 𝛾 ′ ( 𝑡 ) , and ‖ 𝑉 ( 𝑡 )
‖ =
‖ 𝑎 + (1 − 𝑡 )
𝑧 𝑥 ‖ .
We will make use of the potential 𝑤 : [0 , 𝑟 ] → (cid:82) deﬁned as 𝑤 ( 𝑡 ) =
‖ Log 𝛾 ( 𝑡 ) ( 𝑥 ) − 𝑉 ( 𝑡 ) ‖ 2 .
We can compute
[Equation]
Now, we bound the ﬁrst summand.
We use that for the function Φ 𝑝 ( 𝑥 ) =
12 𝑑 ( 𝑥, 𝑝 ) 2 it holds, for every 𝜉 ≥ 𝜁 𝐷 :
[Equation]
due to Fact 2.
So for 𝛽 ∈ {− 1 , 1 } we obtain the following bound:
[Equation]
Gauss lemma is used in the last summand of 1 .
Now, if 𝛽 = − [Equation] we have −
[Equation]
𝐷
[Equation]
[Equation]
On the other hand
[Equation]
analogously
[Equation]
if 𝛽 =
[Equation]
we have
[Equation]
where 1 is Young’s inequality 2 𝑐𝑑 ≤ 𝑐
+ 𝑑 2 .
Combining (13), (14) [Equation] [Equation] we obtain −
[Equation]
𝛿 𝐷
[Equation]
Integrating between 0 and 𝑟 < 1 , it results in
[Equation]
Using the bound − log(1 − 𝑟 ) ≤ 𝑟 1 − 𝑟 for 𝑟 ∈ [0 , 1) and using the values of 𝑤 ( · ) and 𝑉 ( · ) , we obtain the result.
Corollary 13 Let 𝑥, 𝑦, 𝑝 ∈ ℳ be the vertices of a uniquely geodesic triangle of diameter 𝐷 , and let 𝑧 𝑥 ∈ 𝑇 𝑥 ℳ , 𝑧 𝑦 def =
Γ 𝑦𝑥 ( 𝑧 𝑥 ) + Log 𝑦 ( 𝑥 ) , such that 𝑦 =
Exp 𝑥 ( 𝑟𝑧 𝑥 ) for some 𝑟 ∈ [0 , 1) .
Then, the following holds
[Equation]
and
[Equation]
Proof Use Lemma 12 with 𝑎 𝑦 = 0 .
Note that this corollary allows 𝑟 = 1 as well.
We obtain this result, by continuity, by taking a limit when 𝑟 → 1 .
The following is a lemma that is already known and is used extensively in Riemannian ﬁrst-order optimization.
It turns out it is a special case of Corollary 13.
Corollary 14 (Cosine-Law Inequalities)
For the vertices 𝑥, 𝑦, 𝑝 ∈ ℳ of a uniquely geodesic triangle of diameter 𝐷 , we have
[Equation]
and
[Equation]
Proof This is Corollary 13 for 𝑟 = 1 .
Indeed, given 𝑦 ∈ 𝒯 we can use Corollary 13 with 𝑧 𝑥 =
Log 𝑥 ( 𝑦 ) .
Note that in such a case we have ‖ 𝑧 𝑥
‖ = 𝑑 ( 𝑥, 𝑦 ) and 𝑧 𝑦 = 0 .
Using ‖ Log 𝑦 ( 𝑝 ) ‖
= 𝑑 ( 𝑦, 𝑝 ) and
[Equation]
we obtain the result.
Remark 15 Actually, in Hadamard manifolds, if we substitute the constants 𝛿 𝐷 and 𝜁 𝐷 in the previous Corollary 14 by the tighter constants 𝛿 𝑑 ( 𝑝,𝑥 ) and 𝜁 𝑑 ( 𝑝,𝑥 ) , the result also holds.
See [81].
We now proceed to prove a lemma that intuitively says that solving the exact proximal point problem can be used to lower bound 𝑓 .
One should think about the following lemma as being applied to 𝑦 ← 𝑦 * 𝑘 , 𝑥 ← 𝑥 𝑘 .
Compare the result of the following lemma with the Euclidean equality ⟨ 𝑔, 𝑝 − 𝑦 ⟩ = ⟨ 𝑔, 𝑝 − 𝑥 ⟩ + ‖ 𝑔 ‖ 2 , for 𝑔 = 𝑥 − 𝑦 and 𝑥, 𝑦, 𝑝 ∈ (cid:82) 𝑛 .
Lemma 16 Let 𝑥, 𝑦, 𝑝 ∈ ℳ be the vertices of a uniquely geodesic triangle of diameter 𝐷 .
Deﬁne the vectors 𝑔 def = Log 𝑦 ( 𝑥 )
∈ 𝑇 𝑦 ℳ and 𝑔 𝑥 = Γ 𝑥𝑦 ( 𝑔 ) = − Log 𝑥 ( 𝑦 ) ∈
𝑇 𝑥 ℳ .
Then we have
[Equation]
and
[Equation]
Proof [Lemma 16]Using the deﬁnition of 𝑔 , we have 1 below, by the ﬁrst part of Corollary 14:
[Equation]
and in 2 we used Corollary 14 again but with a different choice of vertices so we have 𝑑 ( 𝑦,𝑝 ) {+
+}
[Equation] 𝛿 𝐷 2 ‖ 𝑔 𝑥 [Equation] 𝑑
[Equation]
The proof of the second part is analogous: using the deﬁnition of 𝑔 , we have 1 below, by the second part of Corollary 14:
[Equation]
and in 2 we used Corollary 14 again but with a different choice of vertices so we have 𝑑 ( 𝑦,𝑝 ) [-
-]2≤ 𝜁
[Equation]