Non-convex online learning via algorithmic equivalence
Abstract
We study an algorithmic equivalence technique between non-convex gradient descent and convex mirror descent.
We start by looking at a harder problem of regret minimization in online non-convex optimization.
We show that under certain geometric and smoothness conditions, online gradient descent applied to nonconvex functions is an approximation of online mirror descent applied to convex functions under reparameterization.
In continuous time, the gradient flow with this reparameterization was shown to be exactly equivalent to continuous-time mirror descent by Amid and Warmuth [4], but theory for the analogous discrete time algorithms is left as an open problem.
We prove an O ( T 23 ) regret bound for non-convex online gradient descent in this setting, answering this open problem.
Our analysis is based on a new and simple algorithmic equivalence method.
Introduction
Gradient descent is probably the simplest and most popular algorithm in convex optimization, with numerous and extensive studies on its convergence properties.
For non-convex objectives, it is known that gradient descent does not necessarily converge to the global optimum, which is in general NP hard.
Thus recent research focuses on alternate objectives, such as finding first-order stationary points efficiently, or the importance of higher order stationary points, e.g. [19, 22, 1].
However, the study of global convergence of gradient descent for non-convex objectives is increasingly important due to the fact that in practice, gradient descent and its variants can achieve zero error on a highly non-convex loss function of a deep neural network.
To explain the success of modern deep learning it is thus important to understand why gradient descent can converge to a global minimum for some highly non-convex functions.
Several important research directions have stemmed from this motivation, including the study of optimization for deep linear networks [20, 21, 6], non-convex matrix factorization [7], provable convergence for neural networks in the linearization regime of the neural tangent kernel [26, 12, 13, 2, 32, 18, 9], and more.
A recent promising approach, proposed in [4, 3], is to consider reparameterizing mirror descent as gradient descent.
In particular, [4] shows that a continuous-time online mirror descent with a convex loss can be written in an equivalent form of a continuous-time online gradient descent whose loss isn’t necessarily convex after reparameterization, but has significant structure.
[3] further explores this idea, showing that after reparameterization, online gradient descent has the same worst-case regret bound as the exponentiated gradient algorithm.
Both [4] and [3] provide a new prospect on understanding the convergence of non-convex gradient descent.
More recently, [23] provides a more general result, tightly characterizing when gradient flow can be written as mirror flow by introducing the notion of a commuting parameterization.
1.1 The Amid-Warmuth question
The works of [4, 3] are limited in two respects: either they are restricted to the continuous-time gradient-flow setting, which is impractical, or they address specific algorithms (Winnow, Hedge,
EG for linear regression) with a relative entropy divergence.
Amid and Warmuth pose the question of extending the reparameterization idea to the general online convex optimization as well as the conditions where this can be obtained.
Solving the question from [4] would give an answer to realistic scenarios and algorithms, with precise regret and running time bounds, for certain non-convex optimization problems that can be solved to global optimality using gradient descent.
1.2 Statement of results
Our study starts from the more general problem of regret minimization in online convex optimization.
Specifically, we study the reparameterization of online mirror descent (OMD) as online gradient descent (OGD) [30] for general online convex optimization.
We show that under certain geometric and smoothness conditions, a non-convex reparameterized OGD algorithm closely match the original convex OMD algorithm.
Quantitative bounds on this matching allow us to prove that the OGD algorithm after reparameterization achieves an O ( T 23 ) regret bound.
This bound is discrete-time and fairly general, applying broadly to smooth, bounded mirror descent regularizers rather than anything more specific.
We also provide sufficient conditions for non-convex losses such that OGD implicitly has an equivalent mirror descent regularizer and hence will have O ( T 23 ) regret.
Our analysis relies on a simple and new algorithmic equivalence technique, which may be of independent interest.
The key step is to show that the outputs of OMD and reparameterized OGD are very close to each other when initialized from the same point.
Considering the OGD update as a perturbed version of OMD along with the fact that OMD can tolerate bounded (adversarial) perturbation per trial allows us to prove an O ( T 23 ) regret bound for the OGD update.
This answers the open question raised in [4], generalizing from continuous-time gradient flow and specialized algorithms, to general online convex optimization.
The regularization term of MD is also generalized from relative entropy to any strongly convex function.
Further, for discrete-time reparameterization, our results extend to arbitrary Lipschitz convex loss functions, as opposed to custom analysis for specific settings of expert prediction, linear prediction, and linear regression [3].
1.3 Paper structure
Section 2 introduces the setting and sketches out background work in the continuous setting.
Sectionprovides our main result, which is subsequently proven in Section 4.
Section 5 provides analysis in the opposite direction, showing the existence of an equivalent mirror descent regularizer for some non-convex gradient descent problems.
Some analysis is deferred to the Appendix.
1.4 Related works
Gradient descent is the simplest and one of the most popular algorithms in convex optimization.
Extensive studies have been conducted to show the convergence of GD to the global minimizer in both the stochastic optimization setting [24], and the online convex optimization setting [31].
For the non-convex setting, however, finding the global minimum is NP-hard and most work focuses on the convergence to first-order stationary points instead.
The well-known descent lemma guarantees the convergence of non-convex GD when the objective is smooth, and dropping smoothness is possible if other conditions are introduced [10].
Many works consider avoiding saddle points; [22] shows that GD converges to local minimizers a.s.
if all saddle points are strict while [19] proposes perturbed gradient descent to escape saddle points.
It’s also shown that even in the online setting, chasing first-order stationary points is possible [17].
Mirror descent, first introduced by [25], generalizes the gradient descent method in the sense that it can adapt to the ‘geometry’ of the optimization problem [11].
Its analogue in the online setting, online mirror descent also achieves tight regret bounds [27] and without projection is shown to be equivalent to the classical regularized follow-the-leader (RFTL) method with constant step-size.
Though it’s natural to think of mirror descent as changing the geometry of the optimization objective in gradient descent [15], the idea to explain non-convex GD by a corresponding convex MD is explored only recently.
The most relevant works to ours are [4] and [3], which prove the equivalence between continuous time
OMD and OGD after reparameterization, and special discrete-time algorithms when the regularization is the relative entropy, respectively.
Our results extend [4] and [3] to general discrete-time OCO with general regularization.
Preliminaries 2.1 Notation
For a function f : R d → R d , we use the notation J f ( x ) :
R d → R d to denote the Jacobian of f at x .
We use the notaion ⊙ to represents an element-wise product of vectors.
Given a strictly convex function R : R d → R d , we denote the Bregman divergence as
[Equation]
For a convex set K and strictly convex regularizer R , we use Π R K ( x ) :=
arg min y ∈K D R
( y ∥ x ) to denote Bregman projection, with shorthand Π K :
= Π ∥·∥ 2 K for Euclidean projection.
Given a positive-definite matrix M ∈ R d × d , we define the norm ∥ x ∥ M := √ x ⊤ Mx .
We use the notation,
B p := { x ∈ R d : ∥ x ∥ p ≤ 1 } for an ℓ p ball and B + p to denote the intersection of the ℓ p ball and the positive orthant.
2.2 Online convex optimization
We consider the online convex optimization (OCO) problem.
At each round t the player
A chooses x t ∈ K
where K ⊂ R d is some convex domain, then the adversary reveals loss function f t ( x ) and player suffers loss f t ( x t ) .
The goal is to minimize regret:
[Equation]
The player is allowed to get access to the (sub-)gradient ∇ x f t ( x t ) as well.
.3 Reparameterizing continuous-time mirror flow as gradient flow
In the continuous-time setting, we have exact conditions from [4] where the trajectory of mirror-flow exactly coincides with that of gradient descent in a reparameterized space.
Concretely, mirror flow on f : R d → R is defined by the following ordinary differential equation (ODE):
[Equation]
In Theorem 2 of [4], this update is shown to be equivalent to the following gradient flow ODE with x ( t ) =
q ( u ( t )) :
[Equation]
if the mirror descent regularizer R and reparameterization function q satisfy [ ∇ 2 R ( q ( u ))] − 1 =
J q ( u ) J q ( u ) ⊤ .
This relationship between the Hessian of the OMD regularizer and the reparameterization function assures that, up to second order factors in ∥ u − v ∥[-
-]
[Equation]
so the geometry induced by R is approximately transformed into a Euclidean geometry by q − 1 .
Because higher order factors vanish in continuous time, this assures that mirror flow and this reparameterized gradient flow coincide as desired.
2.4 Discretizing the updates
Unfortunately, continuous-time updates cannot be implemented in practice so we must rely on discretization.
Using the forward Euler scheme for discretization of (1), yields the mirror descent algorithm.
After discretization, the exact equivalence of mirror descent and reparameterized gradient descent no longer holds as higher order factors in (3) are relevant.
This motivates the open problem of [4] of finding general conditions under which the discretizations of these continuous time trajectories closely track each other.
We tackle extending these results through the online setting described above.
The Online Mirror Descent (OMD) algorithm has the following update on x t :
[Equation]
where R : K → R is a 1-strongly convex regularization over the domain 4 , D R is the Bregman divergence, and y 1 is initialized to satisfy ∇ R ( y 1 )
= 0 .
Another equivalent interpretation of OMD is
[Equation]
The most important special case is online gradient descent (OGD): x t +1 =
Π K ( x t − η ∇ f t ( x t )) , which is OMD with D R ( x || x t ) = 12 ∥ x − x t ∥ 22 .
In this paper we consider reparameterizing general OMD update as a simple OGD update.
We introduce assumptions of the regularizer, reparameterization, and losses in the sequel.
2.5 Assumptions
Assumption 1.
There exists a convex domain K ′ ⊂ R d and a bijective reparameterization function q : K ′
→ K satisfying [ ∇ 2 R ( x )] − 1 =
J q ( u ) J q ( u ) ⊤ , with x = q ( u ) .
We also make the following smoothness/Lipschitz assumptions:
Assumption 2.
Let G > 1 be a constant.
We assume q is G -Lipschitz, and the 1 - strongly convex regularization R is smooth with its first and third derivatives upper bounded by some constant G .
The first and second derivatives of q − 1 are bounded by G .
Furthermore, assume that for all x ∈ K
D R ( x ∥· ) is G -Lipschitz over K .
Finally, we assume loss functions have bounded gradients and the convex domain has bounded diameter with respect to the Bregman divergence.
Assumption 3.
The loss functions f t have gradients bounded by ∥∇ f t ( x ) ∥ 2 ≤ G F for all x ∈ K .
The diameter of K , sup x,y ∈K D R ( x ∥ y ) ≤ D .
Algorithm
We now provide our main result, a regret bound for Algorithm 2.
We note, Algorithm 1 and
Algorithm 2 are the (projected) forward euler discretizations of (1) and (2).
It’s tempting to directly analyze the regret of OGD with losses ˜ f t ( u ) =
f t ( q ( u )) .
The main barrier in doing so is that ˜ f isn’t necessarily convex.
However, we show that the OMD and OGD updates are in fact O ( η 32 ) -close to each other, therefore the OGD update still suffers only sublinear regret.
1: Input: Initialization x 1 ∈ K , regularizer R .
2: for t = 1 , . . .
, T do 3:
Predict x t 4:
Observe ∇ f t ( x t ) 5:
Update y t
+1 = ( ∇ R ) − 1 ( ∇ R ( x t ) − η ∇ f t ( x t ))
x t +
= Π R K ( y t +1 ) 6: end for
1: Input: Initialization u 1 ∈ K ′ =
q − 1 ( K ) .
2: for t = 1 , . . .
, T do 3:
Predict u t 4:
Observe ∇ ˜ f t ( u t ) = ∇ f t ( q ( u t ))) 5:
Update v t +1 =
u t − η ∇ ˜
f t ( u t ) u t
+
=
Π K ′ ( v t +1 ) 6: end for
Remark 1.
The gradient oracle ∇ ˜
f t can also be calculated from an oracle of ∇ f t if we know the reparameterization function q .
Remark 2.
The reparameterized gradient descent uses Euclidean projection rather than Bregman projection.
This can potentially be an advantage if Euclidean projection can be computed efficiently on q − 1 ( K ) (eg. via method like [29]), while Bregman projection is less efficient on K .
Theorem 3.
Under Assumptions 1-3, by setting η = T − 2 / 3 D 2 / 3 G − 10 / 3 G − 1 F the regret of Algorithm 2 is upper bounded by O ( T 2 / 3 D 1 / 3 G 10 / 3 G F ) .
Proof of the theorem is in Section 4.
3.1 Examples
We now provide a few example settings.
Note that in all three settings, q are such that ˜ f ( u ) =
f ( q ( u )) can be nonconvex for a convex f .
Exponentiated gradient using quadratic reparameterization One of the most popular algorithms for online learning over the filled-in simplex 5 K
= B +1 is the Exponentiated Gradient (EG) [8].
In this case, we show that OGD with quadratic reparameterization q ( u ) =
14 u ⊙ u has vanishing average regret.
We empirically verify that reparameterized GD iterates and EG iterates stay close on a toy problem in Figure 1.
In [3] custom analysis is provided for linear losses that achieves optimal regret, but our approach has a simpler and more general analysis.
If R ( x ) = (cid:80) di
=1 x i log( x i ) then OMD is
EG.
In this case, if x = q ( u ) , then
[Equation]
We note that for this example, Assumptions 2 will not hold as relative entropy is not sufficiently behaved near the boundary of the simplex.
This can be handled by modifying K to be the smoothed simplex where all weights are at least ε for some ε .
This is a fairly standard technique.
ε can be chosen such that the full regret in sublinear (see App. B).
Log barrier with exponential reparameterization Consider the case of a log-barrier regularization R ( x ) = − (cid:80) di =1 log( x i ) with K =
[ ε, 1] d .
It can readily be shown that the box constraint maps to a box constraint.
Consider x = q ( u ) =
exp( u ) , where the exponential is elementwise.
We see
Assumption 1 is satisfied as
[Equation]
Tempered Bregman divergences with power reparameterization A more recently studied family of mirror descent regularizers interpolate between ℓ 2 regularization and negative entropy regularization [5].
Regularization for this family uses link function ∇ R ( x ) = log τ ( x ) = 11 − τ ( x 1 − τ −
1) , where τ is a temperature that can range between 0 and 1 .
As τ approaches 1 , the link function approaches the natural logarithm, and hence the update approaches EG, while τ = 0 corresponds to ℓ 2 regularization.
Such divergences are amenable to reparameterization when K = B + p .
To see this, ignoring constant factors [ ∇ 2 R ( x )] − 1 = x τ for τ ∈ [0 , 1) .
Now if q ( u ) = u 22 − τ where the power is elementwise, then J q ( u ) =
diag ( u τ 2 − τ ) =
diag ( q ( u ) τ/ 2 ) =
diag ( x τ/ 2 ) as desired.
Now, we note that the reparameterization is a power, so B + p would get mapped to the B + 2 p 2 − τ , which is convex as long as p > 1 − τ/ 2 .
3.2 Challenges
In some situations, we do not have a closed form solution to the differential equation described by J q .
For example with the β -hypentropy regularizer of [14], the reparameterization involves q − 1 ( x ) = (cid:82) ( x 2 + β 2 ) − 1 / 4 dx , and this integral does not have a known closed form.
Furthermore, other cases of interest, like von Neumann Entropy regularization for Matrix Multiplicative Weights [28] seem difficult to fit into this framework as Assumption 1 is challenging to use with spectral functions.
Reparameterization analysis
We now move on to the proof of Theorem 3.
The proof idea is to show that the OMD and OGD iterates are close to each other after a single update step starting from the same initial point.
Then we can view the OGD update a perturbed version of OMD, and combine it with the fact that the OMD algorithm can tolerate bounded noise per trial.
We begin with the following key lemma showing that the updates x t +1 and q ( u t +1 ) created by OMD and OGD respectively, are close to each other from the same initial point x t
= q ( u t ) .
To do this, we carefully analyze the errors that occur due to the approximation in (3) starting from a proximal formulation of mirror descent.
We can show that the two updates are solutions to approximately the same strongly-convex objective, hence the solutions must be close.
Lemma 4.
Suppose Assumptions 1-3 hold and x t = q ( u t ) , then we have that ∥ x t +1 − q ( u t +1 )
∥ 2 =
O ( G 4 G 3 / 2 F η 3 / 2 ) .
Proof.
We consider the following forms of Algorithms 1 and 2.
[Equation]
We observe that both objectives we are minimizing can be written as the sum of a linear function plus a strongly convex function.
In particular, if ∥ x − x t ∥ 2 > 2 ηG F , then the objective takes positive value which can’t be minimal because taking x = x t gives 0.
Define K r,x t = K ∩ { x : ∥ x − x t ∥ 2 ≤ r } , then the OMD iterate x t +1 ∈ K t :=
K 2 ηG F ,x t .
Likewise, u t +1 ∈ K ′ t := K ′ 2 ηG F · G,u t .
Using Taylor expansion with Assumption 2, for D R ( x || x t ) with ∥ x − x t ∥ 2 ≤ 2 ηG F , we rewrite it as
[Equation]
where ε ( x ) =
O ( G · G 3 F η 3 ) for x ∈ K t .
And the OMD update becomes
[Equation]
We now use Taylor expansion to relate ∇ ˜ f t ( u t ) ⊤ ( u − u t ) to ∇ f t ( x t ) ⊤ ( x − x t ) .
Let x = q ( u ) for u ∈ K ′ t , hence x ∈ ¯ K t := K 2 ηG F · G 2 ,x t , then we have ∇
[Equation]
⊤ t
[Equation]
where ˜ ε ( x ) =
O ( G 5 G 3 F η 2 ) .
Using the above approximation, we can rewrite the OGD update as
[Equation]
On the other hand, we can rewrite 12 ∥ q − 1 ( x ) − q − 1 ( x t ) ∥ 22 as
[Equation]
Then by Assumption 1, ∇ 2 R ( x t ) =
J − 1 q ( x t )
J − 1 q ( x t ) ⊤ , so we conclude that 1 η B R ( x || x t ) and 12 η ∥ q − 1 ( x ) − q − 1 ( x t ) ∥ 22 are O ( G 8 G 3 F η 2 ) -close.
We can then rewrite the OGD update as
[Equation]
where ¯ ε ( x ) = needed.
Comparing both OMD and OGD updates, the objective functions are only off by an O ( G 8 G 3 F η 2 ) term, therefore the minimizers are also O ( G 4 G 3 / 2 F η 32 ) -close
since the objective functions are both Θ( 1 η ) strongly convex.
Remark 5.
It can be shown that if projection is not required, ∥ x t +1 − q u t +1 ∥ 2 =
O ( η 2 ) in Lemma 4.
Furthermore, if the projection operation that Algorithms 1 and 2 perform are the same, the distance only gets closer.
Such a bound would result in an O ( √ T ) regret bound.
For the case of EG over the simplex, the projection operations in both are the same weight scaling operations, so this could be a part of making O ( √ T ) regret possible, though this still requires a better approach for handling exploding Lipschitz constants (see App. B).
Next, we show that the OMD algorithm is actually robust to noise: with a (potentially non-stochastic) bounded perturbation on the output x t per round, we can still get vanishing regret.
Lemma 6.
Suppose Assumptions 2 and 3 hold and Algorithm A does the following update:
[Equation]
where ∥ r t ∥ 2 ≤ C
The regret of Algorithm A can be upper bounded by
[Equation]
This follows from a slightly modified analysis of OMD from [16], where the effect of the perturbation is bounded due to the Bregman divergence being Lipschitz.
In particular, we end up with the standard regret bound for mirror descent that follows from a telescoping argument with an additional error term
[Equation]
where x Rt is the counterfactual mirror descent iterate with r t = 0 .
Full proof of Lemma 6 can be found in Appendix A. Combining both lemmas, we complete our proof of Theorem 3:
Proof.
Algorithm 2 gives a perturbed version of OMD with perturbation bounded by C
= O ( η 32 )
due to Lemma 4.
Plugging C
=
O ( G 4 G 3 / 2 F η 3 / 2 ) into the bound of Lemma 6, the regret is upper bounded by O ( G 5 G 3 / 2 F √ ηT + Dη ) ignoring the lower order term.
Optimizing this and all constants gives the choice of η = T − 2 / 3 D 2 / 3 G − 10 / 3 G − 1 F and regret bound O
( T 2 / 3 D 1 / 3 G 10 / 3 G F ) .
Implicit OMD reparameterization
We have shown that a general convex OMD can be reparameterized as a (potentially) non-convex
OGD, with vanishing O ( T 23 ) regret.
The other direction from OGD to OMD is even more interesting: given a non-convex OGD, can we show its global convergence by showing the existence of a convex
OMD which corresponds to OGD implicitly?
In other words, given a convex and compact domain K ′ , what do we need to know about the domain
K ′ and the (not necessarily convex) loss ˜ f t to ensure the existence of such equivalence?
Fully characterizing the necessary and sufficient condition seems hard, and we provide here a simple sufficient condition which covers some known scenarios.
Assumption 4.
We assume the following properties about q .
• There exists a function q such that ˜ f t ( u ) can be written as f t ( q ( u )) where f t is convex.
• q is a C 3 -diffeomorphism, and J q ( u ) is diagonal.
• q ( K ′ ) is convex and compact.
We argue that once the conditions above are satisfied, there exists a regularization R and a corresponding OMD with the desired equivalence.
The only thing we need to verify is the existence of a strongly convex regularization R which satisfies Assumption 1.
We first show that J q ( u ) always has non-zero (in fact, with the same sign because of continuity) determinant due to the fact that q is a diffeomorphism. Then R can be reconstructed by integrating twice its Hessians which we compute from the equation in Assumption 1.
R is also strongly convex as the Hessians are strictly PSD.
We note that such conditions may not be tight, but are general, succinct and may cover some interesting examples.
Theorem 7.
Given a convex and compact domain K ′ , and not necessarily convex loss ˜ f t satisfying
Assumption 3.
When Assumption 4 is met, there exists an OMD object with convex loss f t , a convex domain and a strongly convex regularization R satisfying Assumption 1.
As a result, running
Algorithm 2 on loss ˜ f t ( u ) has regret upper bound
˜ O
( T 23 ) .
Proof.
The first two properties are satisfied by the assumption.
We verify the third property by constructing a regularization R , which is strongly convex and satisfies Assumption 1.
The rest follows Theorem 3.
By the fact that q is a diffeomorphism, J q ( u ) is an invertible matrix.
The fact that
J q ( u ) is an invertible matrix implies that J q ( u ) J q ( u ) ⊤ is invertible, and further J q ( u ) J q ( u ) ⊤ ≻ 0 .
Denote H ( u ) =
[ J q ( u ) J q ( u ) ⊤ ] − 1 ≻ 0 which is well defined by the above argument, we want to construct R such that ∇ R 2 ( q ( u ))
=
H ( u ) .
By the assumption that J q ( u ) is diagonal, we can write q ( u ) as
[Equation]
where each q i is a scalar function, such that H ( u ) is diagonal with H ( u ) ii
= 1 / ( q ′ i ( u i )) 2 , which only depends on u i .
We set R ( q ( u )) =
(cid:80) di
=1 R i ( q i ( u i )))
and denote the function
˜ R i ( u i ) =
R i ( q i ( u i )) with variable u i , then by the chain rule we have that
[Equation]
Plugging the first formula into the second one, we eliminate the R
′ i
( x i ) term:
[Equation]
Recall, we want to find R such that R ′′
i ( x i ) =
1 / ( q ′ i ( u i )) 2 .
Plugging it back to the equation above, the problem is reduced to find R i such that it satisfies the ODE
[Equation]
It’s known that there exists a solution to any linear and continuous second-order ODE.
In fact, using the standard variation of constant method, one example solution ˜
R i ′ ( u i ) can be
[Equation]
where c i , C i are constants and C i =
min u ∈K ′
u i .
It’s well-defined because q ′ i ( u i ) always has the same sign.
To verify the correctness, we calculate that
[Equation]
which solves the ODE together with the solution of ˜ R i
′ ( u i ) .
Next, plugging ˜ R
i [Equation]
du and u = q − 1 ( x ) into the expression of R gives a solution of the regularization.
Its strong-convexity is implied by the fact that ∇ 2 R ( x ) =
H ( u ) ≻ 0 and is continuous over a compact domain
, thus there exists a constant c > 0 such that ∇ 2 R ( x ) ≻ cI .
Finally, q being a C 3 -diffeomorphism means that both q and q − 1 are 3 times continuously differentiable.
By the compactness of domains and the continuity of the derivatives, there exists a constant G satisfying Assumption 2.
A similar argument holds for R .
Conclusion
We study the convergence of non-convex gradient descent in this paper, by showing (approximate) algorithmic equivalence between gradient descent and mirror descent in the discrete setting.
We prove that under certain geometric and smoothness conditions, running OGD on non-convex losses obtained by reparameterizing a convex OMD has regret bound O ( T 23 ) , answering an open problem in [4].
Our analysis is based on a new algorithmic equivalence technique, combining the one-step closeness between OMD and the reparameterized OGD and the robustness of OMD.
We further extend our result to the other direction, providing sufficient conditions for a non-convex OGD to have an implicit corresponding OMD and thus converge well.
We leave several questions as future directions.
Is the O ( T 23 ) regret bound improvable to O ( √ T ) in general?
The decay in the bound in this work seems to be caused by differences in projection in the reparameterized space and the original Bregman projection.
Perhaps a different analysis technique can produce optimal bounds.
Can assumptions for reparameterization be relaxed?
For example, it is not clear that Assumption 1 is a necessary condition.
In the implicit mirror descent direction, it may be possible to lift the diagonal J q ( u ) assumption in Theorem 7 with more refined conditions on PDEs.
Perhaps the commuting parameterization conditions from [23] can be adapted from the continuous to the discrete setting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Lemma
Proof.
We define the sequences y
Rt , x
Rt as one step of OMD from our current iterate with learning rate η , i.e
[Equation]
Now, we note that ∇ f t ( x t ))
=
∇ R ( x t
[Equation]
−∇ R ( y Rt +1 ) η and hence, we have
Regret ( A ) ≤
[Equation]
where the second equality follows via the three point lemma and the last inequality follows via projection.
Now, we note that by the bound on ∥ r t ∥ 2 and the Lipschitz condition on D R , we have | D R
( x ∗ || x
Rt +1 ) − D R
( x ∗ || x t +1 )
| =
O ( CG ) .
Combining with a telescoping argument, we have
Regret ( A
[Equation]
≤
[Equation]
The final term is bounded via Cauchy-Schwartz:
[Equation]
Noting that D R ( y
Rt
{++1 || x t ) ≥ 
+}
∥ x t − y
{+Rt +
+}
∥ 22 via the 1 -strong convexity of R , we have
D R ( x t ||
y rt +1 )
≤ 12 η 2 G 2 by Assumption 3 and so
Regret ≤ CTG η
[Equation]
B Regret bound of the EG example
We modify the domain K to be the smoothed simplex with all weights being at least T − ϵ , where ϵ is to be determined later.
The smoothing basically worsens the loss of each step by O ( G F T − ϵ ) , and in all worsens the regret by O ( G F T 1 − ϵ ) .
We need to check the constants in Assumption 2 if they have dependence on T .
The Lipschitzness of q is fine, the 1 − st derivative of R and the Lipschitzness of D R ( x ∥· ) have only logarithmic dependence.
However, the first and second derivatives of q − 1 and the third derivative of R do have norms depending on T , with upper bounds to be T ϵ 2 , T 3 ϵ 2 and T 2 ϵ respectively.
As a result, we can only use G = T 2 ϵ .
Plugging it back to Lemma 4, we have that C
= O ( T 8 ϵ G 32 F η 32 ) .
Combining this with Lemma 6 we have that
[Equation]
whenever η < 1 which is the case.
Balancing the first two terms give the optimal choice of η = D 23 G − 1 F T − 20 ϵ +23 .
Then choosing η = 123 gives an O ( T 2223 ) regret bound.