Parameter Free Dual Averaging: Optimizing Lipschitz Functions in a Single Pass
Abstract
Both gradient descent and dual averaging for convex Lipschitz functions have convergence rates that are highly dependent on the choice of learning rate.
Even when the Lipschitz constant is known, setting the learning rate to achieve the optimal convergence rate requires knowing the distance from the initial point to the solution set D .
A number of approaches are known that relax this requirement, but they either require line searches, restarting (hyper-parameter grid search) or do not derive from the gradient descent or dual averaging frameworks (coin-betting).
In this work we describe a single pass method, with no back-tracking or line searches, derived from dual averaging, which does not require knowledge of D yet asymptotically achieves the optimal rate of convergence for the complexity class of Convex Lipschitz functions
1. Introduction
We consider the class of unconstrained convex Lipschitz functions with Lipschitz constant G .
Starting at a point x 0 , at each step we may query a subgradient g k and function value f ( x k ) at a query point x k .
Let x
∗
be any minimizer of f , and denote f ∗ =
f
( x ∗ ) .
For this class, classical convergence results for sub-gradient descent:
[Equation]
with a ﬁxed step size require knowledge of the distance to solution term D = (cid:107) x 0 − x ∗ (cid:107) in order to achieve the optimal rate of convergence.
Using a step size:
[Equation]
the average iterate ˆ x n converges in terms of function value at an inverse-sqrt rate:
[Equation]
This rate is worse case optimal for this complexity class [6].
Knowledge of the constant G can be removed by using AdaGrad [3] step sizes:
[Equation]
Under the mild assumption that loose lower and upper bounds are known on D , just doing a hyperparameter grid search on a log spaced scale from d 0 to d max gives a rate:
[Equation]
which is the approach taken in practice in most applications.
In this work, we describe a modiﬁcation of dual averaging that achieves the optimal rate, for sufﬁciently large n , by maintaining and updating a lower bound on D, which is then used as part of the step size.
Using this lower bound on
D is provably sufﬁcient to achieve the optimal rate of convergence.
The method we describe is "parameter-free" according to the standard usage of the term [7], as it requires knowledge of G but not D.
2. Related Work
There are a number of approaches to optimization of Lipschitz functions that achieve independence of problem parameters, we review the major classes of approaches below.
2.1. Polyak step size
We can trade the requirement of knowledge of D to knowledge of f ∗ , by using the Polyak step size[10]:
[Equation]
This gives the optimal rate of convergence without any additional log factors.
Using estimates or approximations of f ∗ tend to result in unstable convergence, however a restarting scheme that maintains lower bounds on f ∗ can be shown to converge within a multiplicative log factor of the optimal rate [5].
2.2. Exact line searches
The following method relying on an exact line search also gives the optimal rate, without requiring any knowldge of problem parameters [2, 4]:
[Equation]
Relaxing this exact line search to an approximate line search is non-trivial, and will potentially introduce additional dependencies on problem constants.
2.3. Bisection
Instead of running sub-gradient descent on every grid-point on a log spaced grid from d 0 to d max , we can use more sophisticated techniques to instead run a bisection algorithm on the same grid, giving an improvement of an additional log factor[1]:
[Equation]
This can be further improved by estimating d max , which allows us to replace d max with D in this bound.
2.4. Coin-betting
If we assume knowledge of G but not D , coin betting approaches can be used.
Coin-betting [9] is normally analyzed in the online-convex optimization framework, which is more general then our setting and for that class, coin-betting methods achieve optimal regret among methods without knowledge of D, which is a log factor worse than the best possible regret with knowledge of D [7]:
[Equation]
Using online to batch conversion gives a rate of convergence in function value of
[Equation]
3. Algorithm
The algorithm we propose is Algorithm 1.
It is a modiﬁcation of the AdaGrad step-size applied to weighted dual averaging, also known as linear follow-the-regularized-leader in the online learning community.
The key idea is simple.
At each step, we construct a lower bound ˆ d k on D .
If this bound is less than twice our current best estimate of d k of D , we continue to use d k .
Otherwise, we replace d k in our step size by ˆ d k , and proceed normally.
To construct the lower bound, we use the technique of analyzing a “phantom” point:
[Equation]
which differs from the x k sequence used in the algorithm by using a different ﬁxed step size sequence γ ∗
= γ n +1 / 2 .
By using this point, we are able to gain a additional negative term − 14 γ n +1 (cid:107) s n +1 (cid:107) 2 in our upper bound on the weighted sum of the function values:
[Equation]
Using the fact that (cid:80) {+nk =
 d+} k ( f ( x k ) − f ∗ ) ≥ 0 , we have:
[Equation]
Input: d 0 , [-xs 
-]x
[Equation] s [Equation] =for k = 0 to n do
[Equation] if ˆ d k +1 > 2 d k then
[Equation]
else d[-k s k +
-]
[Equation]
=
d k end if x
[Equation]
[-k ˆ d k +
-]+ [Equation]
=
[Equation] γ[-n +1(cid:107) s n +1 (cid:107) 2 − (cid:80) nk =
-]
[Equation]
[-+
-]+
[-ˆ d k + else d k +
-]
[-= d k end if γ k +1 = (cid:113) G 2 + (cid:80) ki =0 (cid:107) g i (cid:107)x k +1 = x 0 − γ k +
-][Equation]
end for
Return ˆ x n =
[-d k (cid:80) nk =
-][Equation]
which can be rearranged to yield a lower bound on D, involving only known quantities:
[Equation]
This bound is potentially vacuous if (cid:107) s n +1 (cid:107) 2 is small in comparison to (cid:80) nk =
γ k d 2 k (cid:107) g k (cid:107) 2 , however we are able to show that the algorithm converges rapidly in that case, and so further increases of d k are not needed.
Theorem 1 For a convex G -Lipschitz function f .
Algorithm 1 returns a point ˆ x n such that:
[Equation]
as n → ∞ , where D = (cid:107) x 0 − x ∗ (cid:107) for any x ∗ in the set of minimizers of f , as long as d 0 ≤ D .
We provide a proof in the Appendix.
The above result is asymptotic due to the potential of worst-case functions.
For any ﬁxed choice of n , a function could be constructed such that Algorithm 1 run for n steps has a dependence on d 0 .
The next theorem shows that even in the worst case this dependence only results in a log 2 ( D/d 0 ) worst rate of convergence, signiﬁcantly better than the D/d 0 worse rate that sub-gradient-descent incurs:
Theorem 2 Consider Algorithm 1 run for n steps, if we return the point ˆ x t [Equation] where t is chosen to be: [Equation]
Then:
[Equation] [Table] 4. Discussion Our analysis applies to a very restricted problem setting of convex Lipschitz functions.
In Carmon and Hinder [1], an approach for the same setting is extended to the stochastic setting in high probability.
The same extension may also be applicable here.
Our approach has an undesirable dependence on the constant G , as it appears in the denominator of the step size.
This dependence is typical for dual averaging methods, and approaches have been developed to remove the dependence [8].
In general, this dependence is mild in practice, as the G term only has a signiﬁcant effect at the early stages of optimization, where it might dominate the sum in the denominator of the step size:
[Equation]
Our algorithm requires an initial lower bound d 0 on D.
The value of d 0 does not appear in the convergence rate bound as it’s contribution goes to zero as k → ∞ , and hence is suppressed when big-
O notation is used.
In practice very small values can be used, as d k will grow exponentially with k when d 0 is extremely small.
5. Conclusion
We have presented a simple approach to achieving parameter free learning of convex Lipshitz functions, by constructing succesively better lower bounds on the key unknown quantity: the distance to solution (cid:107) x 0 − x ∗ (cid:107) .
Our approach for constructing these lower bounds may be of independent interest.
Appendix A. Theory
Let the weighting of the gradient in the s k sum be λ k :
[Equation]
We use the theory technique of a phantom point
[Equation]
using a Lyapunov function:
[Equation] A.1. Lemmas Lemma 3 (From Carmon and Hinder [1])
For any points x 0 , x T , (not necessarily iterates):
[Equation]
Proof We apply a case analysis.
Suppose that (cid:107) x
T − x ∗ (cid:107) ≥ (cid:107) x 0 − x ∗ (cid:107) , then clearly (cid:107) x 0 − x ∗ (cid:107) 2 − (cid:107) x T − x ∗ (cid:107) 2 is negative and thus trivially less than 2 (cid:107) x 0 − x ∗ (cid:107) 2 (cid:107) x 0 − x T (cid:107) .
So next consider the case that (cid:107) x 0 − x ∗ (cid:107) 2 ≥ (cid:107) x
T − x ∗ (cid:107) 2 .
Then:
[Equation]
+ (cid:107) x
T − x ∗ (cid:107) )
(cid:107) x 0 − x T (cid:107) ≤ 2 (cid:107) x 0 − x ∗ (cid:107) (cid:107)
[Equation]
−
[Equation]
T (cid:107) .
The ﬁrst inequality is an application of the triangle inequality in the form (cid:107) x 0 − x ∗ (cid:107) ≤ (cid:107) x
T − x ∗ (cid:107) + (cid:107) x 0 − x T (cid:107) .
The second inequality uses our case assumption.
Lemma [-
-]4 The inner product γ k λ k (cid:104) g k , s k (cid:105) is a key quantity that occurs in our {+theory, where s k +
+}
= s k + λ k g k for some choice of λ k . We can bound the sum of these inner products over time by considering the following expansion, where γ k is any sequence of weights.
[Equation]
This simpliﬁes when the weighting sequence is ﬂat:
[Equation]
with λ weights:
[Equation]
Proof This is straight-forward to show by induction (it’s a consequence of standard DA proof techniques, where (cid:107) s n (cid:107) 2 is expanded).
[Equation]
Therefore
[Equation]
Telescoping −
[Equation]
A.2. Main Theory
Theorem 5 For Algorithm 1, for all steps k it holds that:
[Equation]
Proof The bound simply relies on convexity:
[Equation]
Theorem 6 Theorem 5 can be telescoped and simpliﬁed to give:
[Equation]
Proof Plugging in the phantom step size:
[Equation]
So:
[Equation]
Then telescoping:
[Equation]
Therefore:
[Equation]
We now apply Lemmas 3 as follows:
[Equation]
and 4 to simplify
[Equation]
Combining gives:
[Equation]
We can further simplify with:
[Equation]
Using the fact that γ k +1 − γ k ≤ 0 and that γ ∗ = γ n +1 / 2 we have:
[Equation]
Theorem 7 The distance to solution error term can be lower bounded as follows
[Equation]
Proof The key idea is that the bound:
[Equation]
gives some indication as to the magnitude of D in the case when the other terms on the right are negative.
To proceed, we use (cid:80) nk =
λ k
( f ( x k ) − f ∗ ) ≥ 0 , giving:
[Equation]
which we can rearrange to:
[Equation]
Therefore:
[Equation]
Theorem 8 The norm of s n +1 is bounded as:
[Equation]
Proof By the deﬁnition of ˆ d n +1 as used in Theorem
[Equation]
we have:
[Equation]
and since 2 d n +1 ≥ ˆ d n +1 ,
[Equation]
So:
[Equation]
This is a quadratic equation in (cid:107) s n +1 (cid:107) that we can solve explicitly.
We have equality when:
[Equation]
where:
[Equation]
So we have equality at the value:
[Equation]
By examination of the cases
[Equation]
we see that that the + case provides an upper bound:
[Equation]
We can further simplify this bound using subadditivity:
[Equation]
therefore:
[Equation]
Proposition 9 (From Duchi et al.
The gradient error term can be bounded as:
[Equation]
and therefore:
[Equation]
Appendix B. Putting it together
Theorem 10 For Algorithm 1:
[Equation]
Proof Consider the key bound:
[Equation]
applying the bound from Theorem 8:
[Equation]
gives:
[Equation]
Now using λ k = d k , plugging in the step {+size, and dropping the − γ n +
+}
[Equation]
s n +
[Equation] 2 term: n
[Equation]
Theorem {+
+} For the point returned by Algorithm 1, as n → ∞ :
[Equation]
Proof Since the sequence d k changes by doubling, and is upper bounded by D , there must exist some ﬁnite ˆ n such that after ˆ n steps, d k no longer increases, i.e. d n = d ˆ n for all n ≥ ˆ n. Suppose that n ≥ 2ˆ n .
Then we have that:
[Equation]
Then:
[Equation]
Then using (cid:80) nk [-=
-]=
[Equation]
(cid:107)
[Equation] γ n [Equation] G 2 ( n + 2) to simplify further we get:
[Equation]
So
[Equation]
We can convert to a bound on the average iterate:
[Equation]
via Jensen’s inequality.
Using (cid:107) g k (cid:107) 2 ≤ G 2 and simplifying gives the result.
Appendix C. Non-asymptotic analysis
Lemma 12 Consider a sequence d 0 , . . .
d N +1 , where for each k , d k +1 =
d k or d k +1 ≥ 2 d k .
Let r ≤ log 2 ( d N /d 0 ) be the number of steps for which d k +1 ≥ 2 d k .
Then
[Equation]
Proof We proceed by an inductive argument on r .
In the base case, if r = 0 then the result follows immediately:
[Equation]
So assume that r > 0 .
First we show that no induction is needed if for all, and we may take n = N, if
[Equation]
Since, in that case we have:
[Equation]
Therefore:
[Equation]
So, instead suppose that there is at least one increase within the range k ≥ (cid:106) N + 1 −
[Equation]
Note that +1 is due to the fact that the above case includes the edge case where an increase occurs exactly at the beginning of the interval.
That implies that there are at most r − 1 increases in the range k ≤ (cid:106) N + 1 −
[Equation]
N +
[Equation]
[Equation]
d N +1 /
[Equation]
(cid:107) , therefore, we can apply induction, assuming by induction that:
[Equation]
Under this inductive hypothesis assumption
[Equation]
we note that:
[Equation]
the last inequality follows from d k +1 ≥ 2 d k , as it implies that:
[Equation]
Putting it all together
[Equation]
we have that:
[Equation]
Theorem 13 Consider Algorithm 1 run for n steps, if we return the point ˆ x t =
[Equation]
where t is chosen to be:
[Equation]
Then:
[Equation]
Proof Consider the bound from Theorem 10:
[Equation]
For the middle term
[Equation]
we can simplify via:
[Equation]
For the third term:
[Equation]
So we have:
[Equation]


[Equation]
Now using Lemma
[Equation]
{+we can return the point ˆ x t and at time t at which min n ≤ N d n +1 (cid:80) nk =
+}
d k is smallest, ensuring that:
[Equation]
Giving us an upper bound:
[Equation]
We can further simplify using Proposition 9:
[Equation]