Fast Convergence of Greedy 2-Coordinate Updates for Optimizing with an Equality Constraint
Abstract
We consider minimizing a smooth function subject to an equality constraint. We analyze a greedy 2-coordinate update algorithm, and prove that greedy coordinate selection leads to faster convergence than random selection (under the Polyak-Łojasiewicz assumption).
Our simple analysis exploits am equivalence between the greedy 2-coordinate update and equality-constrained steepest descent in the L1-norm. Unlike previous 2-coordinate analyses, our convergence rate is dimension independent
1. Introduction
Coordinate descent (CD) is an iterative optimization algorithm where on each iteration we perform a gradient descent step on a single variable.
CD methods are appealing because they have a convergence rate similar to gradient descent, but for some common objective functions the iterations have a much lower cost. Thus, there is substantial interest in using CD as an optimization algorithm for training machine learning models.
Coordinate descent with no constraints : Nesterov [8] considered CD with random choices of the coordinate to update, and proved explicit non-asymptotic linear convergence rates for strongly-convex functions with Lipschitz-continuous gradients.
It was later shown that these linear convergence rates can be achieved under a generalization of strong convexity called the Polyak-Łojsiewicz condition [3].
Further, it was shown that greedy selection of the coordinate to update can lead to faster rates than randomized selection [9]. The faster greedy rates do not depend directly on the dimensionality of the problem, and rely on an equivalence between the greedy coordinate update and the steepest descent update on all coordinates in the L1-norm.
Coordinate descent with separable constraints : CD is commonly used for optimization with separable constraints, in the form of lower and/or upper bounds on each variable.
Nesterov [8] showed that the unconstrained rates of randomized CD can be achieved under these separable constraints using a projected-gradient update of the coordinate. Richt´arik and Tak´aˇc [11] generalize this result to include a non-smooth but separable term in the objective function, using a proximalgradient update of the coordinate.
These analyses justify using CD in various constrained and non-smooth settings, including ℓ 1 -regularized least squares and SVMs where we regularize the bias term. Similar to the unconstrained case, Karimireddy et al.
[4] show that several forms of greedy coordinate selection lead to faster convergence rates than random selection for problems with separable constraints and/or separable non-smooth terms. Coordinate descent with an equality constraint : many problems in machine learning require us to satisfy an equality constraint.
The most common example is that discrete probabilities must sum up to 1.
Another common example is SVMs where we do not regularize the bias term. The (non-separable) equality constraint cannot be maintained if we only update one coordinate on each iteration, but it can be maintained if we update 2 variables on each iteration.
Necoara et al. [5] analyze random selection of the two coordinates to update, while Fang et al.
[2] discuss randomized selection with tighter rates. The popular LIBSVM package [1] uses a greedy 2-coordinate update for fitting SVMs without regularizing the bias.
LIBSVM uses greedy coordinate selection since for the SVM problem greedy and random selection have similar iteration costs.
But despite LIBSVM being perhaps the most widely-used CD method of all time, current analyses of greedy 2-coordinate updates [12] do not lead to faster rates than random selection.
Our contribution : we give a new analysis for a particular greedy 2-coordinate update for optimizing a smooth function an equality constraint.
The analysis is based on an equivalence between the greedy update and equality-constrained steepest descent in the L1-norm.
This leads to a dimension-independent analysis of greedy selection showing that it can converge substantially faster than random selection.
In this work, we study the convergence of 2 -GCD under GS-q block selection strategy ([12]) for problems with a single linear equality constraint.
Specifically, we solve the following minimization problem:
[Equation]
We now introduce our notations and definitions.
We call the indices of a parameter vector x ∈ R n , coordinates .
The set of coordinates is C = { 1 , 2 , ..., n } .
Our algorithm will operate on coordinate blocks x b ⊆ x , where b ⊆ C .
We use B to indicate the set of all coordinate blocks.
The notation d b ∈ R | b | denotes the elements, d corresponding to the coordinate block b and ∇ b f ( x ) ∈ R | b | denotes the gradient of the block.
The linear operator U b : R | b | −→ R n is the mapping, [ U b ( d b )] =
(cid:40)
[ d b ] j , if j ∈ b 0 , otherwise .
A basic property we assume is that f has Lipschitz-continuous gradient.
That is, ∥∇ f
( x + α ) − ∇ f ( x ) ∥ q ≤ L p ∥ α ∥ p .
This is a standard assumption for coordinate descent methods ([6–8, 12]). || . || p and || .
|| q are a pair of dual norms. The definitions for GS-q rule and proximal Polyak- Lojasiewicz inequality are given in the Appendix A.1.
3. Algorithm and Theoretical Results
Let x 0 be a feasible initial iterate and f ( x ) be the function to optimize.
Let d b ∈ R | b | be the descent direction.
We analyze a block coordinate descent algorithm that chooses a coordinate block b k according to the GS-q rule assuming block size | b | = 2 (similar to the early versions of libSVM).
Throughout the paper, we call it q-2-GCD .
Without loss of generality, we will assume that b
= { 1 , 2 } .
Proposition 1 For the optimization problem (1) , the GS-q rule selects the optimal block b =
[ i, j ] , where i = argmax i ∇ i f ( x ) , j = argmin j ∇ j f ( x ) and descends in the direction d b = − 1 L 2 (cid:0) ∇ b f ( x ) − 12 ⟨∇ b f ( x ) , 1 ⟩ 1 (cid:1) .
Proof sketch: For the optimization problem (1) , the GS-q rule selects the optimal block b , by solving the following minimization problem:
[Equation]
where d b is the descent direction.
To prove the Proposition 1, we first fix b and solve for d b and then d b is fixed to be the optimal d b for each block, at which point it is easy to solve for the optimal b .
The full proof to Proposition 1 is produced in the appendix A.2.
The algorithm q-2-GCD can be summarized as follows: Algorithm
GS for block size 2 (
1: At iteration k , choose a block b k
= { i, j } that solves: b k = argmax i,j |∇ i f ( x k ) − ∇ j f ( x k ) | .
2: Set x k +
= x k + U b k
( d b k ) , where d b k = − 1 L 2 (cid:0) ∇ b f ( x k ) − 12 ⟨∇
b f ( x k ) , 1 ⟩ 1 (cid:1) .
3.1. Convergence Analysis
In this section, we prove that q-2-GCD converges linearly for functions satisfying the proximal-PL inequality [3].
The key step here is the observation that the descent direction d b of q-2-GCD is (almost) recovered by computing the equality-constrained steepest descent direction in the 1 − norm.
The challenge in analyzing q-2-GCD stems from the presence of equality constraints in (1).
Without constraints, and assuming PL-inequality of f ( x ) , previous works [10] showed that linear convergence can be derived by using GS-q rule in the progress bound and applying the parameter update, where the progress is measured by a norm.
Now, let us derive the progress bound of q-2-GCD and see if we could do the same.
Proposition 2 The progress of q-2-GCD for the problem in 1 can be measured as f ( x k +1 )
=
f
( x k ) − 14 L 2 max b ( ∇ i f ( x k ) − ∇ j f ( x k )) 2 .
The proof to Proposition (2) is in appendix A.3.
Here, the progress depends on the Lipschitz constant and maximum difference between the partial derivatives and there is no straight-forward way to measure this progress in terms of a norm.
This makes it harder to derive linear convergence using existing methods.
Whereas, when b = 1 , the update of the descent direction d b has no coupling of block coordinate gradients, making it easier to represent the progress as a norm [5, 7].
The core idea of our analysis is deriving the convergence of steepest descent in 1 -norm as a proxy for q-2-GCD, thereby bypassing the complication of having block coordinates in the progress bound.
The idea stems from the observation that the descent direction d of the steepest descent in 1 -norm algorithm is almost equivalent to d b of q-2-GCD as in (14), with zeros for all elements in d except the two updated by the q-2-GCD and those coordinates follow (14).
3.1.1. S TEEPEST SUBSPACE DESCENT IN 1 - NORM
The steepest descent method finds the descent direction that minimizes the function value in every iteration.
That is,
[Equation]
Taking the gradient with respect to d and λ of the Lagrangian of (3),
[Equation]
At a solution we need d T 1 = 0 .
Now from ∇ d L ( d, λ ) we get,
[Equation]
This means that any optimal d must satisfy the above condition.
We will use this as a test to show the similarity between the solutions recovered by steepest descent and q-2-GCD and this will be the basis of our convergence analysis.
3.1.2. S IMILARITY BETWEEN S TEEPEST DESCENT AND Q -2-GCD
In this section we will argue that the optimal solution found by q-2-GCD can also be recovered by steepest descent in 1 -norm
Lemma 3 Consider the problem (1) .
The optimal direction d solved by the steepest descent in 1 − norm algorithm recovers the direction d b solved by q-2-GCD with zeroes in all its coordinates except the 2 coordinates chosen and updated by q-2-GCD.
Precisely,
[Equation]
where, τ = L 1 L 2 for a particular solution of the steepest descent problem.
Proof sketch: We begin by proposing a d and show that this d respects the steepest descent optimality condition, derived in (4) .
Now, one possible solution for d would be where d is a vector of all zeros except 2 coordinates d i and d j , where i = argmax i ∇ i f ( x ) and j = argmin
j ∇ j f ( x ) , the coordinates chosen by q-2-GCD.
We prove that this d can be a solution by showing that (4) is satisfied by every coordinate of d .
The full proof to this lemma is in appendix A.4.
Therefore, we can now analyze the convergence of q-2-GCD by directly analyzing steepest descent in 1 -norm.
We will express our results using Proximal-PL inequality [3] in the 1 -norm.
We have defined Prox-PL inequality in 1 -norm and have shown its relationship with Prox-PL inequality in 2 -norm is in appendix A.1.1.
Proposition 4 The Lipschitz constants L 1 and L 2 of the block gradients with block size 2 of any function f measured in ℓ 1 norm and ℓ 2 norm is related as L 2 = 2 L 1 .
The proof to this proposition follows from the result of Lemma 3.
3.1.3. C ONVERGENCE RATE OF Q -2-GCD
Theorem 5 Assume a function F ( x ) = f
( x ) + g ( x ) , where g ( x ) =
if x T 1 = c and ∞ otherwise, with Lipschitz continuous gradient, satisfies Prox-PL inequality in 1 -norm (11) for some µ 1 > 0 .
Then, the Algorithm q-2-GCD for the minimization problem in (1) obtains the following linear convergence rate:
[Equation]
Proof From the descent lemma of q-2-GCD,
[Equation]
Here, f ( x ) is reformulated as F ( x ) =
f ( x ) + g ( x ) where, g ( x )
= (cid:40)if d T 1 =∞ otherwise . .
From the relationship between ℓ 1 and ℓ 2 norms,
[Equation]
Change of notation: WLOG d SD is written simply as d from this point.
From the definition of GS-q rule and (4) we get,
[Equation]
Applying (Prox-PL Inequality in 1 -norm) in (5),
[Equation]
Therefore, from (Prox-PL Inequality in 1 -norm) and (4) we get, F ( x k +
[Equation]
Subtracting F ∗ on both sides and applying the inequality recursively completes the proof.
This shows the method’s linear convergence to optimal F ∗ . Unlike the rates for random block selection [5, 7], this rate is independent of the problem dimension and significantly faster.
4. Conclusion
In this work we analyzed a greedy block-coordinate descent algorithm for a linearly-constrained objective which we call q-2-GCD.
This has been a very successful method with various applications like SVM (LIBSVM).
We provide tighter convergence rate than the existing works.
The key idea behind our analysis was the heuristic notion that greedy block-selection rules can only improve in performance as the set of coordinate blocks is expanded.
We showed the convergence of q-2-GCD on functions satisfying proximal PL inequality thereby furthering our understanding of greedy methods.
Appendix A. Additional proofs A.1. Definitions
Definition 6 (GS-q rule with one linear equality constraint)
The GS-q rule chooses a block that maximizes progress while satisfying a single linear equality constraint, assuming a quadratic upper bound on the function f , where f is a smooth function with L -Lipschitz continuous gradient. That is,
[Equation]
where,
[Equation]
Definition 7 (Proximal Polyak- Lojasiewicz inequality [3])
A composite function F of the form F ( X ) =
f ( x ) + g ( x ) satisfies Prox-PL inequality with constant µ if
[Equation]
where,
[Equation]
A.1.1. P ROX -PL INEQUALITY IN 1 - NORM
In this section, we define Prox-PL inequality in 1-norm.
We compare the constants µ 1 and µcorresponding to Prox-PL inequality in 1-norm and 2-norm and show that µ 2 n ≤ µ 1 ≤ µ 2 .
Then, we analyze q-2-GCD assuming the function satisfies Prox-PL inequality in 1-norm and we derive convergence that is tighter and independent of dimension.
A composite function F of form F ( X ) =
f ( x ) + g ( x ) satisfies Prox-PL inequality with constant µ 1 when,
[Equation]
(Prox-PL Inequality in 1 -norm)
where,
[Equation]
For the problem as in (1)
[Equation]
the update rule of 2-BCD is given by
[Equation]
where d = U b ( d b ) .
Applying (8) in (7)
[Equation]
we get
[Equation]
We know that d has non-zero values in only 2 coordinates i and j .
Using this (9) becomes,
[Equation]
Following similar proof sketch, the Prox-PL inequality in 2 -norm, introduced by [3], can be derived as,
[Equation]
It is easy to see that, min d
[Equation]
Therefore,
[Equation]
Therefore µ 1 is upper-bounded by µ 2 .
This shows that a bound obtained by assuming Prox-PL in 1 -norm can be least as tight as assuming Prox-PL in 2 -norm. Now, we show that µ 1 ≥ µ 2 n . From the relationship between ℓ 1 and ℓ 2 norms, we know that ∥ d ∥ 1 ≤ √ n ∥ d ∥ 2 . Now, (11) becomes
[Equation] Therefore, [Equation] This implies, µ 1 ≥ µ 2 n .
A.2. Proof to Proposition
For the optimization problem (1), the GS-q rule selects the optimal block b , by solving the following minimization problem:
[Equation]
where d b is the descent direction.
First let us fix b and solve for d b .
Solving for d b .
The Lagrangian of (13) is,
[Equation]
Taking the gradient with respect to d b gives,
[Equation]
Setting the gradient equal to 0 and solving for d b gives,
[Equation]
From our constraint, d b 1 + d b 2 = 0 , we get
[Equation]
Substituting in (14) we get,
[Equation]
That is,
[Equation]
Now, we plug in the optimal d b from (15) in (13) and solve for b .
Solving for b .
After substituting the optimal d b , (13) becomes, argmin
[Equation]
≡ argmin
[Equation]
≡ argmin
[Equation]
≡ argmin
[Equation]
≡
[Equation]
Therefore
[Equation]
q-2-GCD chooses two coordinates b
= { i, j } that are farthest apart.
That is, coordinates with maximum and minimum gradient values of f
[Equation]
A.3. Proof
to Proposition
The block wise descent lemma for q-2-GCD can be written as,
[Equation]
Let b = { i, j } .
We can now incorporate the linear equality constraints on d by substituting d b from (14).
Substituting d b and further simplifying we get,
[Equation]
A.4. Proof to Lemma
One possible solution for d would be where d is a vector of all zeros except 2 coordinates d i and d j .
Let i =
argmax
i ∇ i f ( x ) and j = argmin
j ∇ j f ( x ) .
Let us set d i = − δ, d j = δ for some value δ > 0 and d k,k ̸
= i,j = 0 , so that the linear equality constraint is satisfied by our construction.
We then have || d
|| 1 = 2 δ , sgn ( d i )
= − 1 , sgn ( d j )
= 1 and sgn ( d k ) ∈ [ − 1 , 1] .
We now want to see if (4) is satisfied by every coordinate of d , that is, in order for d to be a solution, we would need
[Equation]
Solving for λ in (19) and substituting in (20) we get,
[Equation]
[Equation]
Therefore we get
[Equation]
Recall from (15) the optimal d obtained by q-2-GCD.
Before we derive an exact relationship between these two methods, we need to verify if (21) is satisfied as well.
Substituting (22) in (21) gives:
[Equation]
Now, substituting (23) in (21) gives
[Equation]
As ∇ k f ( x ) is between ∇ i f ( x ) and ∇ j f ( x ) , we can write it as θ ∇ i f ( x ) + (1 − θ ) ∇ j f ( x ) .
[Equation]
which holds because (1 − 2 θ ) ∈
[ − 1 , 1] .
We now showed that steepest descent in 1 -norm recovers the direction d that is equivalent to using q-2-GCD rule with zeros in all the coordinates of d except the two coordinates precisely chosen and updated by the GS-q rule. Therefore,
[Equation]
Where, τ = L 1 L 2 .