Fast Convergence of Greedy 2-Coordinate Updates for Optimizing with an Equality Constraint
Abstract
We consider minimizing a smooth function subject to an equality constraint.
We analyze a greedy 2-coordinate update algorithm, and prove that greedy coordinate selection leads to faster convergence than random selection (under a Polyak-Łojasiewicz assumption).
Our simple analysis exploits an equivalence between the greedy 2-coordinate update and equality-constrained steepest descent in the 1-norm.
Unlike previous 2-coordinate analyses, our convergence rate is dimension independent
1. Introduction
Coordinate descent (CD) is an iterative optimization algorithm where on each iteration we perform a gradient descent step on a single variable.
CD methods are appealing because they have a convergence rate similar to gradient descent, but for some common objective functions the iterations have a much lower cost.
Thus, there is substantial interest in using CD as an optimization algorithm for training machine learning models.
Coordinate descent with no constraints : Nesterov [7] considered CD with random choices of the coordinate to update, and proved explicit non-asymptotic linear convergence rates for strongly-convex functions with Lipschitz-continuous gradients.
It was later shown that these linear convergence rates can be achieved under a generalization of strong convexity called the Polyak-Łojsiewicz condition [3].
Further, it was shown that greedy selection of the coordinate to update can lead to faster rates than randomized selection [8].
The faster greedy rates do not depend directly on the dimensionality of the problem, and are a consequence of an equivalence between the greedy coordinate update and the steepest descent update on all coordinates in the 1-norm.
Coordinate descent with separable constraints : CD is commonly used for optimization with separable constraints, in the form of lower and/or upper bounds on each variable.
Nesterov [7] showed that the unconstrained rates of randomized CD can be achieved under these separable constraints using a projected-gradient update of the coordinate.
Richt´arik and Tak´aˇc [9] generalize this result to include a non-smooth but separable term in the objective function, using a proximalgradient update of the coordinate.
These analyses justify using CD in various constrained and non-smooth settings, including least squares regularized by the 1-norm and support vector machines where we regularize the bias term.
Similar to the unconstrained case, Karimireddy et al.
[4] show that several forms of greedy coordinate selection lead to faster convergence rates than random selection for problems with separable constraints and/or separable non-smooth terms.
Coordinate descent with an equality constraint : many problems in machine learning require us to satisfy an equality constraint.
The most common example is that discrete probabilities must sum up to 1.
Another common example is SVMs with an unregularized bias term.
The (non-separable) equality constraint cannot be maintained if we only update one coordinate on each iteration, but it can be maintained if we update 2 variables on each iteration.
Necoara et al.
[5] analyze random selection of the two coordinates to update, while Fang et al.
[2] discuss randomized selection with tighter rates.
The LIBSVM package [1] uses a greedy 2-coordinate update for fitting SVMs without regularizing the bias.
LIBSVM uses greedy coordinate selection since for the SVM problem greedy and random selection have similar iteration costs.
But despite LIBSVM being perhaps the most widely-used CD method of all time, current analyses of greedy 2-coordinate updates [10] do not lead to faster rates than random selection.
Our contribution : we give a new analysis for a particular greedy 2-coordinate update for optimizing a smooth function an equality constraint.
The analysis is based on an equivalence between the greedy update and equality-constrained steepest descent in the 1-norm.
This leads to a dimension-independent analysis of greedy selection showing that it can converge substantially faster than random selection.
2. Optimization with an Equality, Greedy 2-Coordinate Updates, and Proof Outline
We consider the problem of minimizing a twice-differentiable function f subject to a simple linear equality constraint,
[Equation]
to
[Equation]
where n is the number of variables and γ is a constant.
On iteration k the 2-coordinate descent method chooses a coordinate i k and a coordinate j k and updates these two coordinates using
[Equation]
i [Equation] for a scalar δ k (the other coordinates are unchanged).
We write this update for all coordinates as
[Equation]
where d ki = δ k , d kj = − δ k , and d km = 0 for m ̸
= i and m ̸
= j .
If the iterate x k satisfies the constraint, then this update maintains the constraint.
To choose the coordinate to update, the greedy rule chooses the coordinates to maximize the difference in their partial derivatives,
[Equation]
At the solution of the problem we must have partial derivatives being equal, and intuitively this greedy choice updates the coordinates that are furthest above/below the average partial derivative.
We can also derive this choice as the minimizer among a set of 2-coordinate quadratic approximations to the function
[Equation]
for a step size α — see Appendix A.1.
This is a special case of the Gauss-Southwell-q (GS-q) rule of Tseng and Yun [10].
We assume that the gradient of f is Lipschitz continuous, and our analysis will depend on a quantity we call L 2 .
The quantity L 2 bounds the change in the L 2 -norm of the gradient with respect to any two coordinates i and j under a two-coordinate update of any x of the form in (2).
[Equation]
Note that L 2 is less than or equal to the Lipschitz constant of the gradient of f .
Our analysis will focus on the case of δ k = − 12 L ( ∇ i k f ( x k ) − ∇ j k f ( x k )) , resulting in an update to the coordinates of
[Equation]
However, we note that our analysis also applies if we choose δ k to maximally decrease f .
Our analysis relies on the following properties related to the 1-norm:
For vectors d k of the form given above, we have ∥ d k ∥ 21 = 2 ∥ d k ∥ 22 ,
[Equation]
If a twice-differentiable function’s gradient satisfies the 2-coordinate Lipschitz continuity assumption (5) with constant L 2 , then the full gradient is Lipschitz continuous in the 1 -norm with constant L 1 =
L 2 / 2 (see Appendix B).
Applying the 2-coordinate update (6) is an instance of applying steepest descent over all coordinates in the L 1 -norm.
For a function satisfying the proximal-PL inequality, we can measure the proximal-PL inequality in the L 1 -norm.
The next section outlines the latter components, and we use these to give a simple proof for the greedy 2-coordinate algorithm in Section 4.
3. Connections to the 1-Norm
In this section we outline the connection between the greedy update and steepest descent the 1-norm, and then we discuss the proximal-PL condition.
3.1. Steepest Descent in the 1-Norm
Now we show that steepest descent in the L 1 -norm can also lead to sparse update directions.
In particular, we show that steepest descent in the L 1 -norm always admits at least one solution which updates only two coordinates.
Crucially, this implies that the best two-coordinate update makes as much progress in the 1-norm as any full-coordinate update.
Steepest descent in the L 1 -norm, subject to the equality constraint, takes steps in the direction d that minimizes the following model of the objective:
[Equation]
where α is the step size.
This is a convex optimization problem for which strong duality holds.
Introducing a dual variable λ ∈ R , we obtain the Lagrangian
[Equation]
The subdifferential with respect to d and λ yields necessary and sufficient optimality conditions for a steepest descent direction,
[Equation]
for
[Equation]
The second condition is simply feasibility of d , while from the first we obtain,
[Equation]
where element m of sgn ( d ) is 1 if d m is positive, − 1 if d m is negative, and can be any value in [ − 1 , 1] if d m is 0.
The following lemma shows that these conditions are always satisfied by a two-coordinate update.
Lemma 1 Let α > 0 .
Then at least one steepest descent direction with respect to the L 1 -norm has exactly two non-zero coordinates.
That is,
[Equation]
See Appendix A.2 for the proof.
Lemma 1 allows us to relate the progress of a block-coordinate update on just two coordinates to the progress made by a full-coordinate steepest descent step.
This will be a key step in our analysis of the GS-q method.
However, first we introduce the proximal-PL inequality in the 1 -norm, which is our main technique for lower bounding the sub-optimality of an iterate in terms of the squared 1-norm..
3.2. Proximal-PL Inequality in the 1-Norm
The proximal-PL condition was introduced to allow simpler proofs for various constrained and non-smooth optimization problems [3].
The proximal-PL condition is normally defined based on the
L 2 -norm and non-smooth functions.
Below, we define a variant where distances are measured in the
L 1 -norm and we include the equality constraint explicitly.
Definition 2 A function f , that is L 1 -Lipschitz with respect to the 1 -norm and has a summmation constraint on its parameters, satisfies the proximal-PL condition in the 1 -norm if for a positive constants
µ 1 we have
[Equation]
for all x satisfying the equality constraint, where f ∗ is the constrained optimum function value and where
[Equation]
It follows from the equivalence between norms that summation-constrained functions satisfying the proximal-PL condition in the 2-norm will also satisfy the above proximal-PL conition in the 1-norm.
Functions satisfying these conditions include any strongly-convex function f , as well as relaxations of this such as functions of the form f = g
( Ax ) for a strongly-convex g and a matrix A (in this case f may not be strongly-convex) [3].
4. Convergence Result
We now combine the previously-stated results to give a convergence rate for the greedy 2-coordinate method.
Theorem 3 Let f be a twice-differentiable function whose gradient is 2-coordinate-wise Lipschitz (4) and restricted to the set where x T 1 = γ .
If this function satisfies the proximal-PL inequality in the 1-norm (10) for some positive µ 1 , then the 2-coordinate update (6) with the greedy GS-q rule (3) satisfies: [Equation]
[Equation]
Proof Starting from the descent lemma applied to the function restricted to the coordinates i k and j k that we update, we have
[Equation]
f ( x
[Equation]
[Equation]
= f
( x k )
+ min i,j
[Equation]
min d ij
[Equation]
R
[Equation]
| d i
[Equation] (cid:27) (GS-q rule) = f ( x k ) + min i,j (cid:26) min [Equation]
ij
[Equation]
|
[Equation]
j
[Equation]
[Equation]
= f
( x k )
+ min i,j
[Equation]
min
[Equation] | d [Equation] = f
( x k ) + min d | d T 1=
[Equation]
(cid:27) (Lemma 1)
Now subtracting f ∗ from both sides and using the definition of D we get
[Equation]
Applying the inequality recursively completes the proof.
Unlike the convergence rates shown for random coordinate selection [5, 6], this linear convergence rate avoids requiring a direct dependence on the problem dimension.
5. Extension to Bound Constraints
In this work we only considered a single equality constraint.
But in machine learning where we have a single equality constraint, we typically also have bound constraints on the variables.
This includes our motivating problems of optimizing over the probability simplex, or optimizing SVMs with an unregularized bias.
The first versions of LIBSVM used a variation on the Gauss-Southwell-q for the case of bound constraints an a linear equality.
Unfortunately, our proof technique does not directly apply if we add bound constraints.
This is because with bound constraints and an equality constraint the steepest descent direction in the 1-norm may update more than two coordinates.
We are exploring whether ideas like those of Karimireddy et al.
[4] will allow us to use this simple analysis this more-general setting.
Appendix A. Additional proofs A.1. Greedy GS-q Rule for Summation Constraints
For the optimization problem (1), the GS-q rule selects the optimal block b , by solving the following minimization problem:
[Equation]
where d b is the descent direction.
First let us fix b and solve for d b .
Solving for d b .
The Lagrangian of (13) is,
[Equation]
Taking the gradient with respect to d b gives,
[Equation]
Setting the gradient equal to 0 and solving for d b gives,
[Equation]
From our constraint, d b 1 + d b 2 = 0 , we get
[Equation]
Substituting in (14) we get,
[Equation]
That is,
[Equation]
Now, we plug in the optimal d b from (15) in (13) and solve for b .
Solving for b .
After substituting the optimal d b , (13) becomes, arg min
[Equation]
Therefore, q-2-GCD chooses two coordinates b
= { i, j } that are farthest apart.
That is, coordinates with maximum and minimum gradient values of f .
[Equation] [Equation]
A.2. Proof to Lemma
The proof follows by constructing a solution to the steepest descent problem in Eq.
7 which only has two non-zero entries.
Let i = arg max
i ∇ i f ( x ) and j = arg min j ∇ j f ( x ) .
Our proposal solution is d such that d i = − δ, d j = δ for some δ ∈ R and d k,k ̸
= i,j = 0 .
Clearly d satisfies the sum-to-zero constraint required for feasibility.
Now we check that (8) is satisfied by every coordinate of d .
The definition of the 1-norm implies ||
d || 1 = 2 δ , while sgn ( d i ) = − 1 , sgn ( d j ) = 1 and sgn ( d k ) ∈ [ − 1 , 1] follow by construction.
Thus, for d to be a steepest descent direction we must have the following:
[Equation]
Solving for λ in (18) and substituting in (19) we get,
[Equation]
Therefore we get
[Equation]
It remains only to show that (20) is satisfied by d .
Using the value of λ in the constraint yields,
[Equation]
Now, substituting the value for δ in (20) gives
[Equation]
As ∇ k f ( x ) is between ∇ i f ( x ) and ∇ j f ( x ) , we can write it as θ ∇ i f ( x ) + (1 − θ ) ∇ j f ( x ) .
[Equation]
which holds because (1 − 2 θ ) ∈
[ − 1 , 1] .
We have shown that a two-coordinate update d satisfies the sufficient conditions to be a steepest descent direction in the L 1 -norm.
Substituting d back into the expression for steepest descent gives
[Equation]
Since the reverse inequality follows trivially, we deduce that
[Equation]
as claimed.
Appendix B. Relating Lipschitz Constants
Proposition 4 Suppose f is twice differentiable and
[Equation]
for any x such that ⟨ x, 1 ⟩ = a .
Then f satisfies the following inequality:
[Equation]
[Equation]
for any such x and any d such that ⟨ d, 1 ⟩ = 0 .
Proof Consider the optimization problem
[Equation]
This is a convex maximization problem over a polyhedral constraint set; standard results from convex optimization imply that at least one maximizer of (26) occurs at an extreme point of the constraint set
D = { d : ⟨ d, a ⟩ = 0 , ∥ d ∥ 1 ≤ 1 } .
We now show that all extreme points of D contain exactly two non-zero entries.
Let d e be any extreme point of D and suppose by way of contradiction that d e has at least three non-zero entries.
Denote these entries as d 1 , d 2 , d 3 .
Since at least one entry of d e must negative and one must be positive, we may assume without loss of generality that d 1 , d 2 > 0 and d 3 < 0 .
Let ϵ > 0 and define d ′ e =
d e + e 1 ϵ − e 2 ϵ .
For ϵ sufficiently small ϵ it holds that d 1 + ϵ > 0 and d 2 − ϵ > 0 so that
[Equation]
and
[Equation]
Thus, d ′ e ∈ D .
Repeating this argument for d ′′ e = d e
[Equation]
ϵ , we obtain d ′′ e ∈ D and
[Equation]
which contradicts that d e is an extreme point of D .
Thus, every extreme point of D has at exactly two non-zero entries.
Returning to our argument, we see that (26) is maximized at d e , where supp ( d e ) = 2 .
Thus, max d
[Equation]
The result is now easily obtained using a Taylor expansion and the intermediate value theorem.
Proposition
The constant L 1 in (24) is exactly equal to L 2 2 .
Proof Let d ∈ R n such that supp ( d ) = 2 and ⟨ d, 1 ⟩ = 0 .
WLOG, suppose that the two non-zero entries of d are d 1 and d 2 .
Observe that ⟨ d, 1 ⟩ = 0 implies d 1 = − d 2 .
Using these facts, we obtain
[Equation]
Thus, ∥ d ∥ 1 = √ 2 ∥ d ∥ 2 and
[Equation]
which implies L 2 ≤ 2 L 1 .
Similarly, we have
[Equation]
This completes the proof
[Equation]