Accelerated Projected Gradient Algorithms for Sparsity Constrained Optimization Problems
Abstract
We consider the projected gradient algorithm for the nonconvex best subset selection problem that minimizes a given empirical loss function under an (cid:96) 0 -norm constraint.
Through decomposing the feasible set of the given sparsity constraint as a ﬁnite union of linear subspaces, we present two acceleration schemes with global convergence guarantees, one by same-space extrapolation and the other by subspace identiﬁcation.
The former fully utilizes the problem structure to greatly accelerate the optimization speed with only negligible additional cost.
The latter leads to a two-stage meta-algorithm that ﬁrst uses classical projected gradient iterations to identify the correct subspace containing an optimal solution, and then switches to a highly-efﬁcient smooth optimization method in the identiﬁed subspace to attain superlinear convergence.
Experiments demonstrate that the proposed accelerated algorithms are magnitudes faster than their non-accelerated counterparts as well as the state of the art.
Introduction
We consider the sparsity-constrained optimization problem in (cid:60) n :
[Equation] [Equation]
where f is convex with L -Lipschitz continuous gradient s ∈ N and A s is the sparsity set given by
[Equation]
[Equation]
where (cid:107) w (cid:107)
denotes the (cid:96)
-norm that indicates the number of nonzero components in w .
We further assume that f is lower-bounded on A s .
A classical problem that ﬁts in the framework of (1) is the best subset selection problem in linear regression [6, 20].
Given a response vector y ∈ (cid:60) m and a design matrix of explanatory variables
X ∈ (cid:60) m × n , traditional linear regression minimizes a least squares (LS) loss
[Equation]
However, due to either high dimensionality in terms of the number of features n or having signiﬁcantly fewer instances m than features n (i.e., m (cid:28) n ), we often seek a linear model that selects only a subset of the explanatory variables that will best predict the outcome y .
Towards this goal, we can solve (1) with f given by (3) to ﬁt the training data while simultaneously selecting the best- s features.
Indeed, such a sparse linear regression problem is fundamental in many scientiﬁc applications, such as high-dimensional statistical learning and signal processing [22].
The loss in (3) can be generalized to the following linear empirical risk to cover various tasks in machine learning beyond regression
[Equation]
where g is convex.
Such a problem structure makes evaluations of the objective and its derivatives highly efﬁcient, and such efﬁcient computation is a key motivation for our algorithms for (1).
Related Works.
The discontinuous cardinality constraint in (1) makes the problem difﬁcult to solve.
To make the optimization problem easier, a popular approach is to slightly sacriﬁce the quality of the solution (either not strictly satisfying the sparsity level constraint or the prediction performance is deteriorated) to use continuous surrogate functions for the (cid:96) 0 -norm, which lead to a continuous nonlinear programming problem, where abundant algorithms are at our disposal.
For instance, using a convex penalty surrogate such as the (cid:96) 1 -norm in the case of LASSO [36], the problem (1) can be relaxed into a convex (unconstrained) one that can be efﬁciently solved by many algorithms.
Other algorithms based on continuous nonconvex relaxations such as the use of smoothly clipped absolute deviation [15] and the minimax concave penalty [41] regularizers are also popular in scenarios with a higher level of noise and outliers in the data.
However, for applications in which enforcing the constraints or getting the best prediction performance is of utmost importance, solving the original problem (1) is inevitable.
(For a detailed review, we refer the interested reader to [11, Section 1].)
Unfortunately, methods for (1) are not as well-studied as those for the surrogate problems.
Moreover, existing methods are indeed still preliminary and too slow to be useful in large-scale problems often faced in modern machine learning tasks.
In view of the present unsatisfactory status for scenarios that simultaneously involve high-volume data and need to get the best prediction performance, this work proposes efﬁcient algorithms to directly solve (1) with large-scale data.
To our knowledge, all the most popular algorithms that directly tackles (1) without the use of surrogates involve using the well-known projected gradient (PG) algorithm, at least as a major component [10–13, 3].
1 [10] proved linear convergence of the objective value with the LS loss function (3) for the iterates generated by PG under a scalable restricted isometry property, which also served as their tool to accelerate PG.
However, given any problem instance, it is hard, if not computationally impossible, to verify whether the said property holds.
On the other hand, [11] established global subsequential convergence to a stationary point for the iterates of PG on (1) without the need for such isometry conditions, and their results are valid for general loss functions f beyond (3).
While some theoretical guarantees are known, the practicality of PG for solving (1) remains a big problem in real-world applications as its empirical convergence speed tends to be slow.
The PG approach is called iterative hard thresholding (IHT) in studies of compressed sensing [13] that mainly focuses on the LS case.
To accelerate IHT, several approaches that alternates between a PG step and a subspace optimization step are also proposed [12, 3], but such methods mainly focus on the LS case and statistical properties, while their convergence speed is less studied from an optimization perspective.
Recently, “acceleration” approaches for PG on general nonconvex regularized problems have been studied in [27, 37].
While their proposed algorithms are also applicable to (1), the obtained convergence speed for nonconvex problems is not faster than that of PG.
This work is inspired by our earlier work [1], which considered a much broader class of problems without requiring convexity nor differentiability assumptions for f , and hence obtained only much weaker convergence results, with barely any convergence rates, for such general problems.
Contributions.
In this work, we revisit the PG algorithm for solving the general problem (1) and propose two acceleration schemes by leveraging the combinatorial nature of (cid:96) 0 -norm.
In particular, we decompose the feasible set
A s as the ﬁnite union of s -dimensional linear subspaces, each representing a subset of the coordinates { 1 , . .
. , n } , as detailed in (7) of Section 2.
Such subspaces are utilized in devising techniques to efﬁciently accelerate PG.
Our ﬁrst acceleration scheme is based on a same-space extrapolation technique such that we conduct extrapolation only when two consecutive iterates w k − 1 and w k lie in the same subspace, and the step size for this extrapolation is determined by a spectral initialization combined with backtracking to ensure sufﬁcient function decrease.
This is motivated by the observation that for (4), objective and derivatives at the extrapolated point can be inferred efﬁciently through a linear combination of Xw k − 1 and Xw k .
The second acceleration technique starts with plain PG, and when consecutive iterates stay in the same subspace, it begins to alternate between a full PG step and a truncated Newton step in the subspace to obtain superlinear convergence with extremely low computational cost.
Our main contributions are as follows:
We prove that PG for (1) is globally convergent to a local optimum with a local linear rate, improving upon the sublinear results of Bertsimas et al.
We emphasize that our framework, like [11], is applicable to general loss functions f satisfying the convexity and smoothness
requirements, and therefore covers not only the classical sparse regression problem but also many other ones encompassed by the empirical risk minimization (ERM) framework.
By decomposing A s as the union of linear subspaces, we further show that PG is provably capable of identifying a subspace containing a local optimum of (1).
By exploiting this property, we propose two acceleration strategies with practical implementation and convergence guarantees for the general problem class (1).
Our acceleration provides both computational and theoretical advantages for convergence, and can in particular obtain superlinear convergence.
In comparison with existing acceleration methods for nonconvex problems [27, 37], this work provides new acceleration schemes with faster theoretical speeds (see Theorems 3.2 and 3.3)
and beyond being applied to the classical PG algorithm, those schemes can also easily be combined with existing accelerated PG approaches to further make them converge even faster.
Numerical experiments exemplify the signiﬁcant improvement in both iterations and running time brought by our acceleration methods, in particular over the projected gradient algorithm by [11] as well as the accelerated proximal gradient method for nonconvex problems proposed by [27].
This work is organized as follows.
We review the projected gradient algorithm and prove its local linear convergence and subspace identiﬁcation for arbitrary smooth loss functions in Section 2.
In
Section 3, we propose the acceleration schemes devised through decomposing the constraint set in (1) into subspaces of (cid:60) n .
Experiments in Section 4 then illustrate the effectiveness of the proposed acceleration techniques, and Section 5 concludes this work.
All proofs, details of the experiment settings, and additional experiments are in the appendices.
Projected Gradient Algorithm
The projected gradient algorithm for solving (1) is given by the iterations
[Equation]
(5) where P A s ( w ) denotes the projection of w onto A s , which is set-valued because of the nonconvexity of A s .
When f is given by (3), global linear convergence of this algorithm under a restricted isometry condition is established in [10].
For a general convex f with L -Lipschitz continuous gradients, that is,
[Equation]
the global subsequential convergence of (5) is proved in [11], but neither global nor local rates of convergence is provided.
In this section, we present an alternative proof of global convergence and more importantly establish its local linear convergence.
A useful observation that we will utilize in the proofs of our coming convergence results is that the nonconvex set A s given by (2) can be decomposed as a ﬁnite union of subspaces in (cid:60) n :
[Equation]
where e j is the j th standard unit vector in (cid:60) n .
Throughout this paper, we assume that λ ∈ (0 , L − 1 ) .
Theorem 2.1.
Let { w k } be a sequence generated by (5) .
Then:
(a) (Subsequential convergence)
Either { f ( w k ) } is
strictly decreasing, or there
exists N > 0 such that w k = w N for all
k
≥
N .
In addition, any accumulation point w ∗ of { w k } satisﬁes
w ∗ ∈
P A s ( w ∗ − λ ∇ f ( w ∗ ))
, and is hence a stationary point of (1) .
(b) (Subspace identiﬁcation and full convergence)
There exists N ∈ N such that
[Equation]
whenever w k → w ∗ .
In particular, if T λ PG ( w ∗ ) is a singleton for an accumulation point w ∗ of { w k } , then w ∗ is a local minimum for (1) , w k → w ∗ , and (8) holds.
(c) ( Q -linear convergence)
If T λ PG ( w ∗ ) is a singleton for an accumulation point w ∗ and w (cid:55)→ w − λ ∇ f ( w ) is a contraction over A J for all J ∈
I w ∗ , then { w k } converges to w ∗ at a Q -linear rate.
In other words, there is N 2 ∈ N and γ ∈ [0 , 1) such that
[Equation]
It is well-known that an optimal solution of (1) is also a stationary point of it [8, Theorem 2.2], and therefore (a) proves the global subsequential convergence of PG to candidate solutions of (1).
Consider z ∗ :
= w ∗ − λ ∇ f ( w ∗ ) , and let τ be a permutation of { 1 , . . . , n } such that z ∗ τ (1) ≥ z ∗ τ (2) ≥ · · · ≥ z ∗ τ ( n ) .
The requirement of T λ PG ( w ∗ ) being a singleton in Theorem 2.1 (b) then simply means the mild condition of z ∗ τ ( s ) >
z ∗ τ ( s +1) , which is almost always true in practice.
The requirement for (c) can be fulﬁlled when f conﬁned to A J is strongly convex, even if f itself is not.
This often holds true in practice when f is of the form (4) and we restrict s in (1) to be smaller than the number of data instances m , and is thus also mild.
The existence of a stationary point can be guaranteed when { w k } is a bounded sequence, often guaranteed when f is coercive on A J for each J ∈ J s .
In comparison to existing results in [11, 2, 14], parts (b) and (c) of Theorem 2.1 are new.
In particular, part (b) provides a full convergence result that usually requires stronger regularity assumptions like the Kurdyka-Łojasiewicz (KL) condition [2, 14] (see also (21)) that requries the objective value to decrease proportionally with the minimum-norm subgradient in a neighborhood of the accumulation point, but we just need the very mild singleton condition right at the accumulation point only.
Part (c) gives a local linear convergence for the PG iterates even if the problem is nonconvex, while the rates in [14] requires a KL condition and the rate is measured in the objective value.
The following result further provides rates of convergence of the objective values even without the conventional KL assumption.
The ﬁrst rate below follows from [24].
Theorem 2.2.
Let { w k } be a sequence generated by (5) .
If w k → w ∗ , such as when T λ PG ( w ∗ ) is a singleton at an accumulation point w ∗ of (5) , then
[Equation]
Moreover, under the hypothesis of Theorem 2.1 (c), the objective converges to f ( w ∗ ) R-linearly, i.e.,
[Equation]
By using Theorem 2.1, we can also easily get rates faster than (10) under a version of the KL condition that is easier to understand and verify than those assumed in existing works.
In particular, existing analyses require the KL condition to hold in a neighborhood in (cid:60) n of an accumulation point, but we just need it to hold around w ∗ within A J for the restriction f | A J for each J ∈
I w ∗ .
These results are postponed to Theorem 3.2 in the next section as the PG method is a special case of our acceleration framework.
Accelerated methods
The main focus of this work is the proposal in this section of new techniques with solid convergence guarantees to accelerate the PG algorithm presented in the preceding section.
Our techniques fully exploit the subspace identiﬁcation property described by the inclusion (8), as well as the problem structure of (4) to devise efﬁcient algorithms.
We emphasize that the two acceleration strategies described below can be combined together, and they are also widely applicable such that they can be employed to other existing algorithms for (1) as long as such algorithms have a property similar to (8).
3.1 Acceleration by extrapolation
Traditional extrapolation techniques are found in the realm of convex optimization to accelerate algorithms [9, 31] with guaranteed convergence improvements, but were often only adopted as heuristics in the nonconvex setting, until some recent works showed that theoretical convergence can also be achieved [27, 37].
However, unlike the convex case, these extrapolation strategies for nonconvex problems do not lead to faster convergence speed nor an intuitive reason for doing so.
An extrapolation step proceeds by choosing a positive stepsize along the direction determined by two consecutive iterates.
That is, given two iterates w k − 1 and w k , an intermediate point z k :=
w k + t k ( w k − w k − 1 ) for some stepsize t k ≥ 0 is ﬁrst calculated before applying the original algorithmic map ( T λ PG in our case).
Another popular acceleration scheme for gradient algorithms is the spectral approach pioneered by [5].
They take the differences of the gradients and of the iterates in two consecutive iterations to estimate the curvature at the current point, and use it to decide the step size for updating along the reversed gradient direction.
It has been shown in [39] that equipping this step size with a backtracking procedure leads to signiﬁcantly faster convergence for proximal gradient on regularized optimization problems, which includes our PG for (1) as a special case.
To describe our proposed double acceleration procedure that combines extrapolation and spectral techniques, we ﬁrst observe that all PG iterates lie on A s , and that A s can be ﬁnitely decomposed as (7).
When two consecutive iterates lie on the same convex subspace A J for some J ∈ J s , within these two iterations, we are actually conducting convex optimization.
In this case, an extrapolation step within A J is reasonable because it will not violate the constraint, and acceleration can be expected from the improved rates of accelerated proximal gradient on convex problems in [9, 32].
Judging from Theorem 2.1 (b), the corresponding J is also a candidate index set that belongs to I w ∗ , so extrapolation within A J makes further sense.
We set t k = 0 to skip the extrapolation step if d k is not a descent direction for f at w k .
Otherwise, we start from some
ˆ t k > 0 decided by the curvature information of f , and then execute a backtracking linesearch along d k :=
w k − w k − 1 to set t k = η i ˆ t k for the smallest integer i ≥ 0 that provides sufﬁcient descent
[Equation]
(12) given parameters η, σ ∈ (0 , 1) .
We then apply (5) to z k = w k + t k d k to obtain w k +1 .
For the spectral initialization ˆ t k for accelerating the convergence, instead of directly using approaches of [5, 39] that takes the reversed gradient as the update direction, we need to devise a different mechanism as our direction
d k is not directly related to the gradient.
We observe that for the stepsize
[Equation]
used in [5], the ﬁnal update − α k ∇ f ( w k ) is actually the minimizer of the following subproblem
[Equation]
By juxtaposing the above quadratic problem and the upper bound provided by the descent lemma [7,
Lemma 5.7], we can view α − 1 k as an estimate of the local Lipschitz parameter that could be much smaller than L but still guarantee descent of the objective.
We thus follow this idea to decide ˆ t k using such curvature estimate and the descent lemma by
[Equation]
Another interpretation of (13) is that α − 1 k
I also serves as an estimate of ∇ 2 f ( w k ) , 3 and the objective in (14) is a low-cost approximation of the second-order Taylor expansion of f .
However, we notice that for problems in the form of (4) and with d k ∈
A J , the exact second-order Taylor expansion
[Equation]
can be calculated efﬁciently.
In particular, for (4) and any d k ∈ A J , we get from Xd k
= X : ,J d kJ :
[Equation]
which can be calculated in O ( ms ) time by computing X : ,J d kJ ﬁrst.
This O ( ms ) cost is much cheaper than the O ( mn ) one for evaluating the full gradient of f needed in the PG step, so our extrapolation plus spectral techniques has only negligible cost.
Moreover, for our case of d k = w k − w k − 1 , we can further reduce the cost of calculate X : ,J d k J and thus (17) to O ( m ) by recycling intermediate computational results needed in evaluating f ( w k ) through X : ,J d kJ =
Xw k − Xw k − 1 .
With such tricks for efﬁcient computation, we therefore consider the more accurate approximation to let ˆ t k be the scalar that minimizes the quadratic function on the right-hand side of (16) for problems in the form (4).
That is, we use
[Equation]
Finally, for both (18) and (15), we safeguard ˆ t k by
[Equation]
for some ﬁxed α max ≥ α min > 0 , where
[Equation]
We also note that the low cost of evaluating Xd k is also the key to making the backtracking in (12) practical, as each f ( w k + η i ˆ t k d k ) can be calculated in O ( m ) time through linear combinations of Xw k and Xd k .
The above procedure is summarized in Algorithm 1 with global convergence guaranteed by Theorem 3.1.
In Theorem 3.2, we establish its full convergence as well as its convergence rates under a KL condition at w ∗ : there exists neighborhood U ⊂ (cid:60) n of w ∗ , θ ∈ [0 , 1] , and κ > 0 such that for every J ∈ I w ∗ ,
[Equation]
We denote by n k the number of successful extrapolation steps in the ﬁrst k iterations of Algorithm 1.
The part of θ ∈ [0 , 1 / 2] with f being convex in the last item of Theorem 3.2 is directly from the result of [25].
Theorem 3.1.
Under the hypotheses of Theorem 2.1, any accumulation point of a sequence generated by Algorithm 1 is a stationary point.
Theorem 3.2.
Consider either (5) or Algorithm 1 with η, σ, (cid:15) ∈ (0 , 1) , and α max ≥ α min > 0 , and suppose that there is an accumulation point w ∗ of the iterates at which the KL condition holds.
Then w k → w ∗ .
Moreover, the following rates hold:
(a) If θ ∈ (1 / 2 , 1) :
f ( w k ) − f ( w ∗ )
=
O (( k + n k ) − 1 / (2 θ − 1) ) .
(b) If θ ∈ (0 , 1 / 2] : f ( w k ) − f ( w ∗ ) =
O (exp( − ( k + n k ))) .
(c) If θ = 0 , or θ ∈ [0 , 1 / 2] and f is convex: there is k 0 ≥ 0 such that f ( w k ) =
f ( w ∗ ) for all k ≥ k 0 .
We stress that convexity of f is not required in Theorems 3.1 and 3.2 except the second half of the last item of Theorem 3.2.
There are several advantages of the proposed extrapolation strategy over existing ones in [27, 37].
The most obvious one is the faster rates in Theorem 3.2 over PG such that each successful extrapolation step in our method contributes to the convergence speed, while existing methods only provide the same convergence speed as PG.
Next, existing strategies only use prespeciﬁed step sizes without information from the given problem nor the current progress, and they only restrict such step sizes to be within [0 , 1] .
Our method, on the other hand, fully takes advantage of the function curvature and can allow for arbitrarily large step sizes to better decrease the objective.
In fact, we often observe t k (cid:29) 1 in our numerical experiments.
Moreover, our acceleration techniques utilize the nature of (7) and (4) to obtain very efﬁcient implementation for ERM problems such that the per-iteration cost of Algorithm 1 is almost the same as that of PG, while the approach of [27] requires evaluating f and ∇ f at two points per iteration, and thus has twice the per-iteration cost.
A ﬁnite termination result similar to Theorem 3.2 (c) is presented in [29] under a Hölderian error bound that is closely related to the KL condition, but their result requires convexity of both the smooth term and the regularizer, so it is not applicable to (1) that involves a nonconvex constraint.
3.2 Subspace Identiﬁcation
In line with the above discussion, we interpret (8) as a theoretical property guaranteeing that the iterates of the projected gradient algorithm (5) will eventually identify the subspaces A J that contain a candidate solution w ∗ after a ﬁnite number of iterations.
Consequently, the task of minimizing f over the nonconvex set A s can be reduced to a convex optimization problem of minimizing f over
A J .
Motivated by this, we present a two-stage algorithm described in Algorithm 2 that switches to a high-order method for smooth convex optimization after a candidate piece
A J is identiﬁed to
1 Given an initial vector w 0 ∈ (cid:60) n and parameters (cid:15), η, σ ∈ (0 , 1) , α max ≥ α min > 0 , λ do k w k w k the same A J ; ζ k ≥ then d k ← w k − k
1 , and k
=
do t k ← η i k is satisﬁed then z k w k + k d k , and break else z k k k λ obtain even faster convergence.
Since ∇ f is assumed to be Lipschitz continuous,
the generalized
Hessian of f exists everywhere [19], so we may employ a semismooth Newton (SSN) method [35] with backtracking linesearch to get a faster convergence speed with low cost (details in Appendix A).
In particular, we reduce the computation costs by considering the restriction of f on the subspace
A J by treating the coordinates not in J as non-variables so that the problem considered is indeed smooth and convex.
As we cannot know a priori whether I w ∗ is indeed identiﬁed, we adopt the approach implemented in [26, 28, 23] to consider it identiﬁed when w k activates the same A J for long enough consecutive iterations.
To further safeguard that we are not optimizing over a wrong subspace, we also incorporate the idea of [38, 4, 28, 23] to periodically alternate to a PG step (5) after switching to the SSN stage.
A detailed description of this two-stage algorithm is in Algorithm 2.
In the following theorem, we show that superlinear convergence can be obtained for Algorithmeven if we take only one SSN step every time between two steps of (5), using a simpliﬁed setting of twice-differentiability.
For our next theorem, we need to introduce some additional notations.
Given any w ∈ A J , we use f J ( w J ) :
= f ( w ) to denote the function of considering only the coordinates of w in J as variables and treating the remaining as constant zeros.
We assume that the conditions of
Theorem 2.1 (b) hold with w ∗ ∈
A s , and that f is twice-differentiable around a neighborhood U of w ∗ with ∇ 2 f J Lipschitz continuous in U and ∇ 2 f J ( w ∗ ) positive deﬁnite for all J ∈
I
w ∗ .
Theorem 3.3.
Suppose that starting after k ≥ N and P A s ( w k ) ⊂
U , we conduct t Newton steps between every two steps of (5) for t ≥ 1 :
[Equation]
Then w k → w ∗ at a Q -quadratic rate.
In practice, the linear system for obtaining the SSN step is only solved inexactly via a (preconditioned) conjugate gradient (PCG) method, and with suitable stopping conditions for PCG and proper algorithmic modiﬁcations such as those in [40, 30], superlinear convergence can still be obtained easily.
Interested readers are referred to Appendix A for a more detailed description of our implementation.
Experiments
In this section, we conduct numerical experiments to demonstrate the accelerated techniques presented in Section 3.
We employ Algorithm 1 (APG) with (18) to accelerate PG, and further accelerate APG by incorporating subspace identiﬁcation described in Algorithm 2, which we denote by APG+.
Comparisons with the extrapolated PG algorithm of Li and Lin [27], which we denote by PG-LL, are also presented.
PG-LL is a state-of-the-art approach for nonconvex regularized optimization and thus suitable for (1).
For f in (1), we consider both LS (3) and logistic regression (LR)
[Equation]
1 Given an initial vector w 0 ∈ (cid:60) n and S, t ∈ N .
Set Unchanged ← 0 .
2 for k = 0 , 1 , 2 , . . .
do if k > 0 , and w k − 1 and w k activate the same component of A s then
Let J ∈ J s correspond to the activated component
Unchanged ← Unchanged +  else Unchanged ← if Unchanged ≥ S then
y k ← P
A J ( w k ) and use t steps of SSN described in Appendix A, starting from y k , to ﬁnd z k that approximately minimizes f | A J if SSN fails then z k ← w k and Unchanged ← 0 .
else
z k ← w k w k +1 ← T λ PG ( z k ) where ( x i , y i ) ∈ (cid:60) n × {− 1 , 1 } , i = 1 , . . .
, m , are the training instances, and µ > 0 is a small regularization parameter added to make the logistic loss coercive.
The algorithms are implemented in MATLAB and tested with public datasets in Tables 2 andin
Appendix B. All algorithms compared start from w
=
and terminate when the ﬁrst-order optimality condition
[Equation]
is met for some given ˆ (cid:15) > 0 .
More setting and parameter details of our experiments are in Appendix B.
Comparisons of algorithms for large datasets.
To ﬁt the practical scenario of using (1), we speciﬁcally selected high-dimensional datasets with n larger than m .
We conduct experiments with various s to widely test the performance under different scenarios.
In particular, we consider s ∈
[Equation]
[Equation] [Equation]
, (cid:100) 0 .
1 m (cid:101)} on all data except for the largest dataset webspam , for which we set s ∈ {(cid:100) 0 .
001 m (cid:101) , (cid:100) 0 .
005 m (cid:101) , (cid:100) 0 .
01 m (cid:101)} .
The results of the experiment with the smallest s are summarized in Figure 1, and results of the other two settings of s are in Appendix C.
[Figure]
Evidently, the extrapolation procedure in APG provides a signiﬁcant improvement in the running time compared with the base algorithm PG, and further incorporating subspace identiﬁcation as in APG+ results to a very fast algorithm that outperforms PG and APG by magnitudes.
Since the per-iteration
[Table]
+ 0.
PG
[Table]
LL 22.
APG
[Table]
LL+ 0.
[Table]
[Table] [Table]
webspam
PG ∗ 18660.
∗ 30776.
APG 19683.
APG
[Table] [Table]
[Table] [Table]
LL
[Table] [Table]
[Table] [Table] [Table]
+
[Table]
log1p
PG ∗ 2998.
[Table]
∗
[Table] [Table]
[Table] [Table] [Table]
[Table]
APG
[Table]
[Table] [Table]
-LL ∗ 6049.
APG-LL+ 41.
E
[Table]
tﬁdf
PG ∗ 242.
∗ 666.
APG 1.
APG+ 1.
PG-LL 110.
APG-LL+ 1.
cost of PG and APG are almost the same as argued in Section 3, we note that the convergence of
APG in terms of iterations is also superior to that of PG.
We also report the required time and number of gradient evaluations (which is the main computation at each iteration) for the algorithms to drive (24) below ˆ (cid:15) =
10 − 6 .
For PG, APG, and APG+, one gradient evaluation is needed per iteration, so the number of gradient evaluations is equivalent to the iteration count.
For PG-LL, two gradient evaluations are needed per iteration, so its cost is twice of other methods.
We also report the prediction performance on the test data, and we in particular use the test accuracy for (23) and the mean-squared error for (3).
Results for the two smaller s are in Table 1 while that for the largest s is in Appendix C.
It is clear from the results in Tablethat APG outperforms PG-LL for most of the test instances considered, while APG+ is magnitudes faster than PG-LL.
When we equip PG-LL with our acceleration techniques by replacing T λ PG in
Algorithms 1 and 2 with the algorithmic map deﬁning PG-LL, we can further speed up PG-LL greatly as shown under the name APG-LL+ (see Table 1).
We do not observe a method that consistently possesses the best prediction performance, as this is mainly affected by which local optima is found, while no algorithm is able to ﬁnd the best local optima among all candidates.
With no prediction performance degradation, we see that APG+ and APG-LL+ reduce the time needed to solve (1) to a level signiﬁcantly lower than that of the state of the art.
In Appendix C.3, we demonstrate the effect on prediction performance when we vary the residual (24) and illustrate that tight residual level is indeed required to obtain better prediction.
Comparisons with a greedy method is shown in Appendix C.4.
Transition Plots.
To demonstrate the behavior of the algorithm for increasing values of s , we ﬁt the smaller datasets in Table 3 using logistic loss (23) and least squares loss (3) for varying s = (cid:100) km (cid:101) , where k = 0 .
The transition plots are presented in Figure 2.
We note that the time is in log scale.
We can see clearly that APG+ and APG-LL+ are consistently magnitudes faster than the baseline PG method throughout all sparsity levels.
On the other hand, the same-subspace extrapolation scheme of APG is consistently faster than PG and APG-LL and slower than the two Newton acceleration schemes, although the performance is sometimes closer to APG+/APG-LL+ while sometimes closer to PG.
APG-LL tends to outperform PG in most situations as well, but in several cases when solving the least square problem, especially when s is small, it can sometimes be slower than PG.
Overall speaking, the results in the transition plots show that our proposed acceleration schemes are indeed effective for all sparsity levels tested.
Sparse regularized
[Table]
[Figure]
Conclusions
In this work
[Figure]
we revisited the projected gradient algorithm for solving (cid:96)
-norm constrained optimization problems.
Through a natural decomposition of the constraint set into subspaces and the proven ability of the projected gradient method to identify a subspace that contains a solution, we further proposed effective acceleration schemes with provable convergence speed improvements.
Experiments showed that our acceleration strategies improve signiﬁcantly both the convergence speed and the running time of the original projected gradient algorithm
[Figure]
and outperform the state of the art for (cid:96)
-norm constrained problems by a huge margin.
We plan to extend our analysis and algorithm to the setting of a nonconvex objective in the near future
[Figure]
Acknowledgments
This work was supported in part by Academia Sinica Grand Challenge Program Seed Grant No.
[-= 
-]
AS-GCS
[-Noting that ∇ f and P A J are continuous as A J is closed and convex, we obtain from (28) that w k r +
-]
[-because only entries of ∇ f ( w ∗ ) outside J k ∈ I w ∗ could be nonzero, but those entries are identically 0 for both w k and w ∗ , that is, w ki = w ∗ i = 
-]
[-Then from (50), we have that z k r → w ∗ so that w k r +
-]
[-( − 1 , 0) , for m ≥ k 
-]