The Hessian Screening Rule
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Predictor screening rules, which discard predictors before ﬁtting a model, have hadconsiderable impact on the speed with which sparse regression problems, such asthe lasso, can be solved.
In this paper we present a new screening rule for solvingthe lasso path: the Hessian Screening Rule.
The rule uses second-order informationfrom the model to provide both effective screening, particularly in the case of highcorrelation, as well as accurate warm starts.
The proposed rule outperforms allalternatives we study on simulated data sets with both low and high correlation for(cid:96) 1 -regularized least-squares (the lasso) and logistic regression.
It also performsbest in general on the real data sets that we examine.
Introduction
High-dimensional data, where the number of features ( p ) exceeds the number of observations( n ), poses a challenge for many classical statistical models.
A common remedy for this issue is toregularize the model by penalizing the regression coefﬁcients such that the solution becomes sparse.
Apopular choice of such a penalization is the (cid:96) 1 -norm, which, when the objective is least-squares, leadsto the well-known lasso [1].
More speciﬁcally, we will focus on the following convex optimizationproblem:
[Equation]
where f ( β ; X ) is smooth and convex.
We let ˆ β be the solution vector for this problem and, abusingnotation, equivalently let ˆ β : R (cid:55)→ R p be a function that returns this vector for a given λ .
Our focuslies in solving (1) along a regularization path λ 1 , λ 2 . . .
, λ m with λ 1 ≥ λ 2 ≥ · · · ≥ λ m .
We startthe path at λ max , which corresponds to the null (all-sparse) model 1 , and ﬁnish at some fraction ofλ max for which the model is either almost saturated (in the p ≥ n setting), or for which the solutionapproaches the ordinary least-squares estimate.
The motivation for this focus is that the optimal λis typically unknown and must be estimated through model tuning, such as cross-validation.
Thisinvolves repeated reﬁtting of the model to new batches of data, which is computationally demanding.
Fortunately, the introduction of so-called screening rules has improved this situation remarkably.
Screening rules use tests that screen and possibly discard predictors from the model before it is ﬁt,which effectively reduces the dimensions of the problem and leads to improvements in performanceand memory usage.
There are, generally speaking, two types of screening rules: safe and heuristicrules.
Safe rules guarantee that discarded predictors are inactive at the optimum—heuristic rules donot and may therefore cause violations: discarding active predictors.
The possibility of violationsmean that heuristic methods need to validate the solution through checks of the Karush–Kuhn–Tucker(KKT) optimality conditions after optimization has concluded and, whenever there are violations, rerun optimization, which can be costly particularly because the KKT checks themselves are expensive.
This means that the distinction between safe and heuristic rules only matters in regards to algorithmic details—all heuristic methods that we study here use KKT checks to catch these violations, whichmeans that these methods are in fact also safe.
Screening rules can moreover also be classiﬁed as basic , sequential , or dynamic .
Basic rules screenpredictors based only on information available from the null model.
Sequential rules use informationfrom the previous step(s) on the regularization path to screen predictors for the next step.
Finally,dynamic rules screen predictors during optimization, reducing the set of screened predictors repeatedlythroughout optimization.
Notable examples of safe rules include the basic SAFE rule [2], the sphere tests [3], the R-regiontest [4], Slores [5], Gap Safe [6, 7], and Dynamic Sasvi [8].
There is also a group of dual polytopeprojection rules, most prominently Enhanced Dual Polytope Projection (EDPP) [9].
As notedby Fercoq, Gramfort, and Salmon [6], however, the sequential version of EDPP relies on exactknowledge of the optimal solution at the previous step along the path to be safe in practice, whichis only available for λ max .
Among the heuristic rules, we have the Strong Rule [10], SIS [11], and
ExSIS [12].
But the latter two of these are not sequential rules and solve a potentially reduced formof the problem in (1)—we will not discuss them further here.
In addition to these two types ofrules, there has also recently been attempts to combine safe and heuristic rules into so-called hybridrules [13].
There are various methods for employing these rules in practice.
Of particular interest are so-calledworking set strategies, which use a subset of the screened set during optimization, iteratively updatingthe set based on some criterion.
Tibshirani et al.
[10] introduced the ﬁrst working set strategy, whichwe in this paper will refer to simply as the working set strategy .
It uses the set of predictors that haveever been active as an initial working set.
After convergence on this set, it checks the KKT optimalityconditions on the set of predictors selected by the strong rule, and then adds predictors that violate theconditions to the working set.
This procedure is then repeated until there are no violations, at whichpoint the optimality conditions are checked for the entire set, possibly triggering additional iterationsof the procedure.
Blitz [14] and Celer [15] are two other methods that use both Gap Safe screeningand working sets.
Instead of choosing previously active predictors as a working set, however, both
Blitz and Celer assign priorities to each feature based on how close each feature is of violating the
Gap Safe check and construct the working set based on this prioritization.
In addition to this, Celeruses dual point acceleration to improve Gap Safe screening and speed up convergence.
Both Blitzand Celer are heuristic methods.
One problem with current screening rules is that they often become conservative—including largenumbers of predictors into the screened set—when dealing with predictors that are strongly correlated.
Tibshirani et al.
[10], for instance, demonstrated this to be the case with the strong rule, which was themotivation behind the working set strategy.
(See Appendix F.4 for additional experiments verifyingthis).
Yet because the computational complexity of the KKT checks in the working set strategystill depends on the strong rule, the effectiveness of the rule may nevertheless be hampered in thissituation.
A possible and—as we will soon show—powerful solution to this problem is to make use ofthe second-order information available from (1), and in this paper we present a novel screening rulebased on this idea.
Methods using second-order information (the Hessian) are often computationallyinfeasible for high-dimensional problems.
We utilize two properties of the problem to remedy thisissue: ﬁrst, we need only to compute the Hessian for the active set, which is often much smaller thanthe full set of predictors.
Second, we avoid constructing the Hessian (and it’s inverse) from scratchfor each λ along the path, instead updating it sequentially by means of the Schur complement.
Theavailability of the Hessian also enables us to improve the warm starts (the initial coefﬁcient estimateat the start of each optimization run) used when ﬁtting the regularization path, which plays a key rolein our method.
We present our main results in Section 3, beginning with a reformulation of the strong rule andworking set strategy before we arrive at the screening rule that represents the main result of thispaper.
In Section 4, we present numerical experiments on simulated and real data to showcase theeffectiveness of the screening rule, demonstrating that the rule is effective both when p (cid:29) n andn (cid:29) p , out-performing the other alternatives that we study.
Finally, in Section 5 we wrap up with adiscussion on these results, indicating possible ways in which they may be extended.
Preliminaries
We use lower-case letters to denote scalars and vectors and upper-case letters for matrices.
We use 0and 1 to denote vectors with elements all equal to 0 or 1 respectively, with dimensions inferred fromcontext.
Furthermore, we let sign be the standard signum function with domain {− 1 , 0 , 1 } , allowingit to be overloaded for vectors.
Let c ( λ ) :
=
−∇ β f (cid:0) ˆ β ( λ ); X (cid:1) be the negative gradient, or so-called correlation , and denote A
λ
={ i : | c ( λ ) i | > λ } as the active set at λ : the support set of the non-zero regression coefﬁcientscorresponding to ˆ β ( λ ) .
In the interest of brevity, we will let A :
= A λ .
We will consider β a solutionto (1) if it satisﬁes the stationary criterion
[Equation]
Here ∂ is the subdifferential of (cid:107) β (cid:107) 1 , deﬁned as∂ j ∈ (cid:40) { sign(ˆ β j ) } if ˆ β j (cid:54) = 0 , [ − 1 , 1] otherwise.
This means that there must be a ˜ ∂ ∈ ∂ for a given λ such that∇ β f ( β ; X ) + λ ˜ ∂
= 0 .
[Equation]
Main Results
In this section we derive the main result of this paper: the Hessian screening rule.
First, however,we now introduce a non-standard perspective on screening rules.
In this approach, we note that (2)suggests a simple and general formulation for a screening rule, namely: we substitute the gradientvector in the optimality condition of a (cid:96) 1 -regularized problem with an estimate.
More precisely, wediscard the j th predictor for the problem at a given λ if the magnitude of the j th component of thegradient vector estimate is smaller than this λ , that is
[Equation]
In the following sections, we review the strong rule and working set method for this problem fromthis perspective, that is, by viewing both methods as gradient approximations.
We start with the caseof the standard lasso ( (cid:96) 1 -regularized least-squares), where we have f ( β ; X ) = 12 (cid:107)
Xβ − y (cid:107) 22 .
The Strong Rule The sequential strong rule for (cid:96) 1 -penalized least-squares regression [10] discardsthe j th predictor at λ =
λ k
+
if
[Equation]
This is equivalent to checking that
[Equation]
satisﬁes (4).
The strong rule gradient approximation (5) is also known as the unit bound , since itassumes the gradient of the correlation vector to be bounded by one.
The Working Set Method A simple but remarkably effective alternative to direct use of the strongrule is the working set heuristic [10].
It begins by estimating β at the ( k + 1) th step using only thecoefﬁcients that have been previously active at any point along the path, i.e. A 1: k
= ∪ k i =1 A i .
Theworking set method can be viewed as a gradient estimate in the sense that
[Equation]
where ˜ β ( λ, A )
=
arg min β
|| y −
X
A β
||
+ λ | β | .
The Hessian Screening Rule We have shown that both the strong screening rule and the workingset strategy can be expressed as estimates of the correlation (negative gradient) for the next step ofthe regularization path.
As we have discussed previously, however, basing this estimate on the strongrule can lead to conservative approximations.
Fortunately, it turns out that we can produce a betterestimate by utilizing second-order information.
We start by noting that (3), in the case of the standard lasso, can be formulated as
[Equation]
and consequently that
[Equation]
Note that, for an interval [ λ l , λ u ] in which the active set is unchanged, that is, A λ
= A for allλ ∈ [ λ u , λ k ] , then ˆ β ( λ ) is a continuous linear function in λ (Theorem 3.1) 2 .
Theorem 3.1.
Let ˆ β ( λ ) be the solution of (1) where f ( β ; X )
= 12 (cid:107)
Xβ − y (cid:107) 22 .
Deﬁne
[Equation]
and ˆ β λ ∗ ( λ ) A cλ ∗ = 0 .
If it for λ ∈ [ λ 0 , λ ∗ ] holds that (i) sign (cid:0) ˆ β λ ∗ ( λ ) (cid:1)
= sign (cid:0)
ˆ β ( λ ∗ ) (cid:1) and (ii)max |∇ f
(ˆ β λ ∗ ( λ ))
A λ ∗ | < λ, then ˆ β ( λ ) =
ˆ β λ ∗ ( λ ) for λ ∈ [ λ 0 , λ ∗ ] .
See Appendix A for a full proof.
Using Theorem 3.1, we have the following second-order approximation of c ( λ k +1 ) :
[Equation]
Remark 3.2 .
If no changes in the active set occur in [ λ k +1 , λ k ] , (6) is in fact an exact expression forthe correlation at the next step, that is, ˆ c H ( λ k +1 )
=
c ( λ k +1 ) .
One problem with using the gradient estimate in (6) is that it is expensive to compute due to theinner products involving the full design matrix.
To deal with this, we use the following modiﬁcation,in which we restrict the computation of these inner products to the set indexed by the strong rule,assuming that predictors outside this set remain inactive:
[Equation]
else
[Equation]
For high-dimensional problems, this modiﬁcation leads to large computational gains and seldomproves inaccurate, given that the strong rule only rarely causes violations [10].
Lastly, we make onemore adjustment to the rule, which is to add a proportion of the unit bound (used in the strong rule)to the gradient estimate:
[Equation]
˜ c H ( λ k +1 )
j
+ γ ( λ k +1 − λ k ) sign( c ( λ k ) j ) , where γ ∈ R + .
Without this adjustment there would be no upwards bias on the estimate, whichwould cause more violations than would be desirable.
In our experiments, we have used γ = 0 .
01 ,which has worked well for most problems we have encountered.
This ﬁnally leads us to the Hessianscreening rule : discard the j th predictor at λ k +1 if | ˇ c H ( λ k +1 )
j | <
λ
k
+1 .
We make one more modiﬁcation in our implementation of the Hessian Screening Rule, which is touse the union of the ever-active predictors and those screened by the screening rule as our ﬁnal set ofscreened predictors.
We note that this is a marginal improvement to the rule, since violations of therule are already quite infrequent.
But it is included nonetheless, given that it comes at no cost andoccasionally prevents violations.
As an example of how the Hessian Screening Rule performs, we examine the screening performanceof several different strategies.
We ﬁt a full regularization path to a design with n = 200 , p = 20 000 , and pairwise correlation between predictors of ρ .
(See Section 4 and Appendix F.4 for moreinformation on the setup.)
We compute the average number of screened predictors across iterationsof the coordinate descent solver.
The results are displayed in Figure 1 and demonstrate that ourmethod gracefully handles high correlation among predictors, offering a screened set that is manytimes smaller than those produced by the other screening strategies.
In Appendix F.4 we extend theseresults to (cid:96) 1 -regularized logistic regression as well and report the frequency of violations.
[Figure]
Updating the Hessian A potential drawback to using the Hessian screening rule is the computational costs of computing the Hessian and its inverse.
Let A k be the active set at step k on thelasso path.
In order to use the Hessian screening rule we need H − 1 k
=
( X T A k X
A k ) − 1 .
Computing( X T
A k X A k ) − 1 directly, however, has numerical complexity O (
|A k | 3 + |A k | 2 n ) .
But if we havestored ( H − 1 k − 1 , H k − 1 ) previously, we can utilize it to compute ( H − 1 k , H k ) more efﬁciently via theso-called sweep operator [17].
We outline this technique in Algorithm 1 (Appendix B).
The algorithmhas a reduction step and an augmentation step; in the reduction step, we reduce the Hessian and itsinverse to remove the presence of any predictors that are no longer active.
In the augmentation step,we update the Hessian and its inverse to account for predictors that have just become active.
The complexity of the steps depends on the size of the sets C =
A k − 1 \
A k , D =
A k \
A k − 1 ,and E =
A k ∩ A k −
The complexity of the reduction step is O ( |C| 3 + |C| 2 |E| + |C||E| 2 ) and thecomplexity of the augmentation step is O ( |D| 2 n + n |D||E| + |D| 2 |E| + |D| 3 ) since n ≥ max( |E| , |D| ) .
An iteration of Algorithm 1 therefore has complexity O ( |D| 2 n + n |D||E| + |C| 3 + |C||E| 2 ) .
In most applications, the computationally dominant term will be n |D||E| (since, typically, n > |E| >
D > C ) which could be compared to evaluating the gradient for β A k , which is n ( |D| + |E| ) whenβ
A ck = 0 .
Note that we have so far assumed that the inverse of the Hessian exists, but this need notbe the case.
To deal with this issue we precondition the Hessian.
See Appendix C for details.
Warm Starts
The availability of the Hessian and its inverse offers a coefﬁcient warm start that ismore accurate than the standard, naive, approach of using the estimate from the previous step.
Withthe Hessian screening rule, we use the following warm start.
[Equation]
where H − 1 A k is the Hessian matrix for the differentiable part of the objective.
Remark 3.3 .
The warm start given by (7) is the exact solution at λ k if the active set remains constantin [ λ k +1 , λ k ] .
As a ﬁrst demonstration of the value of this warm start, we look at two data sets: YearPredicitionMSDand colon-cancer .
We ﬁt a full regularization path using the setup as outlined in Section 4, with orwithout Hessian warm starts.
For YearPredictionMSD we use the standard lasso, and for colon-cancer(cid:96) 1 -regularized logistic regression.
The Hessian warm starts offer sizable reductions in the number of passes of the solver (Figure 2), formany steps requiring only a single pass to reach convergence.
On inspection, this is not a surprisingﬁnd.
There are no changes in the active set for many of these steps, which means that the warm startis almost exact—“almost” due to the use of a preconditioner for the Hessian (see Appendix C).
[Figure]
Standard
General Loss Functions
We now venture beyond the standard lasso and consider loss functions ofthe form
[Equation]
where f i is convex and twice differentiable.
This, for instance, includes logistic, multinomial, and
Poisson loss functions.
For the strong rule and working set strategy, this extension does not makemuch of a difference.
With the Hessian screening rule, however, the situation is different.
To see this, we start by noting that our method involving the Hessian is really a quadratic Taylorapproximation of (1) around a speciﬁc point β 0 .
For loss functions of the type (8), this approximationis equal to
[Equation]
where D ( w ( β 0 )) is a diagonal matrix with diagonal entries w ( β 0 )
where w ( β 0 ) i =
f (cid:48)(cid:48)
( x Ti β 0 ) and˜ y ( z ) i =
f (cid:48)
i ( z ) (cid:14) f
(cid:48)(cid:48)
i ( z ) − x Ti β 0 , whilst C ( β 0 ) is a constant with respect to β .
Suppose that we are on the lasso path at λ k and want to approximate c ( λ k +1 ) .
In this case, we simplyreplace f ( β ; X ) in (1) with Q ( β, ˆ β ( λ k )) , which leads to the following gradient approximation:
[Equation]
where w = w (cid:0) ˆ β ( λ k ) (cid:1) .
Unfortunately, we cannot use Algorithm 1 (Appendix B) to update
X T A k D ( w ) X
A k .
This means that we are forced to either update the Hessian directly at each step,which can be computationally demanding when |A k | is large and inefﬁcient when X is very sparse,or to approximate D ( w ) with an upper bound.
In logistic regression, for instance, we can use 1 / 4 assuch a bound, which also means that we once again can use Algorithm 1.
In our experiments, we have employed the following heuristic to decide whether to use an upper bound or compute the full Hessian in these cases: we use full updates at each step ifsparsity( X ) n/ max { n, p } < 10 − 3 and the upper bound otherwise.
Reducing the Impact of KKT Checks
The Hessian Screening Rule is heuristic, which meansthere may be violations.
This necessitates that we verify the KKT conditions after having reachedconvergence for the screened set of predictors, and add predictors back into the working set for whichthese checks fail.
When the screened set is small relative to p , the cost of optimization is often inlarge part consumed by these checks.
Running these checks for the full set of predictors always needsto be done once, but if there are violations during this step, then we need repeat this check, which isbest avoided.
Here we describe two methods to tackle this issue.
We employ a procedure equivalent to the one used in Tibshirani et al.
[10] for the working set strategy:we ﬁrst check the KKT conditions for the set of predictors singled out by the strong rule and then, if there are no violations in that set, check the full set of predictors for violations.
This works wellbecause the strong rule is conservative—violations are rare—which means that we seldom need torun the KKT checks for the entire set more than once.
If we, in spite of the augmentation of the rule, run into violations when checking the full set ofpredictors, that is, when the strong rule fails to capture the active set, then we can still avoid repeatingthe full KKT check by relying on Gap Safe screening: after having run the KKT checks and havefailed to converge, we screen the set of predictors using the Gap Safe rule.
Because this is a safe rule,we can be sure that the predictors we discard will be inactive, which means that we will not need toinclude them in our upcoming KKT checks.
Because Gap Safe screening and the KKT checks relyon exactly the same quantity—the correlation vector–we can do so at marginal extra cost.
To see howthis works, we now brieﬂy introduce
Gap Safe screening.
For details, please see Fercoq, Gramfort,and Salmon [6].
For the ordinary lasso ( (cid:96) 1 -regularized least squares), the primal (1) is P ( β ) =
12 (cid:107) y − Xβ (cid:107) 22 + λ (cid:107) β (cid:107) 1and the corresponding dual is
[Equation]
(9) subject to (cid:107) X T θ (cid:107) ∞ ≤ 1 .
The duality gap is then G ( β, θ ) =
P ( β ) − D ( θ ) and the relation betweenthe primal and dual problems is given by y
= λ
ˆ θ + X ˆ β , where ˆ θ is the maximizer to the dual problem(9).
In order to use Gap Safe screening, we need a feasible dual point, which can be obtained via dualpoint scaling, taking θ =
( y − Xβ )
(cid:14) max (cid:0) λ, (cid:107) X T ( y − Xβ )
(cid:107) ∞ (cid:1) .
The Gap Safe screening rule thendiscards the j th feature if | x Tj θ | < 1 − (cid:107) x j (cid:107)
(cid:112) 2 G ( β, θ ) /λ 2 .
Since we have computed X T ( y − Xβ )as part of the KKT checks, we can perform Gap Safe screening at an additional (and marginal) costamounting to O ( n ) + O ( p ) .
Since this augmentation beneﬁts the working set strategy too, we adopt it in our implementation ofthis method as well.
To avoid ambiguity, we call this version working+.
Note that this makes theworking set strategy quite similar to Blitz.
In Appendix F.3 we show the beneﬁt of adding this type ofscreening.
Approximate Homotopy
In addition to improved screening and warm starts, the Hessian alsoallows us to construct the regularization path adaptively via approximate homotopy [18].
In brief, the
Hessian screening rule allows us to choose the next λ along the path adaptively, in effect distributingthe grid of λ s to better approach the exact (homotopy) solution for the lasso, avoiding the otherwiseheuristic choice, which can be inappropriate for some data sets.
Experiments
Throughout the following experiments, we scale and center predictors with the mean and uncorrectedsample standard deviation respectively.
For the lasso, we also center the response vector, y , with themean.
To construct the regularization path, we adopt the default settings from glmnet : we use a log-spacedpath of 100 λ values from λ max to ξλ max , where ξ = 10 − 2 if p > n and 10 − 4 otherwise.
We stopthe path whenever the deviance ratio, 1 − dev / dev null , reaches 0.999 or the fractional decrease indeviance is less than 10 − 5 .
Finally, we also stop the path whenever the number of coefﬁcients ever tobe active predictors exceeds p .
We compare our method against working+ (the modiﬁed version of the working set strategy from
Tibshirani et al.
[10]), Celer [15], and Blitz [14].
We initially also ran our comparisons against
EDPP [9], the Gap Safe rule [6], and Dynamic Sasvi [8] too, yet these methods performed so poorlythat we omit the results in the main part of this work.
The interested reader may nevertheless consult
Appendix F.6 where results from simulated data has been included for these methods too.
We use cyclical coordinate descent with shufﬂing and consider the model to converge when theduality gap G ( β, θ ) ≤ εζ , where we take ζ to be (cid:107) y (cid:107) 22 when ﬁtting the ordinary lasso, and n log 2when ﬁtting (cid:96) 1 -regularized logistic regression.
Unless speciﬁed, we let ε = 10 − 4 .
These settings are standard settings and, for instance, resemble the defaults used in Celer.
For all of the experiments, weemploy the line search algorithm used in Blitz 3 .
The code used in these experiments was, for every method, programmed in C++ using the Armadillolibrary [19, 20] and organized as an R package via Rcpp [21].
We used the renv package [22]to maintain dependencies.
The source code, including a Singularity [23] container and its recipefor reproducing the results, are available in an anonymized repository at https://github.com/jt2gtwci/HessianScreeningRule .
Additional details of the computational setup are provided in
Appendix D.
Simulated Data Let X ∈ R n × p , β ∈ R p , and y ∈ R n be the predictor matrix, coefﬁcientvector, and response vector respectively.
We draw the rows of the predictor matrix independentlyand identically distributed from N (0 , Σ) and generate the response from N ( Xβ, σ 2 I ) with σ 2 =β T Σ β/ SNR, where SNR is the signal-to-noise ratio.
We set s coefﬁcients, equally spaced throughoutthe coefﬁcient vector, to 1 and the rest to zero.
In our simulations, we consider two scenarios: a low-dimensional scenario and a high-dimensionalscenario.
In the former, we set n = 10 000 , p = 100 , s = 5 , and the SNR to 1.
In the highdimensional scenario, we take n = 400 , p = 40 000 , s = 20 , and set the SNR to 2.
These SNRvalues are inspired by the discussion in Hastie, Tibshirani, and Tibshirani [24] and intend to cover themiddle-ground in terms of signal strength.
We run our simulations for 20 iterations.
From Figure 3, it is clear that the Hessian screening rule performs best, taking the least time in everysetting examined.
The difference is largest for the high-correlation context in the low-dimensionalsetting and otherwise roughly the same across levels of correlation.
The differences between the other methods are on average small, with the working+ strategy performing slightly better in the p > n scenario.
Celer and Blitz perform largely on par with one another,although Celer sees an improvement in a few of the experiments, for instance in logistic regressionwhen p > n .
Least-Squares 𝑛 = 10000, 𝑝 =
Least-Squares 𝑛 = 400, 𝑝 =
Logistic 𝑛 = 10000, 𝑝 =
Logistic 𝑛 = 400, 𝑝 = 0.
[Figure] [Figure]
l
[Figure]
Real Data
In this section, we conduct experiments on real data sets.
We run 20 iterations for thesmaller data sets studied and three for the larger ones.
For information on the sources of these datasets, please see Appendix E.
For more detailed results of these experiments, please see Appendix F.5.
Starting with the case of (cid:96) 1 -regularized least-squares regression, we observe that the Hessian screeningrule performs best for all ﬁve data sets tested here (Table 1), never taking more than half the timecompared to the runner-up, which in each case is the working+ strategy.
The difference is particularly large for the YearPredictionMSD data set, where our method is more than six times faster than theworking+ method.
In the case of (cid:96) 1 -regularized logistic regression, the Hessian method again performs best for mostof the examined data sets, for instance completing the regularization path for the madelon data setover six times faster than the working+ strategy.
The exceptions are arcene and the rcv1 data set, forwhich the working+ strategy performs best out of the four methods.
[Table]
We have provided additional results related to the effectiveness of our method in Appendix F.
Discussion
In this paper, we have presented the Hessian Screening Rule: a new heuristic predictor screeningrule for (cid:96) 1 -regularized generalized linear models.
We have shown that our screening rule offers largeperformance improvements over competing methods, both in simulated experiments but also in themajority of the real data sets that we study here.
The improved performance of the rule appearsto come not only from improved effectiveness in screening, particularly in the high-correlationsetting, but also from the much-improved warm starts, which enables our method to dominate inthe n (cid:29) p setting.
Note that although we have focused on (cid:96) 1 -regularized least-squares and logisticregression here, our rule is applicable to any composite objective for which the differentiable part istwice-differentiable.
One scenario in which the rule appears to be less successful is the n ≈ p context with very sparsedesign matrices. Here, operations involving the Hessian become too expensive relative to the beneﬁtsit offers.
In these situations, which are easy to identify, the best choice instead appears to be theworking set strategy.
Note also that we, in this paper, focus entirely on the lasso path .
The Hessian Screening Rule is a sequential rule and may therefore not prove optimal when solving for a single λ ,in which case a dynamic strategy such as Celer and Blitz likely performs better.
With respect to the relative performance of the working set strategy, Celer, and Blitz, we note that ourresults deviate somewhat from previous comparisons [15, 14].
We speculate that these differencesmight arise from the fact that we have used equivalent implementations for all of the methods andfrom the modiﬁcation that we have used for the working set strategy.
Many avenues remain to be explored in the context of Hessian-based screening rules and algorithms,such as developing more efﬁcient methods for updating of the Hessian matrix for non-least-squaresobjectives, such as logistic regression and using second-order information to further improve theoptimization method used.
Other interesting directions also include adapting the rules to morecomplicated regularization problems, such as the fused lasso [25] and SLOPE [26].
Finally, we donot expect there to be any negative societal consequences of our work given that it is aimed solely atimproving the performance of an optimization method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesection 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See the supplementarymaterial.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See section 4.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how they werechosen)?
[Yes] See section 4 as well as the supplementary details and the references to
existing data sets for discussions on test and training splits.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the supplement
and section 4.
(b) Did you mention the license of the assets?
[Yes] A LICENSE.md ﬁle has been included
along with the code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See section 4.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All data we used has been made available in the public domain.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] To the best of our knowledge, it does not.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Not applicable to our work.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Not applicable to our work.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Not applicable to our work.