Beyond L1: Faster and Better Sparse Models with skglm
Abstract
We propose a new fast algorithm to estimate any sparse generalized linear modelwith convex or non-convex separable penalties.
Our algorithm is able to solveproblems with millions of samples and features in seconds, by relying on coordinatedescent, working sets and Anderson acceleration.
It handles previously unaddressedmodels, and is extensively shown to improve state-of-art algorithms.
We providea ﬂexible, scikit-learn compatible package, which easily handles customizeddataﬁts and penalties.
Introduction
Sparse generalized linear models play a central role in modern machine learning and signal processing.
The Lasso (Tibshirani, 1996) and its derivatives (Zou and Hastie, 2005; Ng, 2004; Candes et al., 2008;
Simon et al., 2013) have found numerous successful applications to large scale tasks in genomics(Ghosh and Chinnaiyan, 2005), vision (Mairal, 2010), or neurosciences (Strohmeier et al., 2016).
Thisimpact was made possible by two key factors: efﬁcient algorithms and software implementations.
State-of-the-art algorithms for “smooth + non-smooth separable” problems predominantly rely oncoordinate descent (CD, Tseng and S.Yun 2009; Nesterov 2012), which, when it can be applied,is more efﬁcient than full gradient methods (Richtárik and Taká ˇ c, 2014, Sec. 6.1).
Coordinatedescent can even be improved with Nesterov-like acceleration, to obtain improved convergence rates(Lin et al., 2014; Fercoq and Richtárik, 2015).
However, these better rates may fail to reﬂect inpractical accelerations.
On the contrary, Bertrand and Massias (2021) relied on Anderson acceleration(Anderson, 1965) to provide both better rates and practical acceleration for coordinate descent.
Even with efﬁcient algorithms such as coordinate descent, the practical use of sparsity hits a computational barrier for problems with more than millions of features (Le Morvan and Vert, 2018).
Multiple techniques have been proposed to make coordinate descent scale to huge problems.
Notably,algorithms can be accelerated by reducing the number of variables to optimize over, using screeningrules or working sets.
Screening rules discard features from the problem in advance (El Ghaouiet al. 2010; Bonnefoy et al. 2015) or dynamically (Fercoq et al., 2015; Ndiaye et al., 2017).
On theother side, working sets (Johnson and Guestrin, 2015; Massias et al., 2018) iteratively solve largersubproblems and progressively include variables identiﬁed as relevant.
For the Lasso and a few convex models, coordinate descent has been broadly disseminated topractitioners in off-the-shelf packages such as glmnet (Friedman et al., 2007) or scikit-learn(Pedregosa et al., 2011).
More recently, celer , a state-of-the-art convex working set algorithm(Massias et al., 2020) allowed for successful applications of the Lasso in large scale problems inmedicine (Reidenbach et al., 2021;
Kim et al., 2021) or seismology (Muir and Zhan, 2021).
Yet the Lasso is limited: non-convex sparse models enjoy better theoretical and empirical properties(Breheny and Huang, 2011; Soubies et al., 2015).
As illustrated in Figure 1, they yield sparser
[Figure]
MCP, γ =(cid:96) 1 /(cid:96) 2 /
[Figure]
Non-convex sparse penalties behave better than the L1 norm.
Due to their lower bias, they achieve perfect support recovery, lower prediction error and their optimal regularization strength λ in estimation (top) and prediction (bottom) correspond.
solutions than convex penalties and mitigate the intrinsic Lasso bias.
Yet, they have not so oftenbeen applied to huge scale applications.
This is mostly an algorithmic barrier: while coordinatedescent can be applied to non-convex penalties (Breheny and Huang, 2011; Mazumder et al., 2011;
Bolte et al., 2014), screening rules and working sets are heavily dependent on convexity or quadraticdataﬁts (Rakotomamonjy et al., 2019, 2022).
In this work, we solve this issue by designing a state-of-the-art generic algorithm to solve a widerange of sparse generalized linear models.
The contributions are the following:
• We propose a non-convex converging working set algorithm relying on Anderson acceleratedcoordinate descent.
For a speciﬁc class of non-convex penalties, we show:(a)
Convergence of the proposed working set algorithm (Proposition 5).(b)
Support identiﬁcation of coordinate descent (Proposition 10).(c)
Local convergence rates for the Anderson extrapolation (Proposition 13).
• We provide an extensive experimental comparison and we show state-of-the-art improvements on a wide range of convex and non-convex problems.
In addition we release an
efﬁcient and modular python implementation, with a scikit-learn API, for practitionersto apply non-convex penalties to large scale problems.
Framework and proposed algorithm
2.1 Problem setting
In this paper, we consider problems of the form:
[Equation]
where F is smooth, and the functions g j are proper and lower semicontinuous but not necessarilyconvex, whose proximal operator can be computed exactly.
We write g = (cid:80) j g j .
Instances of
Problem (1) include convex estimators: the Lasso, the elastic net, the sparse logistic regression,the dual of SVM with hinge loss.
They also include non-convex penalties: (cid:96) 0 .
5 and (cid:96) 2 /
penalties(Foucart and Lai, 2009), the minimax concave penalty (MCP, Zhang 2010) or SCAD (Zhang, 2010),both with regression and classiﬁcation losses.
Formally, the assumptions are the following.
Assumption 1.
f : R p
→ R is convex and differentiable and for all j ∈ [ p ] , the restriction of ∇ j fto the j -th coordinate is L j -Lipschitz: for all ( x, h ) ∈ R p × R , |∇ j f ( x + he j ) − ∇ j f ( x ) | ≤ L j | h | .
Assumption 2.
For any j ∈ [ p ] , g j : R → R is proper, closed, and lower bounded.
Following Attouch and Bolte 2009; Bolte et al.
2014 we focus on ﬁnding a critical point of Φ .
Deﬁnition 3.
Using the Fréchet subdifferential (Kruger, 2003), a critical point x ∈ R p is a pointwhich satisﬁes −∇ f ( x ) ∈ ∂g
( x ) .
Assumptions 1 and 2 are usual, and, under boundedness of the iterates, ensure convergence offorward-backward and coordinate descent algorithms to a critical point (Attouch et al.
2013, Thm5.1, Bolte et al.
2014, Thm. 3.1).
In addition, our work focuses on the case where g j
Deﬁnition 4 (Generalized support) .
The generalized support of β ∈ R p is the set of indices j ∈ [ p ] that g j is differentiable at β j : gsupp( β ) =
{ j ∈ [ p ] : ∂g j ( β j ) is a singleton } Example 1.
With λ, γ > 0 , MCP is deﬁned as: MCP λ,γ ( x ) (cid:55)→
[Equation]
Penalties such as (cid:96) 1 , (cid:96) q ( 0 < q < 1) , MCP or SCAD are only not differentiable at 0, and thiscorresponds to the usual notion of sparsity.
But Deﬁnition 4 goes beyond sparsity and extends toestimators such as SVM, where g j = ι [0 ,C ] and the generalized support corresponds to supportvectors { j ∈ [ p ] : β j = 0 or β j = C } .
The generalized support of a critical point is usually ofcardinality much smaller than p , and its knowledge makes the problem easier and faster to solve.
Ourworking set algorithm exploit this structure in order to converge faster.
2.2 Proposed algorithm
The proposed algorithm exploits two main ideas:
• A working set strategy, able to handle a large class of convex and non-convex penalties(Algorithm 1).• An Anderson accelerated coordinate descent for non-convex problems (Algorithm 2).
Thebuilding blocks of Algorithm 2, coordinate descent ( CD , Algorithm 3) and Anderson extrapolation ( Anderson , Algorithm 4), can be found in Appendix A.
To avoid wasting computation on features outside the generalized support, working set algorithmsiteratively select a subset of coordinates deemed important (the working set ), and solve Problem (1) restricted to them.
The key question is thus the notion of important features.
Stemming from Deﬁnition 3, we rank features by their violation of the optimality condition: score ∂j =dist( −∇ j f ( β ) , ∂g j ( β )) .
For example, the MCP Fréchet subdifferential at 0 is ∂g j (0) =
[ − λ, λ ] ,and the proposed score readsscore
[Equation]
= (cid:26)
[Equation]
∇
[Equation]
β
[Equation] − [Equation] To control the working set growth, we use score ∂j to rank the features.
Then, with n k =max( n k − 1 , 2 | gsupp( β ( t ) ) | ) we take the n k largest of them in the working set, while retainingfeatures currently in the working set.
This growth quickly rises to the unknown size of the generalizedsupport while avoiding overshooting, as backed up by recent theory in Ndiaye and Takeuchi (2021).
Algorithm input : X, β ∈ R p , n out ∈ N , n in ∈ N , ws _ size ∈ N , (cid:15) >1 for t = 1 , . . .
, n out do score = (cid:0) dist ( −∇ j f ( β ) , ∂g j ( β j )) (cid:1)
j ∈
[ p ] ws _ size = max(ws _ size , 2 × | gsupp( β ) | ) //
ws _ size features with largest scores ws = arg
topK(score , K = ws _ size) if max j ∈ [ p ] dist ( −∇ j f ( β ) , ∂g j ( β j )) ≤ (cid:15) then stop else // accelerated CD on working set β ←
inner_solver ( X, β, ws , n in , (cid:15) ) 8 return β input : X, β (0)
∈ R p , ws ⊂ [ p ] , n in , (cid:15) , M =1 for k = 1 , . . .
, n in do β ( k ) ← CD ( X, β ( k − 1) , Xβ, ws) //
Algo.
(3) if k mod M = 0 then // Algo.
(4) , O ( M 2 | ws | + M 3 ) β extrws ← Anderson ( β ( k − M ) ws ,
, β ws ) // test objective O ( n | ws | )
Proposition 5.
Let W t be the t -th working set.
Suppose that Algorithm 2 converges toward a criticalpoint, and for all t ≥ 0 , W t ⊂ W t +1 , then the iterates of Algorithm 1 converges towards a critical 98 point of Problem (1) .
Proof of Proposition 5 can be found in Appendix B.1.
The second key ingredient to our algorithm is touse state-of-the-art Anderson accelerated coordinate descent for non-convex problems.
In Section 2.3we show that coordinate descent yields ﬁnite time support identiﬁcation for a large class of nonconvex problems (Proposition 10), which leads to acceleration (Proposition 13).
As experimentsdemonstrate in Section 3, this rate allows our algorithm to surpass state-of-the-art solvers.
2.3 Anderson accelerated coordinate descent analysis for α -semi-convex penalties
We now turn to our main technical contributions: we show that Algorithm 2 achieves ﬁnite timesupport identiﬁcation (Proposition 10) of the generalized support (Deﬁnition 4) for speciﬁc class ofnon-smooth non-convex penalties (Assumption 6), which includes the MCP (Proposition 7).
Basedon Proposition 10, we are able to derive convergence rates for Anderson acceleration Proposition 13.
We study our inner solver (Algorithm 2); for convenience we still refer to β and X for their counterparts restricted to the working set.
The following assumptions are required.
Assumption 6 ( α -semi-convex) .
For all j ∈ [ p ] g j /L
j is α -semi-convex, i.e., g j /L j + α (cid:107)·(cid:107) 2 / 2 isconvex, with α < 1 .
Note that in statistics, the admissible value range of hyperparameters for MCP and SCAD are dataﬁtdependent, (see Breheny and Huang 2011, Sec. 2.1, normalized columns and γ > 1 = 1 / (cid:107) X : j (cid:107)
=1 /L j or Soubies et al.
2015, Eq. 4.2) and yields α -semi-convexity for MCP and SCAD 1 .
Proposition 7 ( α -semi-convexity of MCP) .
Let MCP λ,γ ( x ) (cid:44)
[Equation]
[Equation]
if | x | > γλ .
If γ > 1 /L j , then MCP λ,γ /L
j is α -semi-convex
( i.e., Assumption 6 holds).
Note that Assumption 6 does not hold for the (cid:96) q -penalties ( 0 < q < 1 ), for which we propose analternative in Appendix C.
Assumption 8 (Existence) .
Problem (1) admits at least one critical point.
In Proposition 10, convergence of Algorithm 2 toward a critical point ˆ β is assumed, and the followingassumption is made on this critical point.
Assumption 9 (Non degeneracy) .
The considered critical point ˆ β ∈ R p is non-degenerated: for allj / ∈ gsupp(ˆ β ) , −∇ f j (ˆ β ) ∈ interior( ∂g j (ˆ β j )) .
Assumption 9 is a generalization of qualiﬁcation constraints (Hare and Lewis, 2007, Sec. 1), and isusual in the machine learning literature (Zhao and Yu, 2006; Bach, 2008; Vaiter et al., 2015).
Forthe (cid:96) 1 -norm, if the entries of the design matrix X are drawn from a i.i.d normal distribution, then
Assumption 9 holds with high probability (Candes and Tao, 2005; Rudelson and Vershynin, 2008).
Equipped with the previous assumptions we show that coordinate descent achieves model identiﬁcation for this class of non-convex problems.
Proposition 10 (Model identiﬁcation of CD) .
Suppose
Assumptions 1, 2, 6 and 8 hold
The sequence ( β ( k ) )
k ≥ 0 generated by coordinate descent (Algorithm 2 without extrapola- tion) converges toward a critical point ˆ β .
Assumption 9 holds for ˆ β .
Then, Algorithm 2 (without extrapolation) identiﬁes the model in ﬁnitely many iterations: there exists
K > 0 such that for all k ≥ K , β ( k ) S c
= ˆ β S c
In other words, for k large enough, β ( k ) shares the generalized support of ˆ β .
The identiﬁcationproperty was proved for a proximal gradient descent algorithm in the non-convex case (Liang et al.
under the assumption that the non-smooth function g is partly smooth (Lewis, 2002).
Forourselves, Proposition 10 not rely on the partly smooth assumption to ensure identiﬁcation property.
Authors are not aware of previous identiﬁcation results for coordinate descent in the non-convex case.
In addition, if f and g are locally regular on the generalized support at the considered critical point,our algorithm enjoys local acceleration when combined with Anderson extrapolation (Proposition 13).
Assumption 11 (Locally C 3 ) .
For all j ∈ S (cid:44) gsupp(ˆ β ) , g j is locally C 3 around ˆ β j , and f is locally
C 3 around ˆ β
[Table]
Assumption 11 on the function f is mild and holds for usual machine learning dataﬁtting terms. Assumption 11 on the functions g j , j ∈ S , is stronger: for instance, for the MCP, it implies ˆ β j (cid:54) = γλfor all j ∈ S . However this assumption is standard in the literature, see Liang et al.
2016, Sec.
Assumption 12.
(Local strong convexity)
The Hessian of f at the considered critical point ˆ β ∈ R p ,restricted to its generalized support S , is positive deﬁnite, i.e., ∇ 2 S , S f
( ˆ β ) + ∇ 2 S , S g ( ˆ β ) (cid:31) 0 .
Assumption 12 requires local strong convexity restricted to the generalized support S , which isstandard in the MCP / SCAD literature (Breheny and Huang 2011, Section 4.1) and is usual to derivelocal linear rates of convergence (Liang et al., 2016, Section 3.3).
For instance, for the Lasso, if theentries of the design matrix X are drawn from a continuous distribution, then Assumption 12 holdswith probability one (Tibshirani, 2013, Lemma 4).
Proposition 13.
Consider a critical point ˆ β and suppose1.
Assumptions 1, 2 and 8 hold.
The functions f and g j , j ∈ [ p are piecewise quadratic (which is the case
for
the MCP
regression
The sequence ( β ( k ) )
k ≥ 0 generated by anderson accelerated coordinate descent with updates
from 1 to p and p to 1 (
Algorithm 2 with extrapolation) converges to a critical point ˆ β .
Assumptions 9, 11 and 12 hold for ˆ β .
Then there exists K ∈ N , and a C 1 function ψ : R |S| → R |S| such that, for all k ∈ N , k ≥ K :
[Equation]
[Equation] , for all j ∈
[Equation]
Let T (cid:44) J ψ
(ˆ β ) , H (cid:44) ∇ 2 S , S f
(ˆ β ) + ∇ 2 S , S g (ˆ β ) , ζ (cid:44) (1 − (cid:112) 1 − ρ ( T )) / (1 + (cid:112) 1 − ρ ( T )) and
B (cid:44) ( T − Id) (cid:62) ( T − Id) .
Then ρ ( T ) < 1 and the iterates of Anderson extrapolation enjoy localaccelerated convergence rate:
[Equation]
The proof can be found in Appendix B.5.
Related work.
Most Anderson acceleration convergence results are shown for quadratic objectives forspeciﬁc algorithms: gradient descent (Golub and Varga, 1961; Anderson, 1965), ADMM (Poon and
Liang, 2019), coordinate descent (Bertrand et al., 2020).
Outside of the quadratic case, convergenceresults are usually signiﬁcantly weaker (Scieur et al., 2016; Sidi, 2017; Brezinski et al., 2018; Maiand Johansson, 2019;
Ouyang et al., 2020).
Regarding the smooth non-convex case, Wei et al.(2021) proposed a stochastic Anderson acceleration and proved convergence towards a critical point.
Proposition 13 generalizes Scieur et al.
(2020, Prop 2.1) and Bertrand and Massias (2021, Prop. 4) tothe proximal convex and α -semi-convex cases.
To our knowledge this is one of the ﬁrst quantitativeresults for Anderson acceleration in a non-convex setting.
2.4 Comparison with existing work
In this section we compare our contribution to existing algorithms and implementations, which aresummarized in Table 1.
Huge scale refers to the fact that the algorithm can run on problems with millions of variables.
Non-convex tells if the algorithm handles non-convex penalties.
Modularindicates that it is easy to add a new model, through a different dataﬁtting term or penalty.
The packages glmnet (Friedman et al., 2010), scikit-learn (Pedregosa et al., 2011) andlightning (Blondel and Pedregosa, 2016) implement coordinate descent (cyclic or random).
Theyrely on compiled code such as Fortran or Cython, making it very difﬁcult to implement new models 2or faster algorithms like working set 3 .
They do not handle non-convex penalties.
More recent algorithms such as blitz (Johnson and Guestrin, 2015), celer (Massias et al., 2018),picasso (Ge et al., 2019) or fireworks (Rakotomamonjy et al., 2022)
use working set strategies.celer and blitz are state-of-the-art algorithms for the Lasso, but their score to prioritize featuresrelies on duality.
fireworks extends blitz to some non-convex penalties (writing as difference ofconvex functions), with score ﬁreworks j = dist( −∇ j f ( β ) , ∂g j (0)) .
Yet this rule does not consider thesubdifferential of g at the current point, but at 0, which is a coarse information.
Finally, fireworksdoes not provide accelerated convergence rates and does not come with a public implementation.picasso (Ge et al., 2019) lacks modularity (penalties are hardcoded), and the solver is not suited forhuge scale (it does not support large sparse matrices).
Deng and Lan (2019) proposed an algorithmbased on inertially accelerated coordinate descent, which fails to provide practical speedups accordingto Bertrand and Massias (2021).
Contrary to these algorithms, ours is generic and relies only on the knowledge of ∇ f and prox g .
Forany new penalty, this information can be written in a few lines of Python code, compiled with numba(Lam et al., 2015) for speed efﬁciency.
We therefore improve of state-of-the-art algorithms in theconvex case, and generalize to virtually any dataﬁt and penalty, even nonconvex.
Experiments
Our package relying on numpy and numba (Lam et al., 2015; Harris et al., 2020) is attached in thesupplementary material and will be open-sourced upon publication under BSD 3-Clause License.
Weuse datasets from libsvm 4 (Fan et al.
2008, see table 2).
We compare multiple algorithms to solve popular Machine Learning and inverse problems: Lasso,
Elastic net, multitask sparse regression, MCP regression.
The compared algorithms are the following:
• scikit-learn (Pedregosa et al., 2011), which implements coordinate descent in Cython,• celer (Massias et al., 2020), which combines working sets, screening rules, coordinatedescent, and Anderson acceleration in the dual, in Cython,• blitz (Johnson and Guestrin, 2015), which combines working sets with prox-Newtoniterations (Lee et al., 2012) in C++,• coordinate descent (CD, Tseng and S.Yun 2009),• skglm (Algorithm 1, ours), using M = 5 iterates for the Anderson extrapolation.
Other solvers.
Experiment per experiment, there exist niche solvers (such as aggressive Gap
Safe
Rules, Ndiaye et al. 2020).
Since our goal is a general purpose algorithm able to deal with manymodels, we do not include them in the comparison.
In addition, we focus on solving a singleinstance of Problem (1), rather than a regularization path ( i.e., a sequence of problems for multipleregularization strengths).
As glmnet is designed to compute regularization paths, we could notinclude it in the comparison.
The reader can refer to Johnson and Guestrin (2015, Fig. 4) or Figure 8in Appendix E for comparisons on single optimization problems with glmnet .
glmnet and additionalalgorithms are discussed in Appendix E.
How to do a fair comparison between solvers?
To plot the convergence curves, we use thebenchopt 5 benchmarking package (Moreau et al., 2022).
In order to automate and reproduceoptimization benchmarks it treats solvers as black boxes. It launches them several times withincreasing maximum number of iterations, and stores the resulting objective values and times to reachit.
As each point on a solver curve is obtained in a different run, the curves are not monotonic, andthere may be several points corresponding to the same time.
This merely reﬂects the variability in {+blitz celer skglm (ours) sklearn [Figure] [Figure] λ max /0 .2 .
+}
{+.λ max /  λ max / [Figure] dd a  
+}
Time (s)
[Figure]
l
Time (s)
Time (s) solvers running time across runs; we refer to Figure 11 in Appendix E.7 for the inevitability of thisphenomenon with black box solvers.
3.1 Convex problems
Lasso.
In Figure 2 we compare solvers for the Lasso: ( f = 12 n (cid:107) y − X
·(cid:107) 2 , g j = λ |·| ).
Weparametrize λ
as a fraction of λ max
= (cid:107) X (cid:62) y (cid:107) ∞ /n , smallest regularization strength for which ˆ β = 0 .
For large scale datasets ( rcv1 , news20 ), skglm yields performances better or similar to the state-ofthe-art algorithms blitz and celer .
For huge scale datasets ( kdda and url ), skglm yields signiﬁcantspeedups over them.
The improvement over the popular scikit-learn can be of two orders ofmagnitude.
Thus, while dealing with many more models, our algorithm still yields state-of-the-artspeed for basic ones .
Elastic net.
Our approach easily generalizes to other problems, such as the elastic net ( f =12 n (cid:107) y − X ·(cid:107) 2 , g j = λ ( ρ |·| + 1 − ρ 2 ( · ) 2 ) ).
Figure 3 shows the duality gap as a function of time 242 for skglm (ours), sklearn , and our numba implementation of coordinate descent.
The proposedalgorithm is orders of magnitude faster than scikit-learn and vanilla coordinate descent, inparticular for large datasets and low regularization parameter values ( ﬁnance , λ max / 1000 ).
Note thatblitz does not implement a solver for the elastic net.
Many Lasso solvers would easily handle theelastic net, but relying on Cython/C++ code makes the implementation time-consuming.
By contrast,it takes 40 lines of code to deﬁne an (cid:96) 1 + (cid:96) 2 -squared penalty with our implementation.
An additionalexperiment on the dual of SVM with hinge loss is in Appendix E.4.
3.2 Non-convex problems
In this subsection we propose a comparison on two non convex problems.
{+sklearn Time (s) 
 [Figure]+}
ﬁ n a n ce λ max / Time (s) λ max / Time (s) λ max /
[Figure]
MCP regression. MCP regression is Problem (1) with f = 12 n (cid:107) y − X ·(cid:107) 2 , g j = MCP λ,γ forγ > 1 .
As usual for this problem, we scale the columns of X to have norm √ n .
On Figure 5,we compare our algorithm to picasso on a dense dataset ( n = 1000 , p = 5000 ); as this packagedoes not support large sparse design matrices, for the rcv1 dataset we use an iterative reweighted L1 algorithm (Candes et al., 2008).
Since the derivative of the MCP vanishes for values bigger thanλγ , this approach requires solving weighted Lassos with some 0 weights.
Up to our knowledge, ouralgorithm is the only efﬁcient one with such a property.
Our algorithm handles problems of largesize, converges to a critical point, and, due to its progressive inclusion of features, is able to reach asparser critical point than it competitors.
Application to neuroscience To demonstrate the usefulness of our algorithm for practitioners,we apply it to the magneto-/electroencephalographic (M/EEG) inverse problem.
It consists inreconstructing the spatial cortical current density at the origin of M/EEG measurements made atthe surface of the scalp.
Non-convex penalties (Strohmeier et al., 2015) exhibit several advantagesover convex ones (Gramfort et al., 2013): they yield sparser physiologically-plausible solutions andmitigate the (cid:96) 1 amplitude bias.
Here the setting is multitask: Y ∈ R n × T and thus we use blockpenalties (details in Appendix D).
We use real data from the mne software (Gramfort et al., 2014);the experiment is a right auditory stimulation, with two expected neural sources to recover in eachauditory cortex.
In Figure 4, while the (cid:96) 2 , 1 penalty fails at localizing one source in each hemisphere,the non-convex penalties recover the correct locations.
This emphasizes on the critical need for fastsolvers for non-convex sparse penalties as well as our algorithm’s ability to handle the latter.
Ablation study.
To evaluate the inﬂuence of the two components of Algorithm 1, an ablation study(Figure 6 and Figure 10 in Appendix E) is performed.
Four algorithms are compared: with/withoutworking sets and with/without Anderson acceleration.
Figure 6 represents the duality gap of the Lassoas a function of time for multiple datasets and values of the regularization parameters λ (parametrizedas a fraction of λ max ).
First, Figure 6 shows that working sets always bring signiﬁcant speedups.
Then,when combined with working set, Anderson acceleration bring signiﬁcant speed-ups, especially forhard problems with low regularization parameters.
An interesting observation is that on large scale coordinate descent picasso skglm (ours)  o b j ec t i v e λ max /  λ max /  λ max / Time (s) [Figure] o p t .
c o nd . Time (s) 10 − Time (s) 10 − [Figure]
descent iterative reweighted L skglm (ours)
[Figure]
0 .o
b j ec t i v e λ max /0 .0 .λ
max /0 .0 .λ
max /0 .0 .0 .
Time
[Figure]
o p t .
c o nd .
Time (
[Figure]
[Figure]
Time
[Figure]
[Figure] [Figure] datasets ( not bring acceleration. This highlights the importance of combining acceleration
Conclusion and broader impact.
In this paper, we have proposed an accelerated versatile algorithm for a speciﬁc class of non-smooth non-convex problems.
Based on working sets, coordinatedescent and Anderson acceleration, we have improved state-of-the-art on convex problems, and handled previously out-of-reach problems.
Thorough experiments demonstrated the speed and interestof our approach.
A limitation of this work is the considered function class ( α -semi-convex), whichcan be seen as restrictive.
One possible extension would be weakly convex functions (Davis and Drusvyatskiy, 2019, Sec. 1).
We deeply believe that the high quality code provided will beneﬁt topractitioners, and ease the use of non-convex penalties for real world problems, from neuroimaging togenomics.
We proposed an optimization algorithm and do not see potential negative societal impacts.