Beyond L1: Faster and Better Sparse Models with skglm
Abstract
We propose a new fast algorithm to estimate any sparse generalized linear modelwith convex or non-convex separable penalties.
Our algorithm is able to solveproblems with millions of samples and features in seconds, by relying on coordinatedescent, working sets and Anderson acceleration.
It handles previously unaddressedmodels, and is extensively shown to improve state-of-art algorithms.
We providea ﬂexible, scikit-learn compatible package, which easily handles customizeddataﬁts and penalties.
Introduction
Sparse generalized linear models play a central role in modern machine learning and signal processing.
The Lasso (Tibshirani, 1996) and its derivatives (Zou and Hastie, 2005; Ng, 2004; Candes et al., 2008;
Simon et al., 2013) have found numerous successful applications to large scale tasks in genomics(Ghosh and Chinnaiyan, 2005), vision (Mairal, 2010), or neurosciences (Strohmeier et al., 2016).
Thisimpact was made possible by two key factors: efﬁcient algorithms and software implementations.
State-of-the-art algorithms for “smooth + non-smooth separable” problems predominantly rely oncoordinate descent (CD, Tseng and S.Yun 2009; Nesterov 2012), which, when it can be applied,is more efﬁcient than full gradient methods (Richtárik and Taká ˇ c, 2014, Sec. 6.1).
Coordinatedescent can even be improved with Nesterov-like acceleration, to obtain improved convergence rates(Lin et al., 2014; Fercoq and Richtárik, 2015).
However, these better rates may fail to reﬂect inpractical accelerations.
On the contrary, Bertrand and Massias (2021) relied on Anderson acceleration(Anderson, 1965) to provide both better rates and practical acceleration for coordinate descent.
Even with efﬁcient algorithms such as coordinate descent, the practical use of sparsity hits a computational barrier for problems with more than millions of features (Le Morvan and Vert, 2018).
Multiple techniques have been proposed to make coordinate descent scale to huge problems.
Notably,algorithms can be accelerated by reducing the number of variables to optimize over, using screeningrules or working sets.
Screening rules discard features from the problem in advance (El Ghaouiet al. 2010; Bonnefoy et al. 2015) or dynamically (Fercoq et al., 2015; Ndiaye et al., 2017).
On theother side, working sets (Johnson and Guestrin, 2015; Massias et al., 2018) iteratively solve largersubproblems and progressively include variables identiﬁed as relevant.
For the Lasso and a few convex models, coordinate descent has been broadly disseminated topractitioners in off-the-shelf packages such as glmnet (Friedman et al., 2007) or scikit-learn(Pedregosa et al., 2011).
More recently, celer , a state-of-the-art convex working set algorithm(Massias et al., 2020) allowed for successful applications of the Lasso in large scale problems inmedicine (Reidenbach et al., 2021;
Kim et al., 2021) or seismology (Muir and Zhan, 2021).
Yet the Lasso is limited: non-convex sparse models enjoy better theoretical and empirical properties(Breheny and Huang, 2011; Soubies et al., 2015).
As illustrated in Figure 1, they yield sparser
Lasso
MCP, γ =(cid:96) 1 /(cid:96) 2 /
[Figure]
c
[Figure] r [Figure]
solutions than convex penalties and mitigate the intrinsic Lasso bias.
Yet, they have not so oftenbeen applied to huge scale applications.
This is mostly an algorithmic barrier: while coordinatedescent can be applied to non-convex penalties (Breheny and Huang, 2011; Mazumder et al., 2011;
Bolte et al., 2014), screening rules and working sets are heavily dependent on convexity or quadraticdataﬁts (Rakotomamonjy et al., 2019, 2022).
In this work, we solve this issue by designing a state-of-the-art generic algorithm to solve a widerange of sparse generalized linear models.
The contributions are the following:
• We propose a non-convex converging working set algorithm relying on Anderson acceleratedcoordinate descent.
For a speciﬁc class of non-convex penalties we show:(a)
Convergence of the proposed working set algorithm (Proposition 5).
(b) Support identiﬁcation of coordinate descent (Proposition 10).
(c) Local convergence rates for the Anderson extrapolation (Proposition 13).
• We provide an extensive experimental comparison and we show state-of-the-art improve- ments on a wide range of convex and non-convex problems.
In addition we release an
efﬁcient and modular python implementation, with a scikit-learn API, for practitioners to apply non-convex penalties to large scale problems.
Framework and proposed algorithm2.1 Problem setting
In this paper, we consider problems of the form:
[Equation]
where F is smooth, and the functions g j are proper and lower semicontinuous but not necessarilyconvex, whose proximal operator can be computed exactly.
We write g = (cid:80) j g j .
Instances of
Problem (1) include convex estimators: the Lasso, the elastic net, the sparse logistic regression,the dual of SVM with hinge loss.
They also include non-convex penalties: (cid:96) 0 .
5 and (cid:96) 2 /
penalties(Foucart and Lai, 2009), the minimax concave penalty (MCP, Zhang 2010) or SCAD (Zhang, 2010),both with regression and classiﬁcation losses.
Formally, the assumptions are the following.
Assumption 1.
f : R p
→ R is convex and differentiable and for all j ∈ [ p ] , the restriction of ∇ j fto the j -th coordinate is L j -Lipschitz: for all ( x, h ) ∈ R p × R , |∇ j f ( x + he j ) − ∇ j f ( x ) | ≤ L j | h | .
Assumption 2.
For any j ∈ [ p ] , g j : R → R is proper, closed, and lower bounded.
Following Attouch and Bolte 2009; Bolte et al.
2014 we focus on ﬁnding a critical point of Φ .
Deﬁnition 3.
Using the Fréchet subdifferential (Kruger, 2003), a critical point x ∈ R p is a pointwhich satisﬁes −∇ f ( x ) ∈ ∂g
( x ) .
Assumptions 1 and 2 are usual, and, under boundedness of the iterates, ensure convergence offorward-backward and coordinate descent algorithms to a critical point (Attouch et al.
2013, Thm5.1, Bolte et al.
2014, Thm. 3.1).
In addition, our work focuses on the case where g j ’s presentnon-differentiability points, leading to the following extended notion of sparsity.
Deﬁnition 4 (Generalized support) .
The generalized support of β ∈ R p is the set of indices j ∈ [ p ]such that g j is differentiable at β j : gsupp( β ) =
{ j ∈ [ p ] : ∂g j ( β j ) is a singleton } .
Penalties such as (cid:96) 1 , (cid:96) q ( 0 < q < 1) , MCP or SCAD are only not differentiable at 0, and thiscorresponds to the usual notion of sparsity.
But Deﬁnition 4 goes beyond sparsity and extends toestimators such as SVM, where g j = ι [0 ,C ] and the generalized support corresponds to supportvectors { j ∈ [ p ] : β j = 0 or β j = C } .
The generalized support of a critical point is usually ofcardinality much smaller than p , and its knowledge makes the problem easier and faster to solve.
Ourworking set algorithm exploit this structure in order to converge faster.
2.2 Proposed algorithm
The proposed algorithm exploits two main ideas:
• A working set strategy, able to handle a large class of convex and non-convex penalties (Algorithm 1).• An Anderson accelerated coordinate descent for non-convex problems (Algorithm 2).
Thebuilding blocks of Algorithm 2, coordinate descent ( CD , Algorithm 3) and Anderson extrapolation ( Anderson , Algorithm 4), can be found in Appendix A.
To avoid wasting computation on features outside the generalized support, working set algorithmsiteratively select a subset of coordinates deemed important (the working set ), and solve Problem (1) restricted to them.
The key question is thus the notion of important features.
Stemming from Deﬁnition 3, we rank features by their violation of the optimality condition: score ∂j =dist( −∇ j f ( β ) , ∂g j ( β )) .
For example, the MCP Fréchet subdifferential at 0 is ∂g j (0) =
[ − λ, λ ] ,and the proposed score reads
[Equation]
To control the working set growth, we use score ∂j to rank the features.
Then, with n k =max( n k − 1 , 2 | gsupp( β ( t ) ) | ) we take the n k largest of them in the working set, while retainingfeatures currently in the working set.
This growth quickly rises to the unknown size of the generalizedsupport while avoiding overshooting, as backed up by recent theory in Ndiaye and Takeuchi (2021).
Algorithm skglm ) input : X, β ∈ R p , n out ∈ N , n in ∈ N , ws _ size ∈ N , (cid:15) >1 for t = 1 , . . .
, n out do score = (cid:0) dist ( −∇ j f ( β ) , ∂g j ( β j )) (cid:1)
j ∈
[ p ] ws _ size = max(ws _ size , 2 | gsupp( β ) | ) //
ws _ size features with largest scores ws = arg
topK(score , K = ws _ size) if max j ∈ [ p ] dist ( −∇ j f ( β ) , ∂g j ( β j )) ≤ (cid:15) then stop else // accelerated CD on working set β ←
inner_solver ( X, β, ws , n in , (cid:15) ) 8 return β Algorithm 2 inner_ input : X, β (0)
∈ R p , ws ⊂ [ p ] , n in , (cid:15) , M =1 for k = 1 , . . .
, n in do β ( k ) ← CD ( X, β ( k − 1) , Xβ, ws) //
Algo.
(3) if k mod M = 0 then // Algo.
(4) , O ( M 2 | ws | + M 3 ) β extrws ← Anderson ( β ( k − M ) ws , . .
β // test objective O ( n ws
Proposition 5.
Let W t be the t -th working set.
Suppose that Algorithm 2 converges toward a criticalpoint, and for all t ≥ 0 , W t ⊂ W t +1 , then the iterates of Algorithm 1 converges towards a critical 97 point of Problem (1) .
Proof of Proposition 5 can be found in Appendix B.1.
The second key ingredient to our algorithm is touse state-of-the-art Anderson accelerated coordinate descent for non-convex problems.
In Section 2.3we show that coordinate descent yields ﬁnite time support identiﬁcation for a large class of nonconvex problems (Proposition 10), which leads to acceleration (Proposition 13).
As experimentsdemonstrate in Section 3, this rate allows our algorithm to surpass state-of-the-art solvers.
2.3 Anderson accelerated coordinate descent analysis for α -semi-convex penalties
We now turn to our main technical contributions: we show that Algorithm 2 achieves ﬁnite timesupport identiﬁcation (Proposition 10) of the generalized support (Deﬁnition 4) for speciﬁc class ofnon-smooth non-convex penalties (Assumption 6), which includes the MCP (Proposition 7).
Basedon Proposition 10, we are able to derive convergence rates for Anderson acceleration Proposition 13.
We study our inner solver (Algorithm 2); for convenience we still refer to β and X for their counterparts restricted to the working set.
The following assumptions are required.
Assumption 6 ( α -semi-convex) .
For all j ∈ [ p ] g j /L
j is α -semi-convex, i.e., g j /L j + α (cid:107)·(cid:107) 2 / 2 isconvex, with α < 1 .
Note that in statistics, the admissible value range of hyperparameters for MCP and SCAD are dataﬁtdependent, (see Breheny and Huang 2011, Sec. 2.1, normalized columns and γ > 1 = 1 / (cid:107) X : j (cid:107)
=1 /L j or Soubies et al.
2015, Eq. 4.2) and yields α -semi-convexity for MCP and SCAD 1 .
Proposition 7 ( α -semi-convexity of MCP) .
Let MCP λ,γ ( x ) (cid:44) (cid:40) λ
| x | − x
[Equation]
γ
[Equation]
if | x | ≤ γλ 12 γλ 2 , if | x | > γλ .
If γ > 1 /L j , then MCP λ,γ /L
j is α -semi-convex
( i.e., Assumption 6 holds).
Note that Assumption 6 does not hold for the (cid:96) q -penalties ( 0 < q < 1 ), for which we propose analternative in Appendix C.
Assumption 8 (Existence) .
Problem (1) admits at least one critical point.
In Proposition 10, convergence of Algorithm 2 toward a critical point ˆ β is assumed, and the followingassumption is made on this critical point.
Assumption 9 (Non degeneracy) .
The considered critical point ˆ β ∈ R p is non-degenerated: for allj / ∈ gsupp(ˆ β ) , −∇ f j (ˆ β ) ∈ interior( ∂g j (ˆ β j )) .
Assumption 9 is a generalization of qualiﬁcation constraints (Hare and Lewis, 2007, Sec. 1), andis usual in the machine learning literature (Zhao and Yu, 2006; Bach, 2008).
Equipped with theprevious assumptions we show that coordinate descent achieves model identiﬁcation for this class ofnon-convex problems.
Proposition 10 (Model identiﬁcation of CD) .
Suppose1.
Assumptions 1, 2, 6 and 8 hold.
The sequence ( β ( k ) )
k ≥ 0 generated by coordinate descent (Algorithm 2 without extrapolation) converges toward a critical point ˆ β
Assumption 9 holds for ˆ β .
Then, Algorithm 2 (without extrapolation) identiﬁes the model in ﬁnitely many iterations: there exists
K > 0 such that for all k ≥ K , β ( k ) S c
= ˆ β S c .
In other words, for k large enough, β ( k ) shares the generalized support of ˆ β .
The identiﬁcationproperty was proved for a proximal gradient descent algorithm in the non-convex case (Liang et al.
under the assumption that the non-smooth function g is partly smooth (Lewis, 2002).
In thecase of separable functions, Proposition 10 does not rely on the partly smooth assumption to ensureidentiﬁcation property.
Authors are not aware of previous identiﬁcation results for coordinate descentin the non-convex case.
In addition, if f and g are locally regular on the generalized support at the considered critical point,our algorithm enjoys local accelerated convergence rate when combined with Anderson extrapolation(Proposition 13).
Assumption 11 (Locally C 3 ) .
For all j ∈ S (cid:44) gsupp(ˆ β ) , g j is locally C 3 around ˆ β j , and f is locally
C around ˆ β .
[Table]
and Pedregosa
[Table]
Assumption 12.
(Local strong convexity)
The Hessian of f at the considered critical point ˆ β ∈ R p ,restricted to its generalized support S , is positive deﬁnite, i.e., ∇ 2 S , S f
(ˆ β ) + ∇ 2 S , S g (ˆ β ) (cid:31) 0 .
For instance, for a quadratic f , if the entries of the design matrix X are drawn from a continuousdistribution, then Assumption 12 holds with probability one (Tibshirani, 2013, Lemma 4).
Proposition 13.
Consider a critical point ˆ β and suppose1.
Assumptions 1, 2 and 8 hold.
The functions f and g j , j ∈ [ p ] are piecewise quadratic (which is the case for the MCP
regression).
The sequence ( β ( k ) )
k ≥ 0 generated by anderson accelerated coordinate descent with updates
from 1 to p and p to 1 (Algorithm 2 with extrapolation) converges to a critical point ˆ β .
Assumptions 9, 11 and 12 hold for ˆ β .
Then there exists K ∈ N , and a C 1 function ψ : R |S| → R |S| such that, for all k ∈ N , k ≥ K :
[Equation]
all
[Equation]
Let T (cid:44) J ψ
(ˆ β ) , H (cid:44) ∇ 2 S , S f
(ˆ β ) + ∇ 2 S , S g (ˆ β ) , ζ (cid:44) (1 − (cid:112) 1 − ρ ( T )) / (1 + (cid:112) 1 − ρ ( T )) and
B (cid:44) ( T − Id) (cid:62) ( T − Id) .
Then ρ ( T ) < 1 and the iterates of Anderson extrapolation enjoy localaccelerated convergence rate:
[Equation]
The proof can be found in Appendix B.5.
Related work.
Most Anderson acceleration convergence results are shown for quadratic objectives forspeciﬁc algorithms: gradient descent (Golub and Varga, 1961; Anderson, 1965), ADMM (Poon and
Liang, 2019), coordinate descent (Bertrand et al., 2020).
Outside of the quadratic case, convergenceresults are usually signiﬁcantly weaker (Scieur et al., 2016; Sidi, 2017; Brezinski et al., 2018; Maiand Johansson, 2019;
Ouyang et al., 2020).
Regarding the smooth non-convex case, Wei et al.(2021) proposed a stochastic Anderson acceleration and proved convergence towards a critical point.
Proposition 13 generalizes Scieur et al.
(2020, Prop 2.1) and Bertrand and Massias (2021, Prop. 4) tothe proximal convex and α -semi-convex cases.
To our knowledge this is one of the ﬁrst quantitativeresults for Anderson acceleration in a non-convex setting.
2.4 Comparison with existing work
In this section we compare our contribution to existing algorithms and implementations, which aresummarized in Table 1.
Huge scale refers to the fact that the algorithm can run on problems withmillions of variables.
Non-convex tells if the algorithm handles non-convex penalties.
Modularindicates that it is easy to add a new model, through a different dataﬁtting term or penalty.
The packages glmnet (Friedman et al., 2010), scikit-learn (Pedregosa et al., 2011) andlightning (Blondel and Pedregosa, 2016) implement coordinate descent (cyclic or random). They
[Figure] [Table] [Figure] rely on compiled code such as Fortran or Cython, making it very difﬁcult to implement new models 2or faster algorithms like working set 3 .
They do not handle non-convex penalties.
More recent algorithms such as blitz (Johnson and Guestrin, 2015), celer (Massias et al., 2018),picasso (Ge et al., 2019) or fireworks (Rakotomamonjy et al., 2022)
use working set strategies.celer and blitz are state-of-the-art algorithms for the Lasso, but their score to prioritize featuresrelies on duality.
fireworks extends blitz to some non-convex penalties (writing as difference ofconvex functions), with score ﬁreworks j = dist( −∇ j f ( β ) , ∂g j (0)) .
Yet this rule does not consider thesubdifferential of g at the current point, but at 0, which is a coarse information.
Finally, fireworksdoes not provide accelerated convergence rates and does not come with a public implementation.picasso (Ge et al., 2019) lacks modularity (penalties are hardcoded), and the solver is not suited forhuge scale (it does not support large sparse matrices).
Deng and Lan (2019) proposed an algorithmbased on inertially accelerated coordinate descent, which fails to provide practical speedups accordingto Bertrand and Massias (2021).
Contrary to these algorithms, ours is generic and relies only on the knowledge of ∇ f and prox g .
Forany new penalty, this information can be written in a few lines of Python code, compiled with numba(Lam et al., 2015) for speed efﬁciency.
We therefore improve of state-of-the-art algorithms in theconvex case, and generalize to virtually any dataﬁt and penalty, even nonconvex.
Experiments
Our package relying on numpy and numba (Lam et al., 2015; Harris et al., 2020) is attached in thesupplementary material and will be open-sourced upon publication under BSD 3-Clause License.
Weuse datasets from libsvm 4 (Fan et al.
2008, see table 2).
We compare multiple algorithms to solve popular Machine Learning and inverse problems: Lasso,
Elastic net, multitask sparse regression, MCP regression.
The compared algorithms are the following:
• scikit-learn (Pedregosa et al., 2011), which implements coordinate descent in Cython, • celer (Massias et al., 2020), which combines working sets, screening rules, coordinatedescent, and Anderson acceleration in the dual, in Cython,• blitz (Johnson and Guestrin, 2015), which combines working sets with prox-Newtoniterations (Lee et al., 2012) in C++,• lightning (Blondel and Pedregosa, 2016), which implements coordinate descent andshrinking heuristics in Cython,• coordinate descent (CD, Tseng and S.Yun 2009), • skglm (Algorithm 1, ours), using M = 5 iterates for the Anderson extrapolation
Other solvers.
Experiment per experiment, there exist niche solvers (such as aggressive Gap
Safe
Rules, Ndiaye et al. 2020).
Since our goal is a general purpose algorithm able to deal with manymodels, we do not include them in the comparison.
In addition, we focus on solving a singleinstance of Problem (1), rather than a regularization path ( i.e., a sequence of problems for multipleregularization strengths).
As glmnet is designed to compute regularization paths, we could notinclude it in the comparison.
The reader can refer to Johnson and Guestrin (2015, Fig. 4) or Figure 7in Appendix E for comparisons on single optimization problems with glmnet .
glmnet and additionalalgorithms are discussed in Appendix E.
How to do a fair comparison between solvers?
To plot the convergence curves, we use thebenchopt tool 5 .
This tool to automate and reproduce optimization benchmarks treats solvers asblack boxes: it launches them several times with increasing maximum number of iterations, andstores the resulting objective values and times to reach it.
As each point on a solver curve is obtainedin a different run, the curves are not monotonic, and there may be several points corresponding tothe same time.
This merely reﬂects the variability in [-blitz celer skglm (ours) sklearn [Figure] [Figure] λ max /0 .2 .
-]solvers running time across runs; we refer to
[-dd a  
-]
Appendix E.6 for the inevitability of this phenomenon with black box solvers.
3.1 Convex problems
Lasso.
In Figure 2 we compare solvers for the Lasso: ( f = 12 n (cid:107) y − X
·(cid:107) 2 , g j = λ |·| ).
Weparametrize λ
as a fraction of λ max
= (cid:107) X (cid:62) y (cid:107) ∞ /n , smallest regularization strength for which ˆ β = 0 .
For large scale datasets ( rcv1 , news20 ), skglm yields performances better or similar to the state-ofthe-art algorithms blitz and celer .
For huge scale datasets ( kdda and url ), skglm yields signiﬁcantspeedups over them.
The improvement over the popular scikit-learn can be of two orders ofmagnitude.
Thus, while dealing with many more models, our algorithm still yields state-of-the-artspeed for basic ones .
Elastic net.
Our approach easily generalizes to other problems, such as the elastic net ( f =12 n (cid:107) y − X ·(cid:107) 2 , g j = λ ( ρ |·| + 1 − ρ 2 ( · ) 2 ) ).
Figure 3 shows the duality gap as a function of time 235 for skglm (ours), sklearn , and our numba implementation of coordinate descent.
The proposedalgorithm is orders of magnitude faster than scikit-learn and vanilla coordinate descent, inparticular for large datasets and low regularization parameter values ( ﬁnance , λ max / 1000 ).
Note that blitz does not implement a solver for the elastic net.
Many Lasso solvers would easilyhandle the elastic net, but relying on Cython/C++ code makes the implementation time-consuming.
By contrast, it takes 40 lines of code to deﬁne an (cid:96) 1 + (cid:96) 2 -squared penalty with our implementation.
An additional experiment on the dual of SVM with hinge loss is in Appendix E.3. [Figure] [Figure]
3.2 Non-convex problems
In this subsection we propose a comparison on two non convex problems.
[-sklearn Time (s) 
 [Figure]-]MCP regression.
The MCP for x ∈ R is deﬁned as: MCP λ,γ ( x ) =
[Equation]
MCP regression is Problem (1) with f = 12 n (cid:107) y − X ·(cid:107) 2 , g j = MCP λ,γ for γ > 1 .
As usual for thisproblem, we scale the columns of X to have norm √ n .
On Figure 5, we compare our algorithm topicasso on a dense dataset ( n = 1000 , p = 5000 ); as this package does not support large sparsedesign matrices, for the rcv1 dataset we use an iterative reweighted L1 algorithm (Candes et al., 2008).
Since the derivative of the MCP vanishes for values bigger than λγ , this approach requires solvingweighted Lassos with some 0 weights.
Up to our knowledge, our algorithm is the only efﬁcient onewith such a property.
Our algorithm handles problems of large size, converges to a critical point, and,due to its progressive inclusion of features, is able to reach a sparser critical point than it competitors.
Application to neuroscience To demonstrate the usefulness of our algorithm for practitioners,we apply it to the magneto-/electroencephalographic (M/EEG) inverse problem.
It consists inreconstructing the spatial cortical current density at the origin of M/EEG measurements made atthe surface of the scalp.
Non-convex penalties (Strohmeier et al., 2015) exhibit several advantagesover convex ones (Gramfort et al., 2013): they yield sparser physiologically-plausible solutions andmitigate the (cid:96) 1 amplitude bias.
Here the setting is multitask: Y ∈ R n × T and thus we use blockpenalties (details in Appendix D).
We use real data from the mne software (Gramfort et al., 2014);the experiment is a right auditory stimulation, with two expected neural sources to recover in eachauditory cortex.
In Figure 4, while the (cid:96) 2 , 1 penalty fails at localizing one source in each hemisphere,the non-convex penalties recover the correct locations.
This emphasizes on the critical need for fastsolvers for non-convex sparse penalties as well as our algorithm’s ability to handle the latter.
3.3 Ablation study
To evaluate the inﬂuence of the two components of Algorithm 1, an ablation study (Figure 6 and Figure 9 in Appendix E) is performed.
Four algorithms are compared: with/without working setsand with/without Anderson acceleration.
Figure 6 represents the duality gap of the Lasso as a
[Figure]
[Figure]
function of time for multiple datasets and values of the regularization parameters λ (parametrized asa fraction of λ max ).
First, Figure 6 shows that working sets always bring signiﬁcant speedups.
Then,when combined with working set, Anderson acceleration bring signiﬁcant speed-ups, especially forhard problems with low regularization parameters.
An interesting observation is that on large scaledatasets ( news20 and ﬁnance in Appendix E.5) and for low regularization parameters ( λ max / 100 andλ max / 1000 )
Anderson acceleration without working set does not bring acceleration.
This highlightsthe importance of combining Anderson acceleration with working sets.
Conclusion and broader impact
In this paper, we have proposed an accelerated versatile algorithm for a speciﬁc class of non-smoothnon-convex problems.
Based on working sets, coordinate descent and Anderson acceleration, wehave improved state-of-the-art on convex problems, and handled previously out-of-reach problems.
Thorough experiments demonstrated the speed and interest of our approach.
A limitation of thiswork is the considered function class ( α -semi-convex), which can be seen as restrictive.
One possibleextension would be weakly convex functions (Davis and Drusvyatskiy, 2019, Sec. 1).
We deeplybelieve that the high quality code provided will beneﬁt to practitioners, and ease the use of non-convexpenalties for real world problems, from neuroimaging to genomics.
We proposed an optimizationalgorithm and do not see potential negative societal impacts.