Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Anonymous Author(s)
Abstract
Numerical validation is at the core of machine learning research as it allows toassess the actual impact of new methods, and to confirm the agreement betweentheory and practice.
Yet, the rapid development of the field poses several challenges:researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work.
As a result, validation is often very partial, which can lead to wrong conclusions thatslow down the progress of research.
We propose Benchopt , a collaborative framework to automate, reproduce and publish optimization benchmarks in machinelearning across programming languages and hardware architectures.
Benchoptsimplifies benchmarking for the community by providing an off-the-shelf tool forrunning, sharing and extending experiments.
To demonstrate its broad usability, weshowcase benchmarks on three standard learning tasks: ℓ 2 -regularized logistic regression, Lasso, and ResNet18 training for image classification.
These benchmarkshighlight key practical findings that give a more nuanced view of the state-of-the-artfor these problems, showing that for practical evaluation, the devil is in the details.
We hope that Benchopt will foster collaborative work in the community henceimproving the reproducibility of research findings.
Introduction
Numerical experiments have become an essential part of statistics and machine learning (ML).
Itis now commonly accepted that every new method needs to be validated through comparisons withexisting approaches on standard problems.
Such validation provides insight into the method’s benefitsand limitations and thus adds depth to the results.
While research aims at advancing knowledgeand not just improving the state of the art, experiments ensure that results are reliable and supporttheoretical claims (Sculley et al., 2018).
Practical validation also helps the ever-increasing numberof ML users in applied sciences to choose the right method for their task.
Performing rigorous andextensive experiments is, however, time-consuming (Raff, 2019), particularly because comparisonsagainst existing methods in new settings often requires reimplementing baseline methods from theliterature.
In addition, ingredients necessary for a proper reimplementation may be missing, such asimportant algorithmic details, hyperparameter choices, and preprocessing steps (Pineau et al., 2019).
In the past years, the ML community has actively sought to overcome this “reproducibility crisis”(Hutson, 2018) through collaborative initiatives such as open datasets ( OpenML , Vanschoren et al.2013), standardized code sharing (Forde et al., 2018), benchmarks ( MLPerf , Mattson et al.
2020),the NeurIPS and ICLR reproducibility challenges (Pineau et al., 2019; Pineau et al., 2021) and newjournals ( e.g., Rougier and Hinsen 2018).
As useful as these endeavors may be, they do not, however, since, in this area, there are no clear community guidelines on how to perform, share, and publish benchmarks
[Figure]
Optimization algorithms pervade almost every area of ML, from empirical risk minimization, variational inference to reinforcement learning (Sra et al., 2012).
It is thus crucial to know which methodsto use depending on the task and setting (Bartz-Beielstein et al., 2020).
While some papers inoptimization for ML provide extensive validations (Lueckmann et al., 2021), many others fall shortin this regard due to lack of time and resources, and in turn feature results that are hard to reproduceby other researchers.
In addition, both performance and hardware evolve over time, which eventuallymakes static benchmarks obsolete.
An illustration of this is the recent work by Schmidt et al.
(2021),which extensively evaluates the performances of 15 optimizers across 8 deep-learning tasks.
Whiletheir benchmark gives an overall assessment of the considered solvers, this assessment is boundto become out-of-date if it is not updated with new solvers and new architectures.
Moreover, thebenchmark does not reproduce state-of-the-art results on the different datasets, potentially indicatingthat the considered architectures and optimizers could be improved.
We firmly believe that this critical task of maintaining an up-to-date benchmark in a field cannot be solved without a collective effort .
We want to empower the community to take up this challenge and build a living, reproducible and standardized state of the art that can serve as a foundation for future research.
Benchopt provides the tools to structure the optimization for machine learning (Opt-ML) communityaround standardized benchmarks, and to aggregate individual efforts for reproducibility and resultssharing.
Benchopt can handle algorithms written in Python , R , Julia or C/C++ via binaries.
It offersbuilt-in functionalities to ease the execution of benchmarks: parallel runs, caching, and automaticalresults archiving.
Benchmarks are meant to evolve over time, which is why Benchopt offers amodular structure through which a benchmark can be easily extended with new objective functions,datasets, and solvers by the addition of a single file of code.
The paper is organized as follows.
We first detail the design and usage of Benchopt , before presentingresults on three canonical problems:
• ℓ 2 -regularized logistic regression: a convex and smooth problem which is central to the evaluationof many algorithms in the Opt-ML community, and remains of high relevance for practitioners;
• the Lasso: the prototypical example of non-smooth convex problem in ML;• training of ResNet18 architecture for image classification: a large scale non-convex deep learningproblem central in the field of computer vision.
The reported benchmarks, involving dozens of implementations and datasets, shed light on thecurrent state-of-the-art solvers for each problem, across various settings, highlighting that the bestalgorithm largely depends on the dataset properties ( e.g., size, sparsity), the hyperparameters, as wellas hardware.
A variety of other benchmarks ( e.g., MCP, TV1D, etc.) are also presented in Appendix,with the goal to facilitate contributions from the community.
By the open source and collaborative design of Benchopt (BSD 3-clause license), we aim to openthe way towards community-endorsed and peer-reviewed benchmarks that will improve the trackingof progress in optimization for ML.
The Benchopt library
The Benchopt library aims to provide a test bed to benchmark algorithms for optimization in ML,where the problems depend on some input dataset D .
The considered problems are of the form
[Equation]
where f is the objective function, Λ are its hyperparameters, and Θ is the feasible set forθ .
The scope of the library is to evaluate optimization methods in their wide sense by considering the sequence { θ t } t produced to approximate θ ∗ .
To provide a flexible and extendable test bed, benchmarks are defined as the association of three types of object classes:benchmark/ objective.py datasets/
dataset1.py dataset2.py solvers/ solver1.py solver2.py
Objective: It defines the function f to be minimized as well as the hyperparameters Λ or the set Θ , and the metrics to track along the iterations( e.g., objective value, gradient norm for smooth problems, or validationloss).
Multiple metrics can be registered for each θ t .
Datasets: The dataset objects provide the data D to be passed to theobjective class.
They control how data is loaded and preprocessed.
Datasetsare separated from the objective, making it easy to add new ones, providedthey are coherent with the objective.
Solvers: The solver objects define how to run the algorithm.
They areprovided with the objective and dataset objects and output a sequence{ θ t } t .
This sequence can be obtained using a single run of the method, orwith multiple runs in case the method only returns its final iterate.
Each of these objects can have parameters that change their behavior, e.g., the regularization parameters for the objective, the choice of preprocessing for the datasets, or the step size for the solvers.
By exposing these parameters in the different objects, Benchopt can evaluate their influence on thebenchmark results.
The Benchopt library defines an application programming interface (API) foreach of these concepts and provides a command line interface (CLI) to make them work together.
A benchmark is defined as a folder that contains an Objective as well as subfolders containing the Solvers and Datasets .
Code examples of the three classes can be found in Appendix A. Benchopt iswritten in Python , but solvers run with implementations in different languages ( e.g., R and Julia , asin Section 4) and frameworks ( e.g., PyTorch and TensorFlow , as in Section 5).
For each dataset and solver, and for each set of parameters, Benchopt retrieves a sequence { θ t } t andevaluates the metrics defined in the Objective for each θ t .
To ensure fair evaluation, the computationof these metrics is done off-line.
The metrics are gathered in a CSV file that can be used to display thebenchmark results, either locally or as HTML files published on a website that reference the benchmarksrun with Benchopt .
This workflow is described in Figure 1.
This modular and standardized organization for benchmarks empowers the optimization communityby making numerical experiments easily reproducible, shareable, flexible and extendable.
Thebenchmark can be shared as a git repository or a folder containing the different definitions forthe Objective , Datasets and Solvers and it can be run with the Benchopt CLI, hence becominga convenient reference for future comparisons.
This also ensures fair evaluation of baselines infollow-up experiments, as implementations validated by the community are available.
Finally, by supporting multiple metrics the Objective class limits the risk of mismatch in the evaluation.
Moreover, benchmarks can be run in parallel with checkpointing of the results, enabling large scaleevaluations on many CPU or GPU nodes.
Benchmark results are automatically exported as interactivevisualizations, helping with the exploration of the many different settings.
Hardware All presented benchmarks are run on Intel Xeon Gold 6248 CPUs and NVIDIA {+V
 GPUs.+}
First example: ℓ 2 -regularized logistic regression
Logistic regression is a very popular method for binary classification.
From a design matrix
X ∈ R n × p with rows
X i and a vector of labels y ∈ {− 1 , 1 } n with corresponding element y i , ℓ 2 -regularized logistic regression provides a generalized linear model indexed by θ
∗ ∈
R p to discriminate the classes by solving
[Equation]
where λ > 0 is the regularization hyperparameter.
Thanks to the regularization part, Problem (2) isstrongly convex with a Lipschitz gradient, and thus its solution can be estimated efficiently usingmany iterative optimization schemes.
The most classical methods to solve this problem take inspiration from Newton’s method (Wrightand Nocedal, 1999).
On the one hand, quasi-Newton methods aim at approximating the Hessianof the cost function with cheap to compute operators.
Among these methods, L-BFGS (Liu and
Nocedal, 1989) stands out for its small memory footprint, its robustness and fast convergence in avariety of settings.
On the other hand, truncated Newton methods (Dembo et al., 1982) try to directlyapproximate Newton’s direction by using e.g., the conjugate gradient method (Fletcher and Reeves,1964) and Hessian-vector products to solve the associated linear system.
Yet, these methods sufferwhen n is large: each iteration requires a pass on the whole dataset.
In this context, methods based on stochastic estimates of the gradient have become standard (Bottou,2010), with Stochastic Gradient Descent (SGD) as a main instance.
The core idea is to use cheap andnoisy estimates of the gradient (Robbins and Monro, 1951; Kiefer and Wolfowitz, 1952).
While SGDgenerally converges either slowly due to decreasing step sizes, or to a neighborhood of the solutionfor constant step sizes, variance-reduced adaptations such as SAG (Schmidt et al., 2017), SAGA(Defazio et al., 2014) and SVRG (Johnson and Zhang, 2013) make it possible to solve the problemmore efficiently and are often considered to be state-of-the-art for large scale problems.
Finally, methods based on coordinate descent (Shi et al., 2017) have also been proposed to solve
Problem (2).
While these methods are usually less popular, they can be efficient in the context ofsparse datasets, where only few samples have non-zero values for a given feature, or when acceleratedon distributed systems or GPU (Dünner et al., 2018).
To reflect the diversity of solvers available, we showcase a Benchopt benchmark with 3 datasets,10 optimization strategies implemented in 5 packages, leveraging GPU hardware when possible.
We also consider different scenarios for the objective function: (i) scaling (or not) the features, arecommended data preprocessing step, crucial in practice to have comparable regularization strengthon all variables; (ii) fitting (or not) an unregularized intercept term, important in practice and makingoptimization harder when omitted from the regularization term (Koh et al., 2007); (iii) working (ornot) with sparse features, which prevent explicit centering during preprocessing to keep memoryusage limited.
Details on packages, datasets and additional scenarios are available in Appendix B.
Results Figure 3 presents the results of the benchmarks, in terms of suboptimality of the iterates,f ( θ t ) − f ( θ ∗ ) , for three datasets and three scenarios.
Here, because the problem is convex, θ ∗ isapproximated by the best iterate across all runs (see Section A.4).
Overall, the benchmark shows thebenefit of using GPU solvers ( cuML and snapML ), even for small scale tasks such as ijcnn1 .
It canalso be noted that these two accelerated solvers converge to a higher suboptimality level compared toother solvers, due to operating with 32-bit float precision.
Another observation is that data scalingcan drastically change the picture.
In the case of madelon , most solvers have a hard time convergingfor the scaled data.
For the solvers that converge, we note that the convergence time is one order ofmagnitude smaller with the scaled dataset compared to the raw one.
This stems from the fact that inthis case, the scaling improves the conditioning of the dataset.
For news20.binary , the stochasticsolvers such as SAG and SAGA have degraded performances on scaled data.
Here, the scaling makesthe problem harder.
On CPU, quasi-Newton solvers are often the most efficient ones, and provide a reasonable choice inmost situations.
For large scale news20.binary , stochastic solvers such as SAG, SAGA or SVRG –thatare often considered as state of the art for such problem– have worst performances for the presenteddatasets.
While this dataset is often used as a test bed for benchmarking new stochastic solvers, wefail to see an improvement over non-stochastic ones for this experimental setup.
In contrast, the lastrow in Figure 3 displays an experiment with the larger scale criteo dataset, which demonstrates a [Figure] deviation third column is for an objective which includes an unregularized intercept. regime where variance-reduced stochastic gradient methods outperform quasi-Newton methods.
Forfuture benchmarking of stochastic solvers, we therefore recommend using such a large dataset.
Finally, the third column in Figure 3 illustrates a classical problem when benchmarking differentsolvers: their specific (and incompatible) definition and resolution of the corresponding optimizationproblem.
Here, the objective is modified to account for an intercept (bias) in the linear model.
Inmost situations, this intercept is not regularized when it is fitted.
However, snapML and liblinearsolvers do regularize it, leading to incomparable losses.
Second example: the Lasso
The Lasso, (Tibshirani, 1996; Chen et al., 1998), is an archetype of non-smooth ML problems, whose impact on ML, statistics and signal processing in the last three decades has been considerable (Bühlmann and van de Geer, 2011; Hastie et al., 2015).
It consists of solving
[Equation]
where X ∈ R n × p is a design matrix containing p features as columns, y ∈ R n is the target vector,and λ > 0 is a regularization hyperparameter.
The Lasso estimator was popularized for variableselection: when λ is high enough, many entries in θ ∗ are exactly equal to 0.
This leads to moreinterpretable models and reduces overfitting compared to the least-squares estimator.
Solvers for Problem (3) have evolved since its introduction by Tibshirani (1996).
After genericquadratic program solvers, new dedicated solvers were proposed based on iterative reweighted leastsquares (IRLS) (Grandvalet, 1998), followed by LARS (Efron et al., 2004), a homotopy methodcomputing the full Lasso path 3 .
The LARS solver helped popularize the Lasso, yet the algorithmsuffers from stability issues and can be very slow for worst case situations (Mairal and Yu, 2012).
General purpose solvers became popular for Lasso-type problems with the introduction of the iterativesoft thresholding algorithm (ISTA, Daubechies et al.
, an instance of forward-backward splitting(Combettes and Wajs, 2005).
These algorithms became standard in signal and image processing,especially when accelerated (FISTA, Beck and Teboulle 2009).
In parallel, proximal coordinate descent has proven particularly relevant for the Lasso in statistics.
Early theoretical results were proved by Tseng (1993) and Sardy et al.
(2000), before it became thestandard solver of the widely distributed packages glmnet in R and scikit-learn in Python.
Forfurther improvements, some solvers exploit the sparsity of θ ∗ , trying to identify its support to reducethe problem size.
Best performing variants of this scheme are screening rules ( e.g., El Ghaoui et al.
,2012; Bonnefoy et al., 2015; Ndiaye et al., 2017) and working/active sets ( e.g., Johnson and Guestrin2015; Massias et al. 2018), including strong rules (Tibshirani et al., 2012).
While reviews of Lasso solvers have already been performed (Bach et al., 2012, Sec. 8.1), they arelimited to certain implementation and design choices, but also naturally lack comparisons with morerecent solvers and modern hardware, hence drawing biased conclusions.
Results obtained on 4 datasets, with 9 standard packages and some custom reimplementations, possibly leveraging GPU hardware, and 17 different solvers written in Python / numba / Cython , R , Julia or
C++ (Table C.1) are presented in Figure 4.
All solvers use efficient numerical implementations, possibly leveraging calls to BLAS , precompiled code in Cython or just-in-time compilation with numba .
The different parameters influencing the setup are
• the regularization strength λ , controlling the sparsity of the solution, parameterized as a fractionof λ max =
(cid:13)(cid:13)
X ⊤ y (cid:13)(cid:13) ∞ (the minimal hyperparameter such that θ ∗ = 0 ),
• the dataset dimensions: MEG has small n and medium p ; rcv1.binary has medium n and p ;news20.binary has medium n and very large p while MillionSong has very large n and small p(Table C.2).
Results Figure 4 presents the result of the benchmark on the Lasso, in terms of objective suboptimality f ( θ t ) − f ( θ ∗ ) as a function of time.
Similarly to Section 3, the GPU solvers obtain good performances in most settings, but their advantageis less clear.
A consistent finding across all settings is that coordinate descent-based methodsoutperform full gradient ones (ISTA and FISTA, even restarted), and are improved by the use ofworking set strategies ( blitz , celer , skglm , glmnet ).
This observation is even more pronouncedwhen the regularization parameter is large, as the solution is sparser.
When observing the influence of the dataset dimensions, we observe 3 regimes.
When n is small( MEG ), the support of the solution is small and coordinate descent, LARS and noncvx-pro performthe best.
When n is much larger than p ( MillionSong ), noncvx-pro clearly outperforms other solvers,and working set methods prove useless.
Finally, when n and p are large ( rcv1.binary , news20.binary ),
CD and working sets vastly outperforms the rest while noncvx-pro fails, as it requires solving alinear system of size min( n, p ) .
We note that this setting was not tested in the original experiment of
Poon and Peyré (2021), which highlights the need for extensive, standard experimental setups.
When the support of the solution is small (either small λ , either small n since the Lasso solution hasat most n nonzero coefficients), LARS is a competitive algorithm.
We expect this to degrade whenn increases, but as the LARS solver in scikit-learn does not support sparse design matrices wecould not include it for news20.binary and rcv1.binary . [Figure]
This benchmark is the first to evaluate solvers across languages, showing the competitive behaviorof lasso.jl and glmnet compared to Python solvers.
We observe that both solvers have a largeinitialization time, and then converge very fast.
To ensure that the benchmark is fair, even though the Benchopt library is implemented in Python , we made sure to track conversion overhead, as well asjust-in-time compilation cost, and ignore them.
We also checked the timing’s consistency with nativecalls to the libraries.
Since the Lasso is massively used for it feature selection properties, the speed at which the solversidentify the support of the solution is also an important performance measure.
Monitoring this with
Benchopt is straightforward, and a figure reporting this benchmark is in Appendix C.
Third example: How standard is a benchmark on ResNet18?
As early successes of deep learning have been focused on computer vision tasks (Krizhevsky et al.
,2012), image classification has become a de facto standard to validate novel methods in the field.
Among the different network architectures, ResNets (He et al., 2016) are extensively used in thecommunity as they provide strong and versatile baselines (Xie et al., 2017; Tan and Le, 2019; Dosovitskiy et al., 2021; Brock et al., 2021; Liu et al., 2022). While many papers present results
[Figure]
Time (s)
[Figure]
T e s t
e rr o r
[Figure]
CIFAR [Figure] [Figure] Time (s)
[Figure]
SVHN
[Figure]
Time (s)
MNIST Curves are exponentially smoothed.
with such model on classical datasets, with sometimes extensive ablation studies (He et al., 2019;
Wightman et al., 2021; Bello et al., 2021; Schmidt et al., 2021), the lack of standardized codebaseand missing implementation details makes it hard to replicate their results.
We provide a cross-dataset – SVHN , Netzer et al.
; MNIST , LeCun et al.
(2010) and CIFAR10 , Krizhevsky (2009)– and cross-framework – TensorFlow / Keras , Martín Abadi et al.
and Chollet et al.
; PyTorch , Paszke et al.
(2019)– evaluation of the training strategies for imageclassification with ResNet18 (see Appendix D for details on architecture and datasets).
We train thenetwork by minimizing the cross entropy loss relatively to the weights θ of the model.
Contrary tologistic regression and the Lasso, this problem is non-convex due to the non-linearity of the model f θ .
Another notable difference is that we report the evolution of the test error rather than the training loss.
The Solvers are defined as the combination of an optimization algorithm, its hyperparameters, thelearning rate (LR) and weight decay schedules, and the data augmentation strategy.
We focus on 2standard methods: stochastic gradient descent (SGD) with momentum and Adam (Kingma and Ba,2015), as well as a more recently published one: Lookahead (Zhang et al., 2019).
The LR schedulesare chosen among fixed LR, step LR 4 , and cosine annealing (Loshchilov and Hutter, 2017).
Wealso consider decoupled weight decay for Adam (Loshchilov and Hutter, 2019), and coupled weightdecay ( i.e., ℓ 2 -regularization) for SGD.
Regarding data augmentation, we use random cropping for alldatasets and add horizontal flipping only for CIFAR-10 , as the digits datasets do not exhibit a mirrorsymmetry.
We detail the remaining hyperparameters in Table D.2.
Aligning cross-framework implementations Due to some design choices, components with thesame name in the different frameworks do not have the same behavior.
For instance, when it comesto applying weight decay, PyTorch ’s SGD uses coupled weight decay, while in TensorFlow / Kerasweight decay always refers to decoupled weight decay.
These two methods lead to significantlydifferent performance and it is not straightforward to apply coupled weight decay in a post-hocmanner in TensorFlow / Keras (see further details in Section D.2).
We conducted an extensive effortto align them using unit testing to make the conclusions of our benchmarks independent of the chosenframework.
We found additional significant differences (reported in Table D.3) in the initialization,the batch normalization, the convolutional layers and the weight decay scaling.
Results
The results of the benchmark are reported in Figure 5.
Each graph reports the test errorrelative to time, with an ablation study on the solvers parameters.
Note that we only report selectedsettings for clarity but that we run every possible combinations.
The results are available in thesupplementary materials as a user-friendly interactive HTML file.
[Figure]
T e s t /
V a l e rr o r ( % )
CIFAR
[Figure] [Figure]
Time (s)
SVHN
Time (s)
MNIST
Firstly, we see that reaching the state of the art for a vanilla ResNet18 is not straightforward.
On thepopular website Papers with code it has been so far underestimated.
It can achieve 4.45% and 2.65%test error rates on CIFAR-10 and SVHN respectively (compared to 4.73% and 2.95% – for a PreAct ResNet18 – before that).
What our ablation study shows, is that a variety of techniques is required toreach it.
The most significant one is an appropriate data augmentation strategy, which lowers the errorrate on CIFAR-10 from about 18% to about 8%.
The second most important one is weight decay,but it has to be used in combination with a proper LR schedule, as well as momentum.
While thesetechniques are not novel, they are regularly overlooked in baselines, resulting in underestimation of their performance level.
This reproducible benchmark not only allows a researcher to get a clear understanding of how toachieve the best performances for this model and datasets, but also provides a way to reproduce andextend these performances.
In particular, we also include in this benchmark the original implementation of Lookahead (Zhang et al., 2019).
We confirm that it slightly accelerates the convergence of the
Best SGD, even with a cosine LR schedule – a setting that had not been studied in the original paper.
Our benchmark also evaluates the relative computational performances of the different frameworks.
We observe that PyTorch-Lightning is significantly slower than the other frameworks we tested,in large part due to their callbacks API.
We also notice that our TensorFlow / Keras implementationis significantly slower ( ≈ 28%) than the PyTorch ones, despite following the best practices and ourprofiling efforts.
Note that we do not imply that TensorFlow is intrinsically slower than PyTorch ,but a community effort is needed to ensure that the benchmark performances are framework-agnostic.
A recurrent critic of such benchmarks is that only the best test error is reported.
In Figure 6, wemeasure the effect of using a train-validation-test split, by keeping a fraction of the training set asa validation set.
The splits we use are detailed in Table D.1.
Our finding is that the results of theablation study do not change significantly when using such procedure, even though their validity isreinforced by the use of multiple trainings.
Yet, a possible limitation of our findings is that some ofthe hyperparameters we used for our study, coming from the PyTorch-CIFAR GitHub repository,may have been tuned while looking at the test set.
Conclusion and future work
We have introduced Benchopt , a library that makes it easy to collaboratively develop fair and extensivebenchmarks of optimization algorithms, which can then be seamlessly published, reproduced, andextended.
In the future, we plan on supporting the creation of new benchmarks, that could becomethe standards the community builds on.
This work is part of a wider effort to improve reproducibilityof machine learning results.
It aims to contribute to raising the standard of numerical validation foroptimization, which is pervasive in the statistics and ML community as well as for the experimentalsciences that rely more and more on these tools for research.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] These are specified in Appendix, on a per-benchmark basis.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Error bars are not reported for clarity, but
Benchopt
allows this in particular in html versions of the plots.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
In the introduction.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[-We summarized in Table E.
-]
spent on participant compensation?
[-Results Figure G.1 shows that the solvers using the synthesis formulation and coordinate descentbased solvers for the Lasso ( ℓ 
-]
[N/A]
[-The result of the benchmark is presented in Figure H.
-]