Original: Opening the Black Box: Automated Software Analysis for Algorithm Selection
Revised: Opening the Black Box: Automated Software Analysis for Algorithm Selection

Original: Anonymous 1 Anonymous Institution

Original: Abstract Impressive performance improvements have been achieved in many areas of AI by meta5 algorithmic techniques, such as automated algorithm selection and configuration.
Revised: Abstract Impressive performance improvements have been achieved in many areas of AI by metaalgorithmic techniques, such as automated algorithm selection and configuration.

Original: However,6 existing techniques treat the target algorithms they are applied to as black boxes – nothing7 is known about their inner workings.
Revised: However, existing techniques treat the target algorithms they are applied to as black boxes – nothing is known about their inner workings.

Original: This allows meta-algorithmic techniques to be used8 broadly, but leaves untapped potential performance improvements enabled by information9 gained from a deeper analysis of the target algorithms.
Revised: This allows meta-algorithmic techniques to be used broadly, but leaves untapped potential performance improvements enabled by information gained from a deeper analysis of the target algorithms.

Original: In this paper, we open the black box10 without sacrificing universal applicability of meta-algorithmic techniques by automatically11 analyzing algorithms.
Revised: In this paper, we open the black box without sacrificing universal applicability of meta-algorithmic techniques by automatically analyzing algorithms.

Original: We show how to use this information to perform algorithm selec12 tion, and demonstrate improved performance compared to previous approaches that treat13 algorithms as black boxes.
Revised: We show how to use this information to perform algorithm selection, and demonstrate improved performance compared to previous approaches that treat algorithms as black boxes.

Original: 1 Introduction
Revised: 1 Introduction

Original: The past decade has seen the rise of meta-algorithmic techniques that enable the more efficient and16 intelligent use of existing algorithms, alleviating some of the challenges encountered in the design of17 new algorithms.
Revised: The past decade has seen the rise of meta-algorithmic techniques that enable the more efficient and intelligent use of existing algorithms, alleviating some of the challenges encountered in the design of new algorithms.

Original: This has given rise to the field of Automated Machine Learning [Hutter et al., 2019],18 which heavily relies on meta-algorithmic techniques.
Revised: This has given rise to the field of Automated Machine Learning [Hutter et al., 2019], which heavily relies on meta-algorithmic techniques.

Original: A prominent example of the success and19 impact of meta-algorithmic techniques is algorithm selection, where we choose from a given20 portfolio of algorithms the one likely to perform best on a given problem instance.
Revised: A prominent example of the success and impact of meta-algorithmic techniques is algorithm selection, where we choose from a given portfolio of algorithms the one likely to perform best on a given problem instance.

Original: 21 The per-instance algorithm selection problem , in which, given a problem instance, one of several22 available algorithms is to be chosen to solve that instance most effectively and efficiently, has been23 first considered in the 1970s [Rice, 1976].
Revised: The per-instance algorithm selection problem , in which, given a problem instance, one of several available algorithms is to be chosen to solve that instance most effectively and efficiently, has been first considered in the 1970s [Rice, 1976].

Original: The first prominent system that demonstrated the promise24 of algorithm selection was SATzilla [Xu et al., 2008], which chooses the most suitable algorithm25 from a portfolio of satisfiability (SAT) solvers.
Revised: The first prominent system that demonstrated the promise of algorithm selection was SATzilla [Xu et al., 2008], which chooses the most suitable algorithm from a portfolio of satisfiability (SAT) solvers.

Original: SATzilla showed excellent performance (including26 several gold medals) in SAT solver competitions.
Revised: SATzilla showed excellent performance (including several gold medals) in SAT solver competitions.

Original: Since then, many more examples have shown the27 promise of algorithm selection, where we choose from among a set of existing algorithms, and algo28 rithm configuration, where we tune the parameters of algorithms to achieve the best performance.
Revised: Since then, many more examples have shown the promise of algorithm selection, where we choose from among a set of existing algorithms, and algorithm configuration, where we tune the parameters of algorithms to achieve the best performance.

Original: 29 Among them are 3S [Kadioglu et al., 2011], Proteus [Hurley et al., 2014], SMAC [Hutter et al., 2011]30 (used in e.g. auto-sklearn[Feurer et al., 2015]) and
Revised: Among them are 3S [Kadioglu et al., 2011], Proteus [Hurley et al., 2014], SMAC [Hutter et al., 2011] (used in e.g. auto-sklearn

Original:  TPOT [Olson et al., 2016].
Revised: [Feurer et al., 2015]) and TPOT [Olson et al., 2016].

Original: The growing inter31 est in per-instance algorithm selection over the past decade has led to the establishment of the32 Algorithm Selection Benchmark Library, ASlib [Bischl et al., 2016], and to two competitions for33 algorithm selection systems [Lindauer et al., 2019].
Revised: The growing interest in per-instance algorithm selection over the past decade has led to the establishment of the Algorithm Selection Benchmark Library, ASlib [Bischl et al., 2016], and to two competitions for algorithm selection systems [Lindauer et al., 2019].

Original: 34 While these systems are based on different techniques and leverage different ideas and assump35 tions, they share a common trait – the algorithm(s) they leverage are treated as black boxes; that36 is, we know nothing about how the algorithms work internally.
Revised: While these systems are based on different techniques and leverage different ideas and assumptions, they share a common trait – the algorithm(s) they leverage are treated as black boxes; that is, we know nothing about how the algorithms work internally.

Original: Instead, we gather information37 by probing given algorithms repeatedly, usually by observing their performance in solving a set38 of problem instances.
Revised: Instead, we gather information by probing given algorithms repeatedly, usually by observing their performance in solving a set of problem instances.

Original: In this work, we argue that by ignoring information that can be obtained39 through the automatic analysis of algorithms, we leave untapped useful knowledge that can sub40 stantially improve the performance of meta-algorithmic approaches.
Revised: In this work, we argue that by ignoring information that can be obtained through the automatic analysis of algorithms, we leave untapped useful knowledge that can substantially improve the performance of meta-algorithmic approaches.

Original: We open the black box without41 compromising the general applicability of meta-algorithmic techniques by automatically analyzing the algorithms under consideration.
Revised: We open the black box without compromising the general applicability of meta-algorithmic techniques by automatically analyzing the algorithms under consideration.

Original: We demonstrate that considerable performance improvements43 can be achieved using this additional information for algorithm selection.
Revised: We demonstrate that considerable performance improvements can be achieved using this additional information for algorithm selection.

Original: The performance improvements achieved by our approach result in improved efficiencies, thus45 reducing the amount of computation necessary to solve a problem.
Revised: The performance improvements achieved by our approach result in improved efficiencies, thus reducing the amount of computation necessary to solve a problem.

Original: This reduction in computation46 can also contribute to a more sustainable and energy-efficient AI.
Revised: This reduction in computation can also contribute to a more sustainable and energy-efficient AI.

Original: We have not identified any47 potential negative societal impacts in either our work or possible applications of our work.
Revised: We have not identified any potential negative societal impacts in either our work or possible applications of our work.

Original: 2 Related Work
Revised: 2 Related Work

Original: Choosing the most promising approach for solving a given problem has always been of interest, in50 particular for hard combinatorial problems (e.g. [Allen and Minton, 1996, Weerawarana et al., 1996,51 Lobjois and Lemaître, 1998]).
Revised: Choosing the most promising approach for solving a given problem has always been of interest, in particular for hard combinatorial problems (e.g. [Allen and Minton, 1996, Weerawarana et al., 1996, Lobjois and Lemaître, 1998]).

Original: The idea of using portfolios for this purpose was introduced52 by [Huberman et al., 1997] and popularized in AI by [Gomes and Selman, 2001].
Revised: The idea of using portfolios for this purpose was introduced by [Huberman et al., 1997] and popularized in AI by [Gomes and Selman, 2001].

Original: These early ap53 proaches that leverage complementarity in the performance characteristics of algorithms did not54 perform algorithm selection as we consider it here, but rely heavily on parallel processing to be able55 to run several or all portfolio algorithms concurrently.
Revised: These early approaches that leverage complementarity in the performance characteristics of algorithms did not perform algorithm selection as we consider it here, but rely heavily on parallel processing to be able to run several or all portfolio algorithms concurrently.

Original: They were soon followed by approaches56 performing algorithm selection in the sense we consider here (e.g. [Guerri and Milano, 2004,57 Beck and Freuder, 2004, Hough and Williams, 2006]).
Revised: They were soon followed by approaches performing algorithm selection in the sense we consider here (e.g. [Guerri and Milano, 2004, Beck and Freuder, 2004, Hough and Williams, 2006]).

Original: However, it took several decades before algorithm selection procedures achieved signif59 icant performance improvements in practical settings.
Revised: However, it took several decades before algorithm selection procedures achieved significant performance improvements in practical settings.

Original: One of the first such systems is60 SATzilla [Leyton-Brown et al., 2003, Xu et al., 2008], which solves the prominent SAT problem61 using a rich set of instance features and draws from a broad set of high-performance SAT solvers.
Revised: One of the first such systems is SATzilla [Leyton-Brown et al., 2003, Xu et al., 2008], which solves the prominent SAT problem using a rich set of instance features and draws from a broad set of high-performance SAT solvers.

Original: 62 Different versions of SATzilla won several SAT competitions and prompted the development of63 further algorithm selection systems for SAT, e.g. [Roussel, 2012, Malitsky et al., 2013].
Revised: Different versions of SATzilla won several SAT competitions and prompted the development of further algorithm selection systems for SAT, e.g. [Roussel, 2012, Malitsky et al., 2013].

Original: In recent years, algorithm selection systems have gained prominence in many65 application domains, including
Revised: In recent years, algorithm selection systems have gained prominence in many application domains, including

Original: ASP [Hoos et al., 2014],
Revised: ASP [Hoos et al., 2014],

Original: CSP [Hurley et al., 2014],66 QBF
Revised: CSP [Hurley et al., 2014],

Original:  [Pulina and Tacchella, 2009], AI planning [Vallati, 2012], TSP [Kerschke et al., 2018],67 and operations research [Tierney and Malitsky, 2015].
Revised: QBF [Pulina and Tacchella, 2009], AI planning [Vallati, 2012], TSP [Kerschke et al., 2018], and operations research [Tierney and Malitsky, 2015].

Original: The growing interest in per-instance68 algorithm selection over the past decade has led to the establishment of the Algorithm Selection69 Benchmark Library, ASlib [Bischl et al., 2016], and to two competitions for algorithm selection sys70 tems [Lindauer et al., 2019].
Revised: The growing interest in per-instance algorithm selection over the past decade has led to the establishment of the Algorithm Selection Benchmark Library, ASlib [Bischl et al., 2016], and to two competitions for algorithm selection systems [Lindauer et al., 2019].

Original: We refer the interested reader to two surveys on algorithm selection for71 additional details and an overview of further work in the area [Kotthoff, 2014, Kerschke et al., 2019].
Revised: We refer the interested reader to two surveys on algorithm selection for additional details and an overview of further work in the area [Kotthoff, 2014, Kerschke et al., 2019].

Original: 72 Per-instance algorithm selection systems usually apply machine learning to determine which73 algorithm to run on a given problem instance, based on expert-defined, cheaply computable74 features of problem instances.
Revised: Per-instance algorithm selection systems usually apply machine learning to determine which algorithm to run on a given problem instance, based on expert-defined, cheaply computable features of problem instances.

Original: These features can either be used to directly predict the best75 performing algorithm (e.g. [Kadioglu et al., 2010, Gent et al., 2010, Pfahringer et al., 2000]), or to76 obtain performance predictions for all given algorithms and choose the algorithm with the best77 predicted performance; the latter approach was used in the original SATzilla system.
Revised: These features can either be used to directly predict the bestperforming algorithm (e.g. [Kadioglu et al., 2010, Gent et al., 2010, Pfahringer et al., 2000]), or to obtain performance predictions for all given algorithms and choose the algorithm with the best predicted performance; the latter approach was used in the original SATzilla system.

Original: 78 Recent work has applied deep learning techniques to obviate the need for human experts to79 explicitly define problem instance features [Loreggia et al., 2016].
Revised: Recent work has applied deep learning techniques to obviate the need for human experts to explicitly define problem instance features [Loreggia et al., 2016].

Original: While this initial study showed80 promising results, to the best of our knowledge, the approach has not been developed further.
Revised: While this initial study showed promising results, to the best of our knowledge, the approach has not been developed further.

Original: 81 Another line of work has applied deep learning more generally to directly learn algorithms and82 when to use them [Khalil et al., 2016, Khalil et al., 2017, Dai et al., 2017].
Revised: Another line of work has applied deep learning more generally to directly learn algorithms and when to use them [Khalil et al., 2016, Khalil et al., 2017, Dai et al., 2017].

Original: As most deep learning83 approaches, they require significant amounts of data and resources to train models compared to84 other approaches to algorithm selection though.
Revised: As most deep learning approaches, they require significant amounts of data and resources to train models compared to other approaches to algorithm selection though.

Revised: While there has been work on taking into the account additional features from the algorithms, as shown in [Tornede et al., 2020], this approach is much more limited than what we present here and requires an “extremely large” set of algorithms.

Revised: To the best of our knowledge, is only the case in this particular paper.

Revised: The algorithm features are latent, similar to recommender systems (as pointed out in the paper) and have no intrinsic meaning, as opposed to our algorithm features.

Revised: Additionally, representing an algorithm by its hyperparameters is a well-established way of taking information on the algorithm into account in the meta-algorithmic process (e.g. [Hutter et al., 2006]), but it requires expert knowledge – the developer must decide which hyperparameters and value ranges to expose.

Original: 85 We take a different approach  and  open the black box of the algorithms that are being modeled.
Revised: In contrast, our approach does not require this and is completely automated. Finally, [Adriaensen and Nowé, 2016] make a general point on opening up the “black box” in meta-algorithmics, in particular in the context of automated algorithm design. The paper proposed, among other things, to analyze the source code of algorithms in order to achieve their goal, but the authors did not present any results. We open the black box of the algorithms that are being modeled.

Original: 86 To the best of our knowledge, there are no existing techniques that allow to automatically analyze87 the inner workings of algorithms to facilitate per-instance algorithm selection.
Revised: To the best of our knowledge, there are no existing techniques that allow to automatically analyze the inner workings of algorithms to facilitate per-instance algorithm selection.

Original: 3 Methodology
Revised: 3 Methodology

Original: In many cases, the source code of an algorithm is available.
Revised: In many cases, the source code of an algorithm is available.

Original: The software engineering community90 has developed ways of quantifying the complexity and other properties of source code for decades91 – we can leverage their work for our purposes.
Revised: The software engineering community has developed ways of quantifying the complexity and other properties of source code for decades – we can leverage their work for our purposes.

Original: Such source code features can be computed quickly92 by readily-available tools for many different programming languages.
Revised: Such source code features can be computed quickly by readily-available tools for many different programming languages.

Original: 93 Our second source of information is the abstract syntax tree (AST) of the source code of an94 algorithm that is constructed as part of the compilation process.
Revised: Our second source of information is the abstract syntax tree (AST) of the source code of an algorithm that is constructed as part of the compilation process.

Original: It provides a higher-level view of95 the algorithm and makes connections between its individual components explicit.
Revised: It provides a higher-level view of the algorithm and makes connections between its individual components explicit.

Original: Moreover, the96 resulting graph structure can be analyzed and quantified using standard tools for the analysis of97 graphs.
Revised: Moreover, the resulting graph structure can be analyzed and quantified using standard tools for the analysis of graphs.

Original: Again, the tools are readily available and most features can be computed quickly.
Revised: Again, the tools are readily available and most features can be computed quickly.

Original: 98 Formally, following [Bischl et al., 2016], we have a set I of problem instances drawn from a99 distribution D , a space of algorithms A , a vector of instance features F I of arity ∥ I ∥ , a vector100 of algorithmic features F A of arity ∥
Revised: Formally, following [Bischl et al., 2016], we have a set I of problem instances drawn from a distribution D , a space of algorithms A , a vector of instance features F I of arity ∥ I ∥ , a vector of algorithmic features F A of arity ∥

Original: A ∥ , and a performance measure 𝑚 : I × A → R .
Revised: A ∥ , and a performance measure 𝑚 : I × A → R .

Original: The perinstance algorithm selection problem with algorithm features is to find a mapping 𝑠 : F I × F A
Revised: The perinstance algorithm selection problem with algorithm features is to find a mapping 𝑠 : F I × F A

Original: → A102 that optimizes E 𝑖 ∼ 𝐷 𝑚 ( 𝑖,𝑠 ( 𝑖 )) , i.e., the expected value of the performance measure for instances 𝑖103 distributed according to D , achieved by running the selected algorithm 𝑠
Revised: → A that optimizes E 𝑖 ∼ 𝐷 𝑚 ( 𝑖,𝑠 ( 𝑖 )) , i.e., the expected value of the performance measure for instances 𝑖 distributed according to D , achieved by running the selected algorithm 𝑠

Original: ( 𝑖 ) for instances 𝑖 .
Revised: ( 𝑖 ) for instances 𝑖 .

Original: The104 difference to the original definition in [Bischl et al., 2016] is that the mapping includes both instance105 ( F I ) and algorithmic features ( F A ).
Revised: The difference to the original definition in [Bischl et al., 2016] is that the mapping includes both instance ( F I ) and algorithmic features ( F A ).

Revised: Following the definitions above, the single best algorithm, or solver, is defined as the solver that has the best performance averaged across all instances, while the virtual best solver is a solver that perfectly selects the best solver from A on a per-instance basis.

Revised: More formally, the virtual best solver is defined as { 𝑎𝑟𝑔𝑚𝑖𝑛 𝑎 E 𝑖 ∼ 𝐷 𝑚 ( 𝑖,𝑠 ( 𝑖 )) | 𝑖 ∈ I , 𝑎 ∈ A } where 𝑖 and 𝑎 are individual instances and algorithms that come from the set of instances and algorithms, I and A , respectively.

Revised: Similarly, the single best solver is defined as 𝑎𝑟𝑔𝑚𝑖𝑛 𝑎

Revised: [Equation]

Original: 106 In addition to identifying sources of information about an algorithm, we need to develop ways of107 taking this information into account during performance modeling and algorithm selection.
Revised: I [Equation] ∥ I ∥ . In addition to identifying sources of information about an algorithm, we need to develop ways of taking this information into account during performance modeling and algorithm selection.

Original: Current108 state-of-the-art approaches to empirical performance modeling build one model per algorithm and109 predict its performance based on the features of a problem instance to solve [Hutter et al., 2014].
Revised: Current state-of-the-art approaches to empirical performance modeling build one model per algorithm and predict its performance based on the features of a problem instance to solve [Hutter et al., 2014].

Original: Simply adding the features of an algorithm to such models will not improve performance, as111 the features will be constant for a given model and algorithm.
Revised: Simply adding the features of an algorithm to such models will not improve performance, as the features will be constant for a given model and algorithm.

Original: However, the reason for building112 individual models for each algorithm is that otherwise there is no way to distinguish between them.
Revised: However, the reason for building individual models for each algorithm is that otherwise there is no way to distinguish between them.

Original: Our proposed algorithm features provide a way to do exactly that, obviating the need for multiple114 performance models – we can build a single, unified model for all algorithms.
Revised: Our proposed algorithm features provide a way to do exactly that, obviating the need for multiple performance models – we can build a single, unified model for all algorithms.

Original: 115 One of the advantages to this approach is that the single model takes much more information116 into account than individual models, making it possible, for example, to learn relationships between117 algorithms that can improve the quality of performance predictions.
Revised: One of the advantages to this approach is that the single model takes much more information into account than individual models, making it possible, for example, to learn relationships between algorithms that can improve the quality of performance predictions.

Original: Note that algorithm features118 do not only permit to distinguish between different algorithms, but also quantifying how similar119 they are – further useful information that a combined performance model can take into account.
Revised: Note that algorithm features do not only permit to distinguish between different algorithms, but also quantifying how similar they are – further useful information that a combined performance model can take into account.

Original: 120 We also consider models that predict the performance differences between pairs of algorithms121 and take the difference between the features of the respective algorithms in addition to the instance122 features into acocunt.
Revised: We also consider models that predict the performance differences between pairs of algorithms and take the difference between the features of the respective algorithms in addition to the instance features into acocunt.

Original: Again, we are able to use just a single unified model for an entire portfolio of123 algorithms.
Revised: Again, we are able to use just a single unified model for an entire portfolio of algorithms.

Original: 4 Empirical Evaluation
Revised: 4 Empirical Evaluation

Original: To evaluate the promise of our approach, we leveraged the widely used ASlib benchmark library [Bischl et al., 2016], which provides algorithm selection scenarios from many different application domains.
Revised: To evaluate the promise of our approach, we leveraged the widely used ASlib benchmark library [Bischl et al., 2016], which provides algorithm selection scenarios from many different application domains.

Original: All experiments were run on compute nodes with Intel Broadwell CPUs with128 cores clocked at 2.1 GHz, 40 MB of CPU cache, and 128 GB of RAM, running Red Hat Enterprise129 Linux 7.7.
Revised: All experiments were run on compute nodes with Intel Broadwell CPUs withcores clocked at 2.1 GHz, 40 MB of CPU cache, and 128 GB of RAM, running Red Hat Enterprise Linux 7.7.

Original: 4.1 Algorithm Selection Scenarios
Revised: 4.1 Algorithm Selection Scenarios

Original: As we require the source code for the algorithms to be available, we limited our empirical study132 to ASlib scenarios that satisfy this constraint.
Revised: As we require the source code for the algorithms to be available, we limited our empirical study to ASlib scenarios that satisfy this constraint.

Original: Further, we created additional scenarios for our133 evaluation and contributed these to ASlib.
Revised: Further, we created additional scenarios for our evaluation and contributed these to ASlib.

Original: Table 1 gives a summary of the scenarios we used.
Revised: Table 1 gives a summary of the scenarios we used.

Original: These scenarios represent a variety of different applications of algorithm selection.
Revised: These scenarios represent a variety of different applications of algorithm selection.

Original: Starred ASlib scenarios were modified to exclude algorithms for various reasons such as lack of available source136 code in the submission, lack of support for a particular language in our tools, etc.
Revised: Starred ASlib scenarios were modified to exclude algorithms for various reasons such as lack of available source code in the submission, lack of support for a particular language in our tools, etc.

Original: 137 For further details on how scenarios were modified, we refer the interested reader to Appendix A.
Revised: For further details on how scenarios were modified, we refer the interested reader to Appendix A.

Original: [Table]
Revised: [Table]

Original: 4.2 Algorithm Features
Revised: 4.2 Algorithm Features

Original: For each algorithm, we extracted both source code and AST features.
Revised: For each algorithm, we extracted both source code and AST features.

Original: Table 2 summarizes the140 algorithm features we use to evaluate the promise of our approach.
Revised: Table 2 summarizes the algorithm features we use to evaluate the promise of our approach.

Original: 141 Our code features were computed using an open-source tool [Metrix++, 2019].
Revised: Our code features were computed using an open-source tool [Metrix++, 2019].

Original: We computed142 cyclomatic complexity, maxindent complexity, number of lines of code, and size in bytes, for the143 entire source code of a given algorithm and the average values across different regions of the code.
Revised: We computed cyclomatic complexity, maxindent complexity, number of lines of code, and size in bytes, for the entire source code of a given algorithm and the average values across different regions of the code.

Original: The resulting feature set is computed within a few seconds for each of our algorithms.
Revised: The resulting feature set is computed within a few seconds for each of our algorithms.

Original: 145 Our AST features are based on the abstract syntax tree obtained during compilation of a given146 algorithm; we generated the ASTs using the open-source compiler Clang [Clang, 2019] and collected147 the proportion of nodes of each type (based on the Clang AST types); of edges that link each pair of148 types; and of each data-types to which the operators nodes are applied.
Revised: Our AST features are based on the abstract syntax tree obtained during compilation of a given algorithm; we generated the ASTs using the open-source compiler Clang [Clang, 2019] and collected the proportion of nodes of each type (based on the Clang AST types); of edges that link each pair of types; and of each data-types to which the operators nodes are applied.

Original: To generate graph features149 from the AST, we used the NetworkX library [NetworkX, 2019].
Revised: To generate graph features from the AST, we used the NetworkX library [NetworkX, 2019].

Original: We computed the number of nodes150 and edges; degrees of the nodes; transitivity; clustering coefficient [Fagiolo, 2007]; and depth.
Revised: We computed the number of nodes and edges; degrees of the nodes; transitivity; clustering coefficient [Fagiolo, 2007]; and depth.

Original: For151 degree and clustering coefficient, we compute minimum, maximum, mean, and variance across all152 nodes.
Revised: For degree and clustering coefficient, we compute minimum, maximum, mean, and variance across all nodes.

Original: For depth, we compute these measures across all leaves.
Revised: For depth, we compute these measures across all leaves.

Original: We also compute the entropy of153 the distributions for degree and depth across the entire tree and the leaf nodes, respectively.
Revised: We also compute the entropy of the distributions for degree and depth across the entire tree and the leaf nodes, respectively.

Original: The154 clustering coefficient is often zero, for which the entropy cannot be computed.
Revised: The clustering coefficient is often zero, for which the entropy cannot be computed.

Original: Computation of the155 AST features takes from a few seconds up to hours per algorithm, but is a one-time expense.
Revised: Computation of the AST features takes from a few seconds up to hours per algorithm, but is a one-time expense.

Original: 156 To investigate whether the algorithm features serve a purpose beyond purely distinguishing157 between different algorithms, i.e. to assess to which degree they are useful for algorithm selection158 for a given problem instance, we include a third set containing only a single dummy feature that159 represents a unique identifier of the algorithm. 
Revised: To investigate whether the algorithm features serve a purpose beyond purely distinguishing between different algorithms, i.e. to assess to which degree they are useful for algorithm selection for a given problem instance, we include a third set containing only a single dummy feature that represents a unique identifier of the algorithm. [Table] [Table] [Table]

Revised: [Table]

Original:  4.3 Algorithm Selection
Revised: identifier [Table] [Table] 4.3 Algorithm Selection

Original: There are different approaches to algorithm selection; we focus on the two methods described in162 Section 3 – models to predict the performance of each individual algorithm and models to predict163 the performance difference between pairs of algorithms.
Revised: There are different approaches to algorithm selection; we focus on the two methods described in Section 3 – models to predict the performance of each individual algorithm and models to predict the performance difference between pairs of algorithms.

Original: These models take our new algorithm164 features for individual algorithms and the difference of those features for pairs of algorithms into [Table]
Revised: These models take our new algorithm features for individual algorithms and the difference of those features for pairs of algorithms into account.

Original: account. Dummy identifier features are encoded in a one-hot way with a one indicating that a166 particular algorithm or pair of algorithms was measured, with features for all other algorithms or[Table] pairs of algorithms set to zero.
Revised: Dummy identifier features are encoded in a one-hot way with a one indicating that a particular algorithm or pair of algorithms was measured, with features for all other algorithms or pairs of algorithms set to zero.

Original: Each ASlib scenario defines 10 cross-validation folds to ensure repeatable and robust performance evaluation.
Revised: Each ASlib scenario defines 10 cross-validation folds to ensure repeatable and robust performance evaluation.

Original: The performance models are trained on the data from nine folds and their170 performance is evaluated on the remaining fold.
Revised: The performance models are trained on the data from nine folds and their performance is evaluated on the remaining fold.

Original: This is repeated for each combination of training and testing folds, for a total of ten iterations.
Revised: This is repeated for each combination of training and testing folds, for a total of ten iterations.

Original: We report the average performance across all172 folds and the standard deviation.
Revised: We report the average performance across all folds and the standard deviation.

Original: We use random forest regression models to predict the performance of each algorithm and imputed missing instance feature values in all scenarios by the174 mean value across all non-missing values for numeric features, and NA for categorical features,175 following [Bischl et al., 2016].
Revised: We use random forest regression models to predict the performance of each algorithm and imputed missing instance feature values in all scenarios by the mean value across all non-missing values for numeric features, and NA for categorical features, following [Bischl et al., 2016].

Original: 176 We performed basic hyperparameter optimization of the random forest models.
Revised: We performed basic hyperparameter optimization of the random forest models.

Original: Specifically, we177 varied the number of trees in the random forest from 10 to 200, and the number of features used at178 each split when growing trees from 1 to 30.
Revised: Specifically, we varied the number of trees in the random forest from 10 to 200, and the number of features used at each split when growing trees from 1 to 30.

Original: The hyperparameters and ranges are the same as in179 those used by [Bischl et al., 2016].
Revised: The hyperparameters and ranges are the same as in those used by [Bischl et al., 2016].

Original: We performed a random search in this parameter space with180 250 samples and a nested cross validation with ten outer and three inner folds, selecting the model181 with the lowest average PAR10 across the outer cross-validation folds for our comparison.
Revised: We performed a random search in this parameter space with 250 samples and a nested cross validation with ten outer and three inner folds, selecting the model with the lowest average PAR10 across the outer cross-validation folds for our comparison.

Original: 182 We consider two metrics for evaluating the performance of algorithm selection systems –183 misclassification penalty (MCP) and penalized average running time with a factor of 10 (PAR10).
Revised: We consider two metrics for evaluating the performance of algorithm selection systems – misclassification penalty (MCP) and penalized average running time with a factor of 10 (PAR10).

Original: The misclassification penalty is defined as the additional cost incurred because a sub-optimal185 algorithm was chosen, i.e. additional running time or decreased accuracy of a given machine186 learning model.
Revised: The misclassification penalty is defined as the additional cost incurred because a sub-optimal algorithm was chosen, i.e. additional running time or decreased accuracy of a given machine learning model.

Original: PAR10 is defined as the average running time over a given set of problem instances,187 with timeouts counted as ten times a user-specified cutoff time.
Revised: PAR10 is defined as the average running time over a given set of problem instances, with timeouts counted as ten times a user-specified cutoff time.

Original: 188 We compare to the single and virtual best solvers, which provide bounds on the performance of189 an algorithm selection system – it cannot be better than the virtual best solver (VBS) and should not be worse than the single best solver (SBS), as there is no performance gain from algorithm selection[Figure] otherwise.
Revised: We compare to the single and virtual best solvers, which provide bounds on the performance of an algorithm selection system – it cannot be better than the virtual best solver (VBS) and should not be worse than the single best solver (SBS), as there is no performance gain from algorithm selection otherwise.

Original: We normalize the performance of an algorithm selection system to the fraction of the[Figure] gap closed between single and virtual best (see e.g. [Bischl et al., 2016]).
Revised: We normalize the performance of an algorithm selection system to the fraction of the gap closed between single and virtual best (see e.g. [Bischl et al., 2016]).

Original: This allows to directly[Figure] compare performance across different scenarios.[Figure]
Revised: This allows to directly compare performance across different scenarios.

Original: Due to the size of the scenario and the amount of computation required, we were unable to195 consider all algorithms for SAT18-EXP when using models that predict performance difference.[Figure]
Revised: Due to the size of the scenario and the amount of computation required, we were unable to consider all algorithms for SAT18-EXP when using models that predict performance difference.

Original: Therefore we modified SAT18-EXP to exclude seven solvers.
Revised: Therefore we modified SAT18-EXP to exclude seven solvers.

Original: However all other scenarios were

Original: [Figure] used without modifications, and the full SAT18-EXP scenario was used for experiments where the198 performance of each algorithm is predicted directly and a selection is made based on this predicted[Figure] performance.
Revised: However all other scenarios were used without modifications, and the full SAT18-EXP scenario was used for experiments where the performance of each algorithm is predicted directly and a selection is made based on this predicted performance.

Original: For further details on what solvers were excluded and the motivation behind this[Figure] decision can be found in Appendix B.[Figure]
Revised: For further details on what solvers were excluded and the motivation behind this decision can be found in Appendix B.

Original: 5 Results
Revised: 5 Results

Original: [Figure]

Original: We now present the results of our experiments, demonstrating the promise of our algorithm features[Figure] and the performance improvements that can be achieved.[Figure]
Revised: We now present the results of our experiments, demonstrating the promise of our algorithm features and the performance improvements that can be achieved.

Original: 5.1 Algorithm Selection
Revised: 5.1 Algorithm Selection

Original: [Figure]

Original: Figure 1 shows the results of our experiments for algorithm selection in terms of the MCP metric.
Revised: Figure 1 shows the results of our experiments for algorithm selection in terms of the MCP metric.

Original: Adding solver features improves algorithm selection for the majority of the scenarios we considered;207 8 out of 9 scenarios for regression models and 6 out of 9 for pairwise regression models with an208 additional scenario resulting in a tie.
Revised: Adding solver features improves algorithm selection for the majority of the scenarios we considered; 8 out of 9 scenarios for regression models and 6 out of 9 for pairwise regression models with an additional scenario resulting in a tie.

Original: While adding algorithm features does not help for all scenarios,209 we do see substantial performance improvements in some cases, with improvements of up to 26%210 for regression models and 28% for pairwise regression models.
Revised: While adding algorithm features does not help for all scenarios, we do see substantial performance improvements in some cases, with improvements of up to 26% for regression models and 28% for pairwise regression models.

Original: 211 For the TSP-LION2015 scenario, adding solver features improves algorithm selection performance to the point where it is better than the single best solver, closing an additional 26% between213 single and virtual best solver in terms of MCP for the regression model and 28% for the pairwise214 regression model.
Revised: For the TSP-LION2015 scenario, adding solver features improves algorithm selection performance to the point where it is better than the single best solver, closing an additional 26% between single and virtual best solver in terms of MCP for the regression model and 28% for the pairwise regression model.

Original: For GRAPHS-2015, we close an additional 15% of the gap for regression and 7%215 for pairwise regression.
Revised: For GRAPHS-2015, we close an additional 15% of the gap for regression and 7% for pairwise regression.

Original: For SAT11-INDU, we close an additional 10% of the gap to the virtual best216 solver for regression and a more modest 6% of the gap for the pairwise regression model.
Revised: For SAT11-INDU, we close an additional 10% of the gap to the virtual best solver for regression and a more modest 6% of the gap for the pairwise regression model.

Original: We also 217 see impressive results in MAXSAT19-UCMS, where we close an additional 9% and 10% of the gap 218 for regression and pairwise regression, respectively.

Original: For other scenarios where we improve, the 219 performance difference is smaller, from 2% to 9% for regression and from 6% to 7% for pairwise 220 regression.

Original: We note that algorithm selection performance improves by several percentage points 221 on the GLUHACK-18 scenario, where all algorithms are modifications of a common code base.

Original: Even 222 when the code bases are very similar, the features we extract add useful information for algorithm 223 selection.

Original: For the scenarios where adding algorithm features decreases performance, we observe 224 smaller changes.

Original: On SAT18-EXP, we tie for pairwise regression.

Original: For SAT11-RAND, the decrease is 225 1% in the gap closed for both regression and pairwise regression approaches.

Original: 226 For comparison, we use dummy algorithm features that allow us to assess to what extent 227 the performance improvements we see are because we train a single model that can take the 228 performance information of all algorithms into account or because algorithm features genuinely 229 help to make the models more predictive. The results for performance predictions we presented in 230 the previous section already strongly indicate that algorithm features genuinely help. Adding the 231 dummy features decreases performance in the majority of cases. In most of the few cases where 232 performance improves compared to only instance features, the differences are small – including 233 our proposed algorithm features improves performance more often and by a larger amount.
Revised: We also

Original: This 234 conclusively demonstrates that algorithm features are useful for per-instance algorithm selection.

Original: We see a similar trend in the results for the experiments when we consider PAR10 scores 236 (Figure 2 in the appendix), where adding algorithm features improves performance for 6 out of

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Regression
Revised: Regression

Original: Pairwise Regression
Revised: Pairwise Regression

Original: I
Revised: I

Original: II
Revised: II

Original: IC
Revised: IC

Original: IA
Revised: IA

Original: ICA
Revised: ICA

Original: I
Revised: I

Original: II
Revised: II

Original: IC
Revised: IC

Original: IA
Revised: IA

Original: ICA
Revised: ICA

Original: GLUHACK−
Revised: GLUHACK−

Original: GRAPHS−2015*
Revised: GRAPHS−2015*

Original: MAXSAT19−UCMS
Revised: MAXSAT19−UCMS

Original: SAT03−16_INDU*
Revised: SAT03−16_INDU*

Original: SAT11−HAND*
Revised: SAT11−HAND*

Original: SAT11−INDU
Revised: SAT11−INDU

Original: SAT11−RAND*
Revised: SAT11−RAND*

Original: SAT18−EXP
Revised: SAT18−EXP

Original: TSP−LION worst baseline best 
Revised: TSP−LION worst baseline best see impressive results in MAXSAT19-UCMS, where we close an additional 9% and 10% of the gap for regression and pairwise regression, respectively. For other scenarios where we improve, the performance difference is smaller, from 2% to 9% for regression and from 6% to 7% for pairwise regression. We note that algorithm selection performance improves by several percentage points on the GLUHACK-18 scenario, where all algorithms are modifications of a common code base. Even when the code bases are very similar, the features we extract add useful information for algorithm selection.

Original: 9 scenarios  for both regression and pairwise regression approaches, with one tie.
Revised: For the scenarios where adding algorithm features decreases performance, we observe smaller changes. On SAT18-EXP, we tie for pairwise regression. For SAT11-RAND, the decrease is 1% in the gap closed for both regression and pairwise regression approaches. For comparison, we use dummy algorithm features that allow us to assess to what extent the performance improvements we see are because we train a single model that can take the performance information of all algorithms into account or because algorithm features genuinely help to make the models more predictive.

Revised: The results for performance predictions we presented in the previous section already strongly indicate that algorithm features genuinely help.

Revised: Adding the dummy features decreases performance in the majority of cases.

Original:  To determine238 statistical significance of our results, we used the Friedman test on the ranks achieved by every239 approach on each problem instance in a given scenario (total 45 cases), along with a Nemenyi240 post-hoc test.
Revised: In most of the few cases where performance improves compared to only instance features, the differences are small – including our proposed algorithm features improves performance more often and by a larger amount. This conclusively demonstrates that algorithm features are useful for per-instance algorithm selection. We see a similar trend in the results for the experiments when we consider PAR10 scores (Figure 2 in the appendix), where adding algorithm features improves performance for 6 out of 9 scenarios for both regression and pairwise regression approaches, with one tie. To determine statistical significance of our results, we used the Friedman test on the ranks achieved by every approach on each problem instance in a given scenario (total 45 cases), along with a Nemenyi post-hoc test.

Original: The difference in ranks between the baseline and our proposed approaches was241 statistically significant at 5% level for all scenarios, except in one case with regression models.
Revised: The difference in ranks between the baseline and our proposed approaches was statistically significant at 5% level for all scenarios, except in one case with regression models.

Original: The difference between instance features and our features was not statistically significant for the243 GRAPHS-2015 scenario.
Revised: The difference between instance features and our features was not statistically significant for the GRAPHS-2015 scenario.

Original: For pairwise regression models, the differences were statistically significant244 at the same level for all but five cases for the MAXSAT19-UCMS, SAT03-16_INDU, SAT11-HAND,245 SAT11-INDU, and TSP-LION2015 scenarios.
Revised: For pairwise regression models, the differences were statistically significant at the same level for all but five cases for the MAXSAT19-UCMS, SAT03-16_INDU, SAT11-HAND, SAT11-INDU, and TSP-LION2015 scenarios.

Original: 5.2 Algorithm Feature Importance
Revised: 5.2 Algorithm Feature Importance

Original: To provide additional insight into the usefulness of our algorithm features and to investigate the248 potential for further improvements, we conducted feature selection experiments for our algorithm249 features.
Revised: To provide additional insight into the usefulness of our algorithm features and to investigate the potential for further improvements, we conducted feature selection experiments for our algorithm features.

Original: Specifically, we performed forward selection without hyperparameter tuning to reduce250 experimental cost, following [Bischl et al., 2016].
Revised: Specifically, we performed forward selection without hyperparameter tuning to reduce experimental cost, following [Bischl et al., 2016].

Original: We only considered our algorithm features for251 selection and always use the full set of instance features; we made this choice to keep the experimental effort manageable, and because results for feature selection experiments on instance253 features are available in the literature [Bischl et al., 2016], showing that a small number of instance254 features are sufficient to achieve good performance.
Revised: We only considered our algorithm features for selection and always use the full set of instance features; we made this choice to keep the experimental effort manageable, and because results for feature selection experiments on instance features are available in the literature [Bischl et al., 2016], showing that a small number of instance features are sufficient to achieve good performance.

Original: We performed this analysis for both types of255 approaches; regression models and pairwise regression models.[Table] GRAPHS [Table]
Revised: We performed this analysis for both types of approaches; regression models and pairwise regression models.

Original: [Table]

Original: fraction of AST nodes that represent a statement fraction of AST edges that link a type node to a declaration node

Original: [Table]

Original: _INDU fraction of operators applied on long long fraction of operators applied on float

Original: [Table]

Original: HAND entropy of the distribution of the degree of AST nodes

Original: [Table]

Original: INDU average cyclomatic complexity fraction of AST edges that link a type node

Original: to a statement node SAT11-RAND average cyclomatic complexity SAT18-EXP total lines of code TSP-LION entropy of the distribution of the depth of AST leaves

Original: The selected features for regression models are shown in Table 3, results for pairwise regression257 models are shown in Table 6 in the appendix.
Revised: The selected features for regression models are shown in Table 3, results for pairwise regression models are shown in Table 6 in the appendix.

Original: They are not consistent across scenarios.
Revised: They are not consistent across scenarios.

Original: Features258 that measure the complexity of the algorithm in some way, such as average cyclomatic complexity,259 indentation, degree of AST nodes, and depth of AST leaves, are often chosen.
Revised: Features that measure the complexity of the algorithm in some way, such as average cyclomatic complexity, indentation, degree of AST nodes, and depth of AST leaves, are often chosen.

Original: Intuitively, it makes260 sense that they are predictive of the algorithm’s performance – more complex code might indicate261 more sophisticated heuristics, but also more complex computation.
Revised: Intuitively, it makes sense that they are predictive of the algorithm’s performance – more complex code might indicate more sophisticated heuristics, but also more complex computation.

Original: 262 Tables 4 and 5 in the appendix show the percent gap closed between single and virtual best263 solvers for the full and reduced feature sets.
Revised: Tables 4 and 5 in the appendix show the percent gap closed between single and virtual best solvers for the full and reduced feature sets.

Original: Our results show that performance can improve when264 irrelevant features are removed, and even a very small set of features is often sufficient for obtaining265 good performance – in no case, more than five features are selected, and in many cases just a 
Revised: Our results show that performance can improve when irrelevant features are removed, and even a very small set of features is often sufficient for obtaining good performance – in no case, more than five features are selected, and in many cases just a Scenario [Table] [Table]

Revised: fraction of AST

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: fraction of AST nodes

Revised: [Table]

Revised: represent a statement fraction of AST edges that link a type node to a declaration node

Revised: [Table]

Revised: 16_INDU fraction of operators

Revised: [Table]

Revised: on

Revised: [Table]

Revised: fraction

Revised: [Table]

Revised: operators

Revised: [Table]

Revised: float

Revised: [Table]

Revised: HAND entropy

Revised: [Table]

Revised: AST

Revised: [Table]

Revised: entropy of the distribution of

Revised: [Table]

Revised: [Table]

Original: single one.
Revised: single one.

Original: This echoes similar findings for the number of instance features in [Bischl et al., 2016].
Revised: This echoes similar findings for the number of instance features in [Bischl et al., 2016].

Original: For TSP-LION2015, performance is worse with the reduced set of features for regression models.
Revised: For TSP-LION2015, performance is worse with the reduced set of features for regression models.

Original: Adding a second feature to the one chosen previously does not improve performance and the feature269 selection process stops.
Revised: Adding a second feature to the one chosen previously does not improve performance and the feature selection process stops.

Original: There is likely a feature set that would lead to an increase in performance,[-
-] but we were unable to exhaustively test all feature subsets because of the computational cost.
Revised: There is likely a feature set that would lead to an increase in performance,[-
-] but we were unable to exhaustively test all feature subsets because of the computational cost.

Revised: 5.3 Discussion

Revised: We posit that the features we extracted from algorithms give useful information about those algorithms.

Revised: However, the features are at a relatively low level that does not, for example, allow to conclude what type of approach or algorithm was used.

Original:  6 Conclusions and Future Work
Revised: There is no intrinsic reason that any of the features should be related directly to the performance of the algorithm on particular inputs, just like there is no intrinsic reason that problem instance features should contain such information, for example, it is unclear why the number of connected components in a constraint graph tells us anything about how difficult the problem instance is to solve. We do see that algorithm selection performance improves, which in itself provides a motivation for our approach, again similar to how algorithm selection in general is motivated by the success of using instance features. Of course, our approach (like algorithm selection and, indeed, automated machine learning) does not always work, but we do show its promise through our experimental results. 6 Conclusions and Future Work

Original: We presented, to the best of our knowledge, the first approach that uses features characterizing273 software for algorithm selection.
Revised: We presented, to the best of our knowledge, the first approach that uses features characterizing software for algorithm selection.

Original: These features can be extracted automatically, using off-theshelf analysis software – as long as the source code for the given set of algorithms is available,275 our approach can be applied.
Revised: These features can be extracted automatically, using off-theshelf analysis software – as long as the source code for the given set of algorithms is available, our approach can be applied.

Original: We hence retain one of the main advantages of current black-box276 meta-algorithmic techniques – their broad availability.
Revised: We hence retain one of the main advantages of current black-box meta-algorithmic techniques – their broad availability.

Original: The results from our experiments clearly277 demonstrated that using these additional features can yield significant performance improvements278 for algorithm selection.
Revised: The results from our experiments clearly demonstrated that using these additional features can yield significant performance improvements for algorithm selection.

Original: 279 We also showed that only a small subset of the rich set of algorithm features we considered is280 required for achieving these improvements.
Revised: We also showed that only a small subset of the rich set of algorithm features we considered is required for achieving these improvements.

Original: While our approach does not improve performance281 for all scenarios we considered in our study, we achieved improvements on average across a wide282 variety of scenarios.
Revised: While our approach does not improve performance for all scenarios we considered in our study, we achieved improvements on average across a wide variety of scenarios.

Original: These results demonstrate the promise of our approach and the potential for283 algorithm features to improve algorithm selection and other meta-algorithmic techniques.
Revised: These results demonstrate the promise of our approach and the potential for algorithm features to improve algorithm selection and other meta-algorithmic techniques.

Original: Our work can be extended in various directions.
Revised: Our work can be extended in various directions.

Original: It would be interesting to consider scenarios285 where the performance metric of interest is the solution quality rather than runtime such as in286 machine learning.
Revised: It would be interesting to consider scenarios where the performance metric of interest is the solution quality rather than runtime such as in machine learning.

Original: This could potentially extend our approach into the domain of automated287 machine learning.
Revised: This could potentially extend our approach into the domain of automated machine learning.

Original: Furthermore, the type of software features we consider here is mostly limited to algorithm289 selection – for parameter tuning and algorithm configuration, where the difference a parameter290 makes is often not directly reflected in the code base and becomes only apparent at execution time,291 different software analysis techniques will be required to effectively distinguish between parameter292 configurations.
Revised: Furthermore, the type of software features we consider here is mostly limited to algorithm selection – for parameter tuning and algorithm configuration, where the difference a parameter makes is often not directly reflected in the code base and becomes only apparent at execution time, different software analysis techniques will be required to effectively distinguish between parameter configurations.

Original: While this requires significant further work, the ability to predict the performance293 of a parameter configuration more accurately has very substantial potential for impact across many294 areas of AI.
Revised: While this requires significant further work, the ability to predict the performance of a parameter configuration more accurately has very substantial potential for impact across many areas of AI.

Revised: Finally, a more in-depth analysis of the relationship between software features, instance features, and the performance of a given algorithm on a given problem instance could provide insight into how the novel features we propose here facilitate improved algorithm selection.

Revised: 6.1 Acknowlegments

Revised: Damir Pulatov and Lars Kotthoff are supported by NSF grant #1813537.

Original: 7 Reproducibility Checklist
Revised: 7 Reproducibility Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] [See Results section.]
Revised: [Yes] [See Results section.

Original:  (b) Did you describe the limitations of your work?
Revised: ] (b) Did you describe the limitations of your work?

Original: [Yes] [See Results section.]
Revised: [Yes] [See Results section.

Original:  (c) Did you discuss any potential negative societal impacts of your work?
Revised: ] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] [See Introduc- tion section.]
Revised: [Yes] [See Introduction section.]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: https://automl.cc/ethics-accessibility/
Revised: https://automl.cc/ethics-accessibility/

Original: [Yes]
Revised: [Yes]

Original: [The paper conforms to the guidelines.]
Revised: [The paper conforms to the guidelines.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: [No theoretical results.]
Revised: [No theoretical results.]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: [No theoretical results.] 
Revised: [No theoretical results.] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimen- tal results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental 
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)?

Original: material or as a url)?

Original: [No] [We have already modified algorithm selection libraries we references (llama and aslib) with our approach.
Revised: [No] [We have already modified algorithm selection libraries we references (llama and aslib) with our approach.

Original: Please do not check the libraries to ensure anonymity.]
Revised: Please do not check the libraries to ensure anonymity.]

Original: (b) Did you include the raw results of running the given instructions on the given code and data?
Revised: (b) Did you include the raw results of running the given instructions on the given code and data?

Original: [No] [We have used our own scripts in order to shows the results shown in the paper before we integrated our work into the standard libraries.]
Revised: [No] [We have used our own scripts in order to shows the results shown in the paper before we integrated our work into the standard libraries.]

Original: (c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
Revised: (c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?

Original: [No] [We have already modified algorithm selection libraries we references (llama and aslib) with our approach.
Revised: [No] [We have already modified algorithm selection libraries we references (llama and aslib) with our approach.

Original: Please do not check the libraries to ensure anonymity.]
Revised: Please do not check the libraries to ensure anonymity.]

Original: (d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?
Revised: (d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?

Original: [Yes]
Revised: [Yes]

Original: [The final code has extensive documentation.
Revised: [The final code has extensive documentation.

Original: Do not check the library to ensure anonymity of the submission.]
Revised: Do not check the library to ensure anonymity of the submission.]

Original: (e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?
Revised: (e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?

Original: [Yes] [See Empirical Evaluation section.]
Revised: [Yes] [See Empirical Evaluation section.]

Original: (f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?
Revised: (f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?

Original: [Yes] [See Empirical Evaluation section.]
Revised: [Yes] [See Empirical Evaluation section.

Original:  (g) Did you run ablation studies to assess the impact of different components of your approach?
Revised: ] (g) Did you run ablation studies to assess the impact of different components of your approach?

Original: [Yes] [See Algorithm Feature Importance section.]
Revised: [Yes] [See Algorithm Feature Importance section.

Original:  (h) Did you use the same evaluation protocol for the methods being compared?
Revised: ] (h) Did you use the same evaluation protocol for the methods being compared?

Original: [Yes] [See Methodology section.]
Revised: [Yes] [See Methodology section.]

Original: (i) Did you compare performance over time?
Revised: (i) Did you compare performance over time?

Original: [N/A]
Revised: [N/A]

Original: [The performance of our approach does[Figure] not change over time.][Figure]
Revised: [The performance of our approach does not change over time.]

Original: (j) Did you perform multiple runs of your experiments and report random seeds?
Revised: (j) Did you perform multiple runs of your experiments and report random seeds?

Original: [Yes] [See[Figure] Empirical Evaluation section (10 fold cross-validation).] [Figure]
Revised: [Yes] [See Empirical Evaluation section (10 fold cross-validation).

Original:  (k) Did you report error bars (e [Figure] g., with respect to the random seed after running experiments [Figure]
Revised: ] (k) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: multiple times)?

Original: [Yes] [See Results section.]
Revised: [Yes] [See Results section.

Original: [Figure]

Original: (l) Did you use tabular or surrogate benchmarks for in-depth evaluations?
Revised: (l) Did you use tabular or surrogate benchmarks for in-depth evaluations?

Original: [N/A]
Revised: [N/A]

Original: [Not applica [Figure] ble to our work.][Figure]
Revised: [Not applicable to our work.]

Original: (m) Did you include the total amount of compute and the type of resources used (e.g., type of[Figure] gpus, internal cluster, or cloud provider)?
Revised: (m) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)?

Original: [Yes] [See Empirical Evaluation section.]
Revised: [Yes] [See Empirical Evaluation section.]

Original: [Figure] (n) Did you report how you tuned hyperparameters, and what time and resources this required[Figure] (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and 
Revised: (n) Did you report how you tuned hyperparameters, and what time and resources this required (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and also hyperparameters of your own method)?

Original: [Figure]

Original: also hyperparameters of your own method)?

Original: [Yes] [See Empirical Evaluation section.]
Revised: [Yes] [See Empirical Evaluation section.

Original: [Figure]

Original: If you are using existing assets (e

Original: [Figure] g., code, data, models) or curating/releasing new assets. [Figure] (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] [See Empirical Evaluation[Figure] section.][Figure]
Revised: [Yes] [See Empirical Evaluation section.]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] [We have already contributed our datasets[Figure] to ASlib.
Revised: [No] [We have already contributed our datasets to ASlib.

Original: Introduced datasets have the same license.
Revised: Introduced datasets have the same license.

Original: Please do not check the libraries to[Figure] ensure anonymity.][Figure]
Revised: Please do not check the libraries to ensure anonymity.]

Original: (c) Did you include any new assets either in the supplemental material or as a url?
Revised: (c) Did you include any new assets either in the supplemental material or as a url?

Original: [No] [We[Figure] have already contributed our datasets and approach to standard libraries.
Revised: [No] [We have already contributed our datasets and approach to standard libraries.

Original: Please do not[Figure] check the libraries to ensure anonymity.]
Revised: Please do not check the libraries to ensure anonymity.]

Original: [Figure] (d) Did you discuss whether and how consent was obtained from people whose data you’re[Figure] using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: [No human subjects were involved.]
Revised: [No human subjects were involved.]

Original: [Figure]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [No human subjects were involved.]
Revised: [No human subjects were involved.

Original: If you used crowdsourcing or conducted research with human subjects..
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if appli- cable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [No human subjects were involved.]
Revised: [No human subjects were involved.]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [No human subjects were involved.]
Revised: [No human subjects were involved.]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [No human subjects were involved.]
Revised: [No human subjects were involved.]

Original: A Algorithm Selection Scenarios
Revised: A Algorithm Selection Scenarios

Original: We require the source code for the algorithms to be available, and hence limited our empirical study468 to ASlib scenarios for which we were able to obtain source code.
Revised: We require the source code for the algorithms to be available, and hence limited our empirical study to ASlib scenarios for which we were able to obtain source code.

Original: As this ruled out quite a few of469 the benchmarks, we created additional scenarios for our evaluation and contributed them back to470 ASlib.
Revised: As this ruled out quite a few of the benchmarks, we created additional scenarios for our evaluation and contributed them back to ASlib.

Original: We focus on scenarios that predict the runtime of algorithms in this paper; the methodology471 extends to scenarios with other performance measures as well though.
Revised: We focus on scenarios that predict the runtime of algorithms in this paper; the methodology extends to scenarios with other performance measures as well though.

Original: 472 We performed experiments on six pre-existing ASlib scenarios – TSP-LION2015, SAT11-RAND,473 SAT11-HAND, SAT11-INDU, SAT03-16_INDU, and GRAPHS-2015.
Revised: We performed experiments on six pre-existing ASlib scenarios – TSP-LION2015, SAT11-RAND, SAT11-HAND, SAT11-INDU, SAT03-16_INDU, and GRAPHS-2015.

Original: These scenarios represent a474 variety of different applications of algorithm selection.
Revised: These scenarios represent a variety of different applications of algorithm selection.

Original: 475 For four ASlib scenarios, we had to exclude some of the solvers.
Revised: For four ASlib scenarios, we had to exclude some of the solvers.

Original: In particular, for SAT11-RAND,476 we excluded sattime2011_2011.03.02, because we were unable to identify the source code and version477 corresponding to the scenario; for SAT11-HAND, we excluded jMiniSat_2011, sattime._2011.03.02,478 and sattime_2011.03.02 for the same reason.
Revised: In particular, for SAT11-RAND, we excluded sattime2011_2011.03.02, because we were unable to identify the source code and version corresponding to the scenario; for SAT11-HAND, we excluded jMiniSat_2011, sattime._2011.03.02, and sattime_2011.03.02 for the same reason.

Original: We also excluded Sol_2011.04.04, as our algorithm479 analysis tools currently do not support C# code.
Revised: We also excluded Sol_2011.04.04, as our algorithm analysis tools currently do not support C# code.

Original: For the SAT03-16_INDU scenario, we excluded480 cominisatps and riss5, again because we were unable to identify the exact version of the source481 code used for the scenario.
Revised: For the SAT03-16_INDU scenario, we excluded cominisatps and riss5, again because we were unable to identify the exact version of the source code used for the scenario.

Original: 482 The Glasgow algorithms from the GRAPHS-2015 scenario were four parameterizations of the483 same underlying algorithm.
Revised: The Glasgow algorithms from the GRAPHS-2015 scenario were four parameterizations of the same underlying algorithm.

Original: Therefore, we excluded three versions of Glasgow from the scenario.
Revised: Therefore, we excluded three versions of Glasgow from the scenario.

Original: B Algorithm Selection
Revised: B Algorithm Selection

Original: Due to the size of the scenario and the amount of computation required, we were unable to consider486 all algorithms for SAT18-EXP when using models that predict performance difference (666 models487 would be required for the full scenario to compare to existing approaches; our proposed approach488 does not suffer from this limitation and we omit solvers only to be able to compare to other489 approaches).
Revised: Due to the size of the scenario and the amount of computation required, we were unable to consider all algorithms for SAT18-EXP when using models that predict performance difference (666 models would be required for the full scenario to compare to existing approaches; our proposed approach does not suffer from this limitation and we omit solvers only to be able to compare to other approaches).

Original: We removed seven solvers that were the best on fewer than six out of 343 problem490 instances: Candy, Minisat.v2.2.0.106.ge2dd095, Lingeling, Riss7.1.fix, Sparrow2Riss.2018.fixfix,491 glucose3.0, and expGlucose.
Revised: We removed seven solvers that were the best on fewer than six out of 343 problem instances: Candy, Minisat.v2.2.0.106.ge2dd095, Lingeling, Riss7.1.fix, Sparrow2Riss.2018.fixfix, glucose3.0, and expGlucose.

Original: The first two of these were never the best for any of the instances.
Revised: The first two of these were never the best for any of the instances.

Original: This leaves us with 30 algorithms in this scenario for 435 pairwise performance models; the overall493 performance did not change significantly.
Revised: This leaves us with 30 algorithms in this scenario for 435 pairwise performance models; the overall performance did not change significantly.

Original: The mean MCP for the single best solver (the individual494 solver that performs best on average across all problem instances) decreased from the original495 1220.06 to 1211.12 CPU seconds, and the mean PAR10 score for the virtual best solver (the oracle496 that for each problem instance determines, without error, the best algorithm for solving it) increased497 from the original 9841.23 to 9850.17 CPU seconds.
Revised: The mean MCP for the single best solver (the individual solver that performs best on average across all problem instances) decreased from the original 1220.06 to 1211.12 CPU seconds, and the mean PAR10 score for the virtual best solver (the oracle that for each problem instance determines, without error, the best algorithm for solving it) increased from the original 9841.23 to 9850.17 CPU seconds.

Original: The PAR10 score for the single best solver and498 MCP for the VBS remained the same.
Revised: The PAR10 score for the single best solver and MCP for the VBS remained the same.

Original: All other scenarios were used without modifications, and the499 full SAT18-EXP scenario was used for experiments where the performance of each algorithm is500 predicted directly and a selection is made based on this predicted performance.
Revised: All other scenarios were used without modifications, and the full SAT18-EXP scenario was used for experiments where the performance of each algorithm is predicted directly and a selection is made based on this predicted performance.

Original: C Results
Revised: C Results

Original: When considering results in terms of PAR10 scores, we see the largest performance improvements503 for TSP-LION2015, where we close additional 71% and 48% of the gap for regression and pairwise504 regression models, respectively.
Revised: When considering results in terms of PAR10 scores, we see the largest performance improvements for TSP-LION2015, where we close additional 71% and 48% of the gap for regression and pairwise regression models, respectively.

Original: For SAT11-INDU, we close 19% more of the gap for both regression505 and pairwise regression models; 3% and 18% more compared to dummy features.
Revised: For SAT11-INDU, we close 19% more of the gap for both regression and pairwise regression models; 3% and 18% more compared to dummy features.

Original: For MAXSAT19UCMS, we close additional 10% and 8% gap for regression and pairwise regression, respectively.
Revised: For MAXSAT19UCMS, we close additional 10% and 8% gap for regression and pairwise regression, respectively.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Regression
Revised: Regression

Original: Pairwise
Revised: Pairwise

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: II
Revised: II

Original: IC
Revised: IC

Original: IA
Revised: IA

Original: ICA
Revised: ICA

Original: I
Revised: I

Original: II
Revised: II

Original: IC
Revised: IC

Original: IA
Revised: IA

Original: ICA
Revised: ICA

Original: GLUHACK−
Revised: GLUHACK−

Original: GRAPHS−2015*
Revised: GRAPHS−2015*

Original: MAXSAT19−UCMS
Revised: MAXSAT19−UCMS

Original: SAT03−16_INDU*
Revised: SAT03−16_INDU*

Original: SAT11−HAND*
Revised: SAT11−HAND*

Original: SAT11−INDU
Revised: SAT11−INDU

Original: SAT11−RAND*
Revised: SAT11−RAND*

Original: SAT18−EXP
Revised: SAT18−EXP

Original: TSP−LION
Revised: TSP−LION

Original: [Figure]
Revised: [Figure]

Original: Scenario
Revised: Scenario

Original: [Table]
Revised: [Table]

Original: GLUHACK
Revised: GLUHACK

Original: [Table] [Table]
Revised: [Table] [Table]

Original: GRAPHS-
Revised: GRAPHS-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: HAND
Revised: HAND

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Scenario
Revised: Scenario

Original: Selected Features
Revised: Selected Features

Original: GLUHACKaverage cyclomatic complexity variance of AST node degrees fraction of AST nodes that represent a type
Revised: GLUHACKaverage cyclomatic complexity variance of AST node degrees fraction of AST nodes that represent a type

Original: GRAPHSaverage maximum indent
Revised: GRAPHSaverage maximum indent

Original: MAXSAT19-UCMS fraction of AST nodes that represent a statement fraction of AST edges that link a type node to a declaration node
Revised: MAXSAT19-UCMS fraction of AST nodes that represent a statement fraction of AST edges that link a type node to a declaration node

Original: SAT03-16_INDU fraction of operators applied on long long fraction of operators applied on float
Revised: SAT03-16_INDU fraction of operators applied on long long fraction of operators applied on float

Original: SAT11-HAND entropy of the distribution of the degree of AST nodes
Revised: SAT11-HAND entropy of the distribution of the degree of AST nodes

Original: SAT11-INDU average cyclomatic complexity fraction of AST edges that link a type node to a statement node
Revised: SAT11-INDU average cyclomatic complexity fraction of AST edges that link a type node to a statement node

Original: SAT11-RAND average cyclomatic complexity
Revised: SAT11-RAND average cyclomatic complexity

Original: SAT18-EXP entropy of the distribution of the degree of AST nodes mean of the AST clustering coefficient variance of the AST clustering coefficient maximum depth of AST leaves fraction of AST nodes that represent a literal
Revised: SAT18-EXP entropy of the distribution of the degree of AST nodes mean of the AST clustering coefficient variance of the AST clustering coefficient maximum depth of AST leaves fraction of AST nodes that represent a literal

Original: TSP-LION entropy of the distribution of the depth of AST leaves
Revised: TSP-LION entropy of the distribution of the depth of AST leaves