Original: E CONOMICAL H YPERPARAMETER O PTIMIZATION WITH B LENDED S EARCH S TRATEGY
Revised: E CONOMICAL H YPERPARAMETER O PTIMIZATION WITH B LENDED S EARCH S TRATEGY

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of using low cost to search for hyperparameter conﬁgurations in a large search space with heterogeneous evaluation cost and model quality.
Revised: We study the problem of using low cost to search for hyperparameter conﬁgurations in a large search space with heterogeneous evaluation cost and model quality.

Original: We propose a blended search strategy to combine the strengths of global and local search, and prioritize them on the ﬂy with the goal of minimizing the total cost spent in ﬁnding good conﬁgurations.
Revised: We propose a blended search strategy to combine the strengths of global and local search, and prioritize them on the ﬂy with the goal of minimizing the total cost spent in ﬁnding good conﬁgurations.

Original: Our approach demonstrates robust performance for tuning both tree-based models and deep neural networks on a large
Revised: Our approach demonstrates robust performance for tuning both tree-based models and deep neural networks on a large

Original: AutoML benchmark, as well as superior performance in model quality, time, and resource consumption for a production transformer-based NLP model ﬁne-tuning task.
Revised: AutoML benchmark, as well as superior performance in model quality, time, and resource consumption for a production transformer-based NLP model ﬁne-tuning task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Hyperparameter optimization (HPO) of modern machine learning models is a resource-consuming task, which is unaffordable to individuals or organizations with little resource (Yang & Shami, 2020)
Revised: Hyperparameter optimization (HPO) of modern machine learning models is a resource-consuming task, which is unaffordable to individuals or organizations with little resource (Yang & Shami, 2020)

Original: Operating HPO in a low-cost regime has numerous beneﬁts, such as democratizing ML techniques, enabling new applications of ML, which requires frequent low-latency tuning, and reducing the carbon footprint.
Revised: Operating HPO in a low-cost regime has numerous beneﬁts, such as democratizing ML techniques, enabling new applications of ML, which requires frequent low-latency tuning, and reducing the carbon footprint.

Original: It is inherently challenging due to the nature of the task: trying a large number of conﬁgurations of heterogeneous cost and accuracy in a large search space.
Revised: It is inherently challenging due to the nature of the task: trying a large number of conﬁgurations of heterogeneous cost and accuracy in a large search space.

Original: The expense can accumulate from multiple sources: either a large number of individually cheap trials or a small number of expensive trials can add up the required resources.
Revised: The expense can accumulate from multiple sources: either a large number of individually cheap trials or a small number of expensive trials can add up the required resources.

Original: There have been multiple attempts to address the efﬁciency of HPO from different perspectives.
Revised: There have been multiple attempts to address the efﬁciency of HPO from different perspectives.

Original: Each of them has strengths and limitations.
Revised: Each of them has strengths and limitations.

Original: For example, Bayesian optimization (BO) (Brochu et al., 2010), which is a class of global optimization algorithms is used to minimize the total number of iterations to reach global optima.
Revised: For example, Bayesian optimization (BO) (Brochu et al., 2010), which is a class of global optimization algorithms is used to minimize the total number of iterations to reach global optima.

Original: However, when the cost of different hyperparameter conﬁgurations is heterogeneous, vanilla BO may select a conﬁguration that incurs unnecessarily high cost.
Revised: However, when the cost of different hyperparameter conﬁgurations is heterogeneous, vanilla BO may select a conﬁguration that incurs unnecessarily high cost.

Original: As opposed to BO, local search (LS) methods (Wu et al., 2021) are able to control total cost by preventing very expensive trials until necessary, but they may get trapped in local optima.
Revised: As opposed to BO, local search (LS) methods (Wu et al., 2021) are able to control total cost by preventing very expensive trials until necessary, but they may get trapped in local optima.

Original: Multi-ﬁdelity methods (Jamieson & Talwalkar, 2016) aim to use cheap proxies to replace some of the expensive trials and approximate the accuracy assessment, but can only be used when such proxies exist.
Revised: Multi-ﬁdelity methods (Jamieson & Talwalkar, 2016) aim to use cheap proxies to replace some of the expensive trials and approximate the accuracy assessment, but can only be used when such proxies exist.

Original: A single search strategy is difﬁcult to meet the generic goal of economical HPO.
Revised: A single search strategy is difﬁcult to meet the generic goal of economical HPO.

Original: In this work, we propose a blended search strategy which combines global search and local search strategy such that we can enjoy beneﬁts from both worlds: (1) global search can ensure the convergence to the global optima when the budget is sufﬁcient; and (2) local search methods enable a better control on the cost incurred along the search trajectory.
Revised: In this work, we propose a blended search strategy which combines global search and local search strategy such that we can enjoy beneﬁts from both worlds: (1) global search can ensure the convergence to the global optima when the budget is sufﬁcient; and (2) local search methods enable a better control on the cost incurred along the search trajectory.

Original: Given a particular global and local search method, our framework, which is named as BlendSearch , combines them according to the following design principles.
Revised: Given a particular global and local search method, our framework, which is named as BlendSearch , combines them according to the following design principles.

Original: (1) Instead of sticking with a particular method for conﬁguration selection, we consider both of the candidate search methods and decide which one to use at each round of the conﬁguration selection.
Revised: (1) Instead of sticking with a particular method for conﬁguration selection, we consider both of the candidate search methods and decide which one to use at each round of the conﬁguration selection.

Original: (2) We use the global search method to help decide the starting points of local search threads.
Revised: (2) We use the global search method to help decide the starting points of local search threads.

Original: (3) We use the local search method to intervene the global search method’s conﬁguration selection to avoid conﬁgurations that may incur unnecessarily large evaluation cost.
Revised: (3) We use the local search method to intervene the global search method’s conﬁguration selection to avoid conﬁgurations that may incur unnecessarily large evaluation cost.

Original: (4) We prioritize search instances of both methods according to their performance and efﬁciency of performance improvement on the ﬂy.
Revised: (4) We prioritize search instances of both methods according to their performance and efﬁciency of performance improvement on the ﬂy.

Original: Extensive empirical evaluation on the AutoML
Revised: Extensive empirical evaluation on the AutoML

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Benchmark (Gijsbers et al., 2019) validates the robust performance of our method on a wide variety of datasets.
Revised: Benchmark (Gijsbers et al., 2019) validates the robust performance of our method on a wide variety of datasets.

Original: BlendSearch is now publicly available in an open-source AutoML Library
Revised: BlendSearch is now publicly available in an open-source AutoML Library

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: We ﬁrst brieﬂy introduce the vanilla Bayesian optimization methods and local search methods, which are among the building blocks of our method.
Revised: We ﬁrst brieﬂy introduce the vanilla Bayesian optimization methods and local search methods, which are among the building blocks of our method.

Original: Bayesian optimization is a class of global optimization algorithms which is suitable for optimizing expensive black-box functions.
Revised: Bayesian optimization is a class of global optimization algorithms which is suitable for optimizing expensive black-box functions.

Original: It models the probabilistic distribution of the objective conditioned on the optimization variables.
Revised: It models the probabilistic distribution of the objective conditioned on the optimization variables.

Original: Typical models include Gaussian process (Snoek et al., 2012), random forest (Hutter et al., 2011), and tree
Revised: Typical models include Gaussian process (Snoek et al., 2012), random forest (Hutter et al., 2011), and tree

Original: Parzen estimator (TPE) (Bergstra et al., 2011).
Revised: Parzen estimator (TPE) (Bergstra et al., 2011).

Original: In BO methods, an acquisition function is used to determine the next point to evaluate.
Revised: In BO methods, an acquisition function is used to determine the next point to evaluate.

Original: Two common acquisition functions are the expected improvement (EI) (Bull, 2011) over the currently best-observed objective and upper conﬁdence bound (UCB) (Srinivas et al., 2009).
Revised: Two common acquisition functions are the expected improvement (EI) (Bull, 2011) over the currently best-observed objective and upper conﬁdence bound (UCB) (Srinivas et al., 2009).

Original: Local search methods are prevalent in the general optimization literature (Spall et al., 1992; Nesterov & Spokoiny, 2017) but less studied in the HPO literature due to the possibility of getting trapped in local optima (Gy¨orgy & Kocsis, 2011).
Revised: Local search methods are prevalent in the general optimization literature (Spall et al., 1992; Nesterov & Spokoiny, 2017) but less studied in the HPO literature due to the possibility of getting trapped in local optima (Gy¨orgy & Kocsis, 2011).

Original: Recent work (Wu et al., 2021) shows that a local search method FLOW 2 can make HPO cost-effective when combined with low-cost initialization and random restart.
Revised: Recent work (Wu et al., 2021) shows that a local search method FLOW 2 can make HPO cost-effective when combined with low-cost initialization and random restart.

Original: At each iteration, it samples a pair of vectors (with opposite directions) uniformly at random from a unit sphere, the center of which is the best conﬁguration found so far (a.k.a. incumbent) and the radius of which is the current stepsize.
Revised: At each iteration, it samples a pair of vectors (with opposite directions) uniformly at random from a unit sphere, the center of which is the best conﬁguration found so far (a.k.a. incumbent) and the radius of which is the current stepsize.

Original: Expensive conﬁgurations are avoided in the beginning as each iteration proposes a conﬁguration near the incumbent.
Revised: Expensive conﬁgurations are avoided in the beginning as each iteration proposes a conﬁguration near the incumbent.

Original: Random restart of the local search is performed once the convergence condition is satisﬁed.
Revised: Random restart of the local search is performed once the convergence condition is satisﬁed.

Original: There are several attempts to address the limitations of vanilla BO or local search methods.
Revised: There are several attempts to address the limitations of vanilla BO or local search methods.

Original: BOwLS (BO with local search) (Gao et al., 2020) uses a BO model to select the starting point of a local search thread.
Revised: BOwLS (BO with local search) (Gao et al., 2020) uses a BO model to select the starting point of a local search thread.

Original: Each local search thread is run until convergence and the BO model is updated with the start point and the converged loss.
Revised: Each local search thread is run until convergence and the BO model is updated with the start point and the converged loss.

Original: Trust region BO (Eriksson et al., 2019) ﬁts a ﬁxed number of local models and performs a principled global allocation of samples across these models via an implicit bandit approach.
Revised: Trust region BO (Eriksson et al., 2019) ﬁts a ﬁxed number of local models and performs a principled global allocation of samples across these models via an implicit bandit approach.

Original: It is primarily designed for HPO problems with high-dimensional numerical hyperparamters.
Revised: It is primarily designed for HPO problems with high-dimensional numerical hyperparamters.

Original: Unfortunately, all existing work that tries to combine global search with local search methods does not consider the heterogeneity of evaluation cost incurred along with the search.
Revised: Unfortunately, all existing work that tries to combine global search with local search methods does not consider the heterogeneity of evaluation cost incurred along with the search.

Original: There are also a lot of attempts in making HPO efﬁcient by speeding up conﬁguration evaluation.
Revised: There are also a lot of attempts in making HPO efﬁcient by speeding up conﬁguration evaluation.

Original: Multiﬁdelity optimizations (Klein et al., 2017; Li et al., 2017; Kandasamy et al., 2017; Falkner et al., 2018;
Revised: Multiﬁdelity optimizations (Klein et al., 2017; Li et al., 2017; Kandasamy et al., 2017; Falkner et al., 2018;

Original: Lu et al., 2019; Li et al., 2020) are proposed for this purpose.
Revised: Lu et al., 2019; Li et al., 2020) are proposed for this purpose.

Original: They usually require an additional degree of freedom in the problem called ‘ﬁdelity’, to allow performance assessment on a conﬁgura- tion with different ﬁdelities.
Revised: They usually require an additional degree of freedom in the problem called ‘ﬁdelity’, to allow performance assessment on a conﬁgura- tion with different ﬁdelities.

Original: There is surprisingly little prior work for generic cost-effective HPO.
Revised: There is surprisingly little prior work for generic cost-effective HPO.

Original: Gaussian process with expected improvement per second (GPEIPS) (Snoek et al., 2012) models the evaluation cost using another Gaussian process, and heuristically adds the estimated cost into the acquisition function.
Revised: Gaussian process with expected improvement per second (GPEIPS) (Snoek et al., 2012) models the evaluation cost using another Gaussian process, and heuristically adds the estimated cost into the acquisition function.

Original: It does not always outperform GPEI as the acqusition function can overpenalize good but expensive conﬁgurations.
Revised: It does not always outperform GPEI as the acqusition function can overpenalize good but expensive conﬁgurations.

Original:  B LEND S EARCH
Revised:  B LEND S EARCH

Original: Our framework needs the following information as inputs.
Revised: Our framework needs the following information as inputs.

Original: • B is the total budget of cost.
Revised: • B is the total budget of cost.

Original: In this work, we measure the cost by CPU/GPU time.
Revised: In this work, we measure the cost by CPU/GPU time.

Original: • P is the input HPO problem, which has the following attributes characterizing the problem.
Revised: • P is the input HPO problem, which has the following attributes characterizing the problem.

Original: – P. X is the search space in which
Revised: – P. X is the search space in which

Original: each element x is a d -dimensional hyperparameter conﬁguration.
Revised: each element x is a d -dimensional hyperparameter conﬁguration.

Original: For a non-categorical hyperparameter coordinate i ∈ [ d ] , if different values of x i lead to heterogeneous cost and there is a known value P.x LowCosti corresponding to a low cost, it is considered as a controlled dimension.
Revised: For a non-categorical hyperparameter coordinate i ∈ [ d ] , if different values of x i lead to heterogeneous cost and there is a known value P.x LowCosti corresponding to a low cost, it is considered as a controlled dimension.

Original: We use P.D to denote such a set of controlled dimensions.
Revised: We use P.D to denote such a set of controlled dimensions.

Original: – P. LossFunc ( · ) is the loss function to be minimized in terms of the conﬁgurations x ∈ P. X . – P. CostFunc ( · ) is the cost function that outputs the cost incurred when evaluating x .
Revised: – P. LossFunc ( · ) is the loss function to be minimized in terms of the conﬁgurations x ∈ P. X . – P. CostFunc ( · ) is the cost function that outputs the cost incurred when evaluating x .

Original: The goal of an HPO algorithm is to minimize the loss P. LossFunc ( x ) with the constraint that the total cost incurred G ( π ) := (cid:80) x ∈I ( π ) P. CostFunc ( x ) ≤ B , where I ( π ) is the search trajectory of algorithm π .
Revised: The goal of an HPO algorithm is to minimize the loss P. LossFunc ( x ) with the constraint that the total cost incurred G ( π ) := (cid:80) x ∈I ( π ) P. CostFunc ( x ) ≤ B , where I ( π ) is the search trajectory of algorithm π .

Original: Note that both P. LossFunc ( x ) and P. CostFunc ( x ) are black-box functions meaning that typically the analytic form is not available, and only function values can be observed.
Revised: Note that both P. LossFunc ( x ) and P. CostFunc ( x ) are black-box functions meaning that typically the analytic form is not available, and only function values can be observed.

Original: In order to distinguish the operation of querying from loss/cost function and the loss/cost observation, we use P. LossFunc ( x ) and P.
Revised: In order to distinguish the operation of querying from loss/cost function and the loss/cost observation, we use P. LossFunc ( x ) and P.

Original: CostFunc ( x ) to denote the former, and use l ( x ) and c ( x ) to denote the latter.
Revised: CostFunc ( x ) to denote the former, and use l ( x ) and c ( x ) to denote the latter.

Original: l and c (omitting x ) are used when there is no ambiguity.
Revised: l and c (omitting x ) are used when there is no ambiguity.

Original: • G is the global search method to be used.
Revised: • G is the global search method to be used.

Original: L is the local search method to be used.
Revised: L is the local search method to be used.

Original: L .
Revised: L .

Original: ∆ is the largest stepsize used in local search method L , i
Revised: ∆ is the largest stepsize used in local search method L , i

Original: e., the largest possible change on a hyperparameter value between two consecutive search steps.
Revised: e., the largest possible change on a hyperparameter value between two consecutive search steps.

Original: The overall design of our framework is presented in Figure 2 and Algorithm 1.
Revised: The overall design of our framework is presented in Figure 2 and Algorithm 1.

Original: The key idea is to maintain a pool of search threads, one of which corresponds to global search and the others local search.
Revised: The key idea is to maintain a pool of search threads, one of which corresponds to global search and the others local search.

Original: The pool starts with one global search thread and gradually adds local search threads as the search goes on.
Revised: The pool starts with one global search thread and gradually adds local search threads as the search goes on.

Original: Here a search thread is an instance of a global search or local search method, each with its own search trajectory.
Revised: Here a search thread is an instance of a global search or local search method, each with its own search trajectory.

Original: At each round, a search thread selector selects one of the search threads from the pool according to a priority metric that reﬂects the search threads’ current performance and efﬁciency of performance improvement.
Revised: At each round, a search thread selector selects one of the search threads from the pool according to a priority metric that reﬂects the search threads’ current performance and efﬁciency of performance improvement.

Original: The selected search thread will then be used to propose a conﬁguration to evaluate in this round.
Revised: The selected search thread will then be used to propose a conﬁguration to evaluate in this round.

Original: When the selected search thread is the global search thread, a conﬁg validator ﬁrst checks whether the proposed conﬁguration is within the ‘admissible’ region for evaluation.
Revised: When the selected search thread is the global search thread, a conﬁg validator ﬁrst checks whether the proposed conﬁguration is within the ‘admissible’ region for evaluation.

Original: If not, it uses a backup local search thread instead.
Revised: If not, it uses a backup local search thread instead.

Original: A local search thread will be created only when the global search thread proposes a valid conﬁg and a certain create thread condition is met, and will be deleted once it converges.
Revised: A local search thread will be created only when the global search thread proposes a valid conﬁg and a certain create thread condition is met, and will be deleted once it converges.

Original: Two local search threads will be merged into one if they are close enough.
Revised: Two local search threads will be merged into one if they are close enough.

Original: The priority of each search thread is updated after each evaluation round.
Revised: The priority of each search thread is updated after each evaluation round.

Original: Algorithm
Revised: Algorithm

Original: Inputs: HPO problem P , a global search method G , a local search method L , total budget B .
Revised: Inputs: HPO problem P , a global search method G , a local search method L , total budget B .

Original: 1: Initialization: Initialize F .
Revised: 1: Initialization: Initialize F .

Original: S =
Revised: S =

Original: [ S 0 ] where S 0 is an instance of G .
Revised: [ S 0 ] where S 0 is an instance of G .

Original: We denote by F . x
Revised: We denote by F . x

Original: 0 the initial point of the search.
Revised: 0 the initial point of the search.

Original: By design, the values of the controlled dimensions of F . x 0 are set to be P. x LowCost and those for the other dimensions are proposed by S 0 .
Revised: By design, the values of the controlled dimensions of F . x 0 are set to be P. x LowCost and those for the other dimensions are proposed by S 0 .

Original: 2: while F .c <
Revised: 2: while F .c <

Original: B do 3: ˜ S, ˜ S bak ← SelectThread ( F ) 4: ( x , l, c ) ← SearchEvaluate1Step ( ˜ S, F , P ) 5: if x is invalid then ( x , l, c ) ← SearchEvaluate1Step ( ˜ S bak , F , P ) 6: if x is proposed by global search & CreateNewLSCondition is satisﬁed then 7:
Revised: B do 3: ˜ S, ˜ S bak ← SelectThread ( F ) 4: ( x , l, c ) ← SearchEvaluate1Step ( ˜ S, F , P ) 5: if x is invalid then ( x , l, c ) ← SearchEvaluate1Step ( ˜ S bak , F , P ) 6: if x is proposed by global search & CreateNewLSCondition is satisﬁed then 7:

Original: Initialize S =
Revised: Initialize S =

Original: InitializeSearchThread
Revised: InitializeSearchThread

Original: ( L , P, ( x , l, c )) 8:
Revised: ( L , P, ( x , l, c )) 8:

Original: Add the new LS thread S into the pool: F . S ← F .
Revised: Add the new LS thread S into the pool: F . S ← F .

Original: S + S 9:
Revised: S + S 9:

Original: DeleteAndMergeLS ( F ) (cid:46) Merge or delete existing LS threads when necessary 10:
Revised: DeleteAndMergeLS ( F ) (cid:46) Merge or delete existing LS threads when necessary 10:

Original: Update
Revised: Update

Original: 11: if x is proposed by global search then UpdateGSModel ( S 0 , ( x , l, c ))
Revised: 11: if x is proposed by global search then UpdateGSModel ( S 0 , ( x , l, c ))

Original: For convenience, we use F to denote a collection of framework-level variables:
Revised: For convenience, we use F to denote a collection of framework-level variables:

Original: • F .
Revised: • F .

Original: S is the list of search threads maintained in our framework.
Revised: S is the list of search threads maintained in our framework.

Original: F . S contains at least one search thread and among them there is one and only one global search thread, i.e., S 0 , in F . S .
Revised: F . S contains at least one search thread and among them there is one and only one global search thread, i.e., S 0 , in F . S .

Original: • F .
Revised: • F .

Original: Priority is a priority dictionary, in which the keys are the search threads in F .
Revised: Priority is a priority dictionary, in which the keys are the search threads in F .

Original: S , and the values are the priority of the corresponding search threads.
Revised: S , and the values are the priority of the corresponding search threads.

Original: • Bookkeeping information of F : F .l ∗ is the best loss achieved among all the search threads and F .c is the total cost consumed in our framework.
Revised: • Bookkeeping information of F : F .l ∗ is the best loss achieved among all the search threads and F .c is the total cost consumed in our framework.

Original: • F .
Revised: • F .

Original: R is the ‘admissible’ region on the controlled dimensions of the current search, which is a hyperrectangle and can be written in the form of F .
Revised: R is the ‘admissible’ region on the controlled dimensions of the current search, which is a hyperrectangle and can be written in the form of F .

Original: R :=
Revised: R :=

Original: { [ F .x min i , F .x max i ] } i ∈ D with F .x min i and F .x max i denoting the minimum and maximum value along the i -th dimension in F .
Revised: { [ F .x min i , F .x max i ] } i ∈ D with F .x min i and F .x max i denoting the minimum and maximum value along the i -th dimension in F .

Original: R respectively.
Revised: R respectively.

Original: They are initially set as F .x min i
Revised: They are initially set as F .x min i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: max i
Revised: max i

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: all i ∈
Revised: all i ∈

Original: D
Revised: D

Original: The ‘admissible’ region gradually expands during the search: (1) it is expanded to cover all the points evaluated by all the search threads and all the points that are possible to be reached by the local search within one search step, as shown in line 7 and 8 of Algorithm 2; (2) it expands if a local search thread converges, as shown in line 3 of Algorithm 5 (included in Appendix A).
Revised: The ‘admissible’ region gradually expands during the search: (1) it is expanded to cover all the points evaluated by all the search threads and all the points that are possible to be reached by the local search within one search step, as shown in line 7 and 8 of Algorithm 2; (2) it expands if a local search thread converges, as shown in line 3 of Algorithm 7 (included in Appendix A).

Original: In the following, we explain the key steps in our algorithm.
Revised: In the following, we explain the key steps in our algorithm.

Original: Step 1: Search thread selector (line 3 of Alg 1).
Revised: Step 1: Search thread selector (line 3 of Alg 1).

Original: In addition to the primary search thread ˜ S ,
Revised: In addition to the primary search thread ˜ S ,

Original: SelectThread also outputs a backup search thread ˜ S bak which is guaranteed to be a local search thread.
Revised: SelectThread also outputs a backup search thread ˜ S bak which is guaranteed to be a local search thread.

Original: It is set to be none when there is no local search thread yet in F .
Revised: It is set to be none when there is no local search thread yet in F .

Original: S .
Revised: S .

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: None
Revised: None

Original: [Equation]
Revised: [Equation]

Original: The design of the priority metric follows the principle of optimism in the face of uncertainty from the multi-armed bandit problem to balance exploitation and exploration (Lattimore & Szepesv´ari, 2020).
Revised: The design of the priority metric follows the principle of optimism in the face of uncertainty from the multi-armed bandit problem to balance exploitation and exploration (Lattimore & Szepesv´ari, 2020).

Original: Speciﬁcally, linear extrapolation is performed adaptively and locally to calculate the improvement speed of each search thread.
Revised: Speciﬁcally, linear extrapolation is performed adaptively and locally to calculate the improvement speed of each search thread.

Original: The estimated future reward based on such a linear extrapolation provides a ﬁrst-order upper bound of the ground truth future reward assuming each search thread has a diminishing return, i.e., the speed of improvement decreases as more resource is spent.
Revised: The estimated future reward based on such a linear extrapolation provides a ﬁrst-order upper bound of the ground truth future reward assuming each search thread has a diminishing return, i.e., the speed of improvement decreases as more resource is spent.

Original: Formally, we introduce the following variables and functions for each search thread S ∈ F . S .
Revised: Formally, we introduce the following variables and functions for each search thread S ∈ F . S .

Original: • Bookkeeping information: S.l 1 st and S.l 2 nd are the best loss so far and second best loss before the best loss is achieved.
Revised: • Bookkeeping information: S.l 1 st and S.l 2 nd are the best loss so far and second best loss before the best loss is achieved.

Original: S.c 1 st and S.c 2 nd are the total cost taken when S.l 1 st and S.l 2 nd are achieved respectively.
Revised: S.c 1 st and S.c 2 nd are the total cost taken when S.l 1 st and S.l 2 nd are achieved respectively.

Original: S.c is the total cost spent in S .
Revised: S.c is the total cost spent in S .

Original: S. x 1 st is the best conﬁguration found so far.
Revised: S. x 1 st is the best conﬁguration found so far.

Original: • S.s is the performance improvement speed of S .
Revised: • S.s is the performance improvement speed of S .

Original: It is calculated as S.s = S.l 2 nd − S.l 1 st S.c − S.c 2 nd .
Revised: It is calculated as S.s = S.l 2 nd − S.l 1 st S.c − S.c 2 nd .

Original: This formula is only valid when there is at least one improvement.
Revised: This formula is only valid when there is at least one improvement.

Original: Otherwise, we do not have enough information to estimate the speed of improvement.
Revised: Otherwise, we do not have enough information to estimate the speed of improvement.

Original: We set the speed to the highest speed of all the search threads when S.l 2 nd =
Revised: We set the speed to the highest speed of all the search threads when S.l 2 nd =

Original: S.l 1 st .
Revised: S.l 1 st .

Original: It is due to an implicit assumption of diminishing return.
Revised: It is due to an implicit assumption of diminishing return.

Original: • S.x min and S.x max are the minimum and maximum value of the i -th dimension of all hyperparameters conﬁgurations evaluated in S respectively.
Revised: • S.x min and S.x max are the minimum and maximum value of the i -th dimension of all hyperparameters conﬁgurations evaluated in S respectively.

Original: • S. CostImp ( · ) is a function whose input is a target loss and output is the anticipated cost for S to generate a better loss than this target.
Revised: • S. CostImp ( · ) is a function whose input is a target loss and output is the anticipated cost for S to generate a better loss than this target.

Original: We use the following formula to compute it, which is intuitively using the cost for improvement in the past to estimate that in the future.
Revised: We use the following formula to compute it, which is intuitively using the cost for improvement in the past to estimate that in the future.

Original: [Equation]
Revised: [Equation]

Original: Our proposed priority metric is essentially the negative of the projected loss of S :
Revised: Our proposed priority metric is essentially the negative of the projected loss of S :

Original: [Equation]
Revised: [Equation]

Original: in which b = min(max S ∈F .
Revised: in which b = min(max S ∈F .

Original: S S. CostImp ( F .l ∗ ) , B − F .c ) .
Revised: S S. CostImp ( F .l ∗ ) , B − F .c ) .

Original: max S ∈F
Revised: max S ∈F

Original: [Equation]
Revised: [Equation]

Original: S. CostImp ( F .l ∗ ) can be considered as the resource needed for every S to have a better performance than the currently best performance F .l ∗ .
Revised: S. CostImp ( F .l ∗ ) can be considered as the resource needed for every S to have a better performance than the currently best performance F .l ∗ .

Original: Our priority metric estimates the loss of each search thread if such an amount of resource (restricted by the budget left B − F .c ) is given.
Revised: Our priority metric estimates the loss of each search thread if such an amount of resource (restricted by the budget left B − F .c ) is given.

Original: By considering both the search threads’ current performance and potential improvement, it provides a fair trade-off between exploiting the currently-best and exploring the potentially-better choices.
Revised: By considering both the search threads’ current performance and potential improvement, it provides a fair trade-off between exploiting the currently-best and exploring the potentially-better choices.

Original: x by adding Guassian noises on the corresponding dimensions of F . x 0 and for the rest of the dimensions sample uniformly at random from the search space P. X .
Revised: x by adding Guassian noises on the corresponding dimensions of F . x 0 and for the rest of the dimensions sample uniformly at random from the search space P. X .

Original: S S
Revised: S S

Original: i
Revised: i

Original: e., S is the global search thread) & x / ∈ F .
Revised: e., S is the global search thread) & x / ∈ F .

Original: R then x ← invalid
Revised: R then x ← invalid

Original: BookKeeping
Revised: BookKeeping

Original: S x c
Revised: S x c

Original: F ) and update speed S
Revised: F ) and update speed S

Original: i
Revised: i

Original: P
Revised: P

Original: D
Revised: D

Original: S i ← min { x i , S.x min i } , F .x min i ← min { S.x min i − L .
Revised: S i ← min { x i , S.x min i } , F .x min i ← min { S.x min i − L .

Original: ∆ , F .x min i } , 8: i ← max { x i , S.x max i } , F .x max i ← max { S.x max i + L .
Revised: ∆ , F .x min i } , 8: i ← max { x i , S.x max i } , F .x max i ← max { S.x max i + L .

Original: ∆ , F .x max i } 9: Outputs
Revised: ∆ , F .x max i } 9: Outputs

Original: Step 2: Conﬁg validator and evaluator (line 4-5 of Alg 1).
Revised: Step 2: Conﬁg validator and evaluator (line 4-5 of Alg 1).

Original: After a search thread (and a backup search thread) is selected, the next step is to propose the next conﬁguration to try with the chosen search thread(s).
Revised: After a search thread (and a backup search thread) is selected, the next step is to propose the next conﬁguration to try with the chosen search thread(s).

Original: Intuitively speaking, we consider generating the next conﬁguration to try primarily according to the selected search thread ˜ S whose priority is ranked the highest.
Revised: Intuitively speaking, we consider generating the next conﬁguration to try primarily according to the selected search thread ˜ S whose priority is ranked the highest.

Original: But we set a guard rail for the global search thread as it may propose an unnecessarily high-cost conﬁguration.
Revised: But we set a guard rail for the global search thread as it may propose an unnecessarily high-cost conﬁguration.

Original: We thus introduce a conﬁg validator to validate the conﬁgurations proposed by global search according to whether they are within the current admissible region of our framework F .
Revised: We thus introduce a conﬁg validator to validate the conﬁgurations proposed by global search according to whether they are within the current admissible region of our framework F .

Original: R (line 3 of Alg 2).
Revised: R (line 3 of Alg 2).

Original: A conﬁguration marked as ‘ invalid ’ means that it is considered to be prone to incur unnecessarily high cost and will not be evaluated at this round.
Revised: A conﬁguration marked as ‘ invalid ’ means that it is considered to be prone to incur unnecessarily high cost and will not be evaluated at this round.

Original: In this case, the selected backup search thread will be used to perform another round of SearchEvaluate1Step (line 5 of Alg 1) if it is a valid search thread (i.e., not none).
Revised: In this case, the selected backup search thread will be used to perform another round of SearchEvaluate1Step (line 5 of Alg 1) if it is a valid search thread (i.e., not none).

Original: In the case where the backup thread is none, we generate the new conﬁguration according to line 1 of Alg 2.
Revised: In the case where the backup thread is none, we generate the new conﬁguration according to line 1 of Alg 2.

Original: The conﬁg validator helps avoid potentially high-cost evaluation and thus avoid creating local search threads from high-cost points until necessary.
Revised: The conﬁg validator helps avoid potentially high-cost evaluation and thus avoid creating local search threads from high-cost points until necessary.

Original: It does not stick to local searches forever because the admissible region F .
Revised: It does not stick to local searches forever because the admissible region F .

Original: R gets expanded.
Revised: R gets expanded.

Original: Note that according to the deﬁnition of F .
Revised: Note that according to the deﬁnition of F .

Original: R , only the controlled dimensions of the hyperparameter conﬁgurations are subject to validation check.
Revised: R , only the controlled dimensions of the hyperparameter conﬁgurations are subject to validation check.

Original: If needed, a multi-ﬁdelity pruning strategy can be used in this conﬁg evaluator component.
Revised: If needed, a multi-ﬁdelity pruning strategy can be used in this conﬁg evaluator component.

Original: Multi-ﬁdelity pruning does not necessarily yield better performance.
Revised: Multi-ﬁdelity pruning does not necessarily yield better performance.

Original: So the adoption of multi-ﬁdelity pruning in BlendSearch is optional.
Revised: So the adoption of multi-ﬁdelity pruning in BlendSearch is optional.

Original: Step 3: Search thread creator, updater and cleaner (line 6-11 of Alg 1).
Revised: Step 3: Search thread creator, updater and cleaner (line 6-11 of Alg 1).

Original: If the newly proposed conﬁguration is proposed by global search and it is not marked as ‘invalid’, we consider creating a new local search thread using the proposed conﬁguration as a starting point.
Revised: If the newly proposed conﬁguration is proposed by global search and it is not marked as ‘invalid’, we consider creating a new local search thread using the proposed conﬁguration as a starting point.

Original: To make sure the newly created local search thread is relatively good, we ﬁrst check whether the proposed conﬁguration’s performance is better than at least half of the existing threads’ performance (speciﬁed in the
Revised: To make sure the newly created local search thread is relatively good, we ﬁrst check whether the proposed conﬁguration’s performance is better than at least half of the existing threads’ performance (speciﬁed in the

Original: CreateNewLSCondition ).
Revised: CreateNewLSCondition ).

Original: If so, a new local search thread will be initialized and added to the active search thread pool S .
Revised: If so, a new local search thread will be initialized and added to the active search thread pool S .

Original: In DeleteAndMergeLS , we check whether a local search thread has converged according to the convergence condition of the speciﬁc local search method.
Revised: In DeleteAndMergeLS , we check whether a local search thread has converged according to the convergence condition of the speciﬁc local search method.

Original: If it is, the search thread will be removed from S .
Revised: If it is, the search thread will be removed from S .

Original: In addition, we also go through all the local search threads to see whether the incumbent of a LS thread is reachable in one step by another LS thread with lower loss (ref.
Revised: In addition, we also go through all the local search threads to see whether the incumbent of a LS thread is reachable in one step by another LS thread with lower loss (ref.

Original: Appendix A).
Revised: Appendix A).

Original: If so, the former LS thread will be deleted.
Revised: If so, the former LS thread will be deleted.

Original: After a conﬁguration proposed by global search is evaluated, the observation tuple ( x , l, c ) is then used to update the model of the global search method through function UpdateGSModel .
Revised: After a conﬁguration proposed by global search is evaluated, the observation tuple ( x , l, c ) is then used to update the model of the global search method through function UpdateGSModel .

Original: For example, when the global search method is a Bayesian optimization method, the model is the surrogate model used.
Revised: For example, when the global search method is a Bayesian optimization method, the model is the surrogate model used.

Original: Due to page limit, detailed pseudocode for several of the straightforward functions mentioned in our framework are provided in Appendix A, including CreateNewLSCondition ,
Revised: Due to page limit, detailed pseudocode for several of the straightforward functions mentioned in our framework are provided in Appendix A, including CreateNewLSCondition ,

Original: DeleteAndMergeLS , InitializeSearchThread and BookKeeping .
Revised: DeleteAndMergeLS , InitializeSearchThread and BookKeeping .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate BlendSearch in tuning tabular machine learning libraries with an AutoML benchmark (Gijsbers et al., 2019), and in ﬁne-tuning NLP models for text data.
Revised: We evaluate BlendSearch in tuning tabular machine learning libraries with an AutoML benchmark (Gijsbers et al., 2019), and in ﬁne-tuning NLP models for text data.

Original: The AutoML benchmark consists of 39 tabular datasets that represent real-world data science classiﬁcation problems.
Revised: The AutoML benchmark consists of 39 tabular datasets that represent real-world data science classiﬁcation problems.

Original: It includes datasets of all sizes, of different problem domains and with various levels of difﬁculty.
Revised: It includes datasets of all sizes, of different problem domains and with various levels of difﬁculty.

Original: As each dataset has 10 cross-validation folds, all the results reported in this paper are averaged over the 10 folds.
Revised: As each dataset has 10 cross-validation folds, all the results reported in this paper are averaged over the 10 folds.

Original: With this benchmark, we are able to evaluate multiple HPO methods on a large number of datasets within a manageable computational budget, for tuning three machine learning libraries:
Revised: With this benchmark, we are able to evaluate multiple HPO methods on a large number of datasets within a manageable computational budget, for tuning three machine learning libraries:

Original: XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017) and DeepTables 2 .
Revised: XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017) and DeepTables 2 .

Original: The ﬁrst two are popular libraries based on gradient boosted trees, and the third is an easy-to-use deep learning toolkit which utilizes latest research ﬁndings for tabular data.
Revised: The ﬁrst two are popular libraries based on gradient boosted trees, and the third is an easy-to-use deep learning toolkit which utilizes latest research ﬁndings for tabular data.

Original: We chose them because gradient boosted trees and deep neural networks are the most frequent winners in data science competitions.
Revised: We chose them because gradient boosted trees and deep neural networks are the most frequent winners in data science competitions.

Original: We run experiments for XGBoost and LightGBM on the AutoML benchmark and report the results in Section 4.1.
Revised: We run experiments for XGBoost and LightGBM on the AutoML benchmark and report the results in Section 4.1.

Original: As a real application, we report an NLP model ﬁne-tuning task for a production use case in Section 4.2.
Revised: As a real application, we report an NLP model ﬁne-tuning task for a production use case in Section 4.2.

Original: Finally, in Section 4.3 we perform ablation study to investigate the effectiveness of several important components of our framework.
Revised: Finally, in Section 4.3 we perform ablation study to investigate the effectiveness of several important components of our framework.

Original: Due to page limit, we include the results for tuning DeepTables in
Revised: Due to page limit, we include the results for tuning DeepTables in

Original: Appendix B.
Revised: Appendix B.

Original: We include the following baselines in our experiments.
Revised: We include the following baselines in our experiments.

Original: • BO (Akiba et al., 2019) – the Bayesian optimization baseline.
Revised: • BO (Akiba et al., 2019) – the Bayesian optimization baseline.

Original: We choose a modern HPO library Optuna and use the TPE sampler because of its ﬂexibility in handling mixed continuous and discrete space and good peformance reported in existing work (Falkner et al., 2018).
Revised: We choose a modern HPO library Optuna and use the TPE sampler because of its ﬂexibility in handling mixed continuous and discrete space and good peformance reported in existing work (Falkner et al., 2018).

Original: • LS (Wu et al., 2021) – the recent baseline of using local search with random restart, based on FLOW 2 .
Revised: • LS (Wu et al., 2021) – the recent baseline of using local search with random restart, based on FLOW 2 .

Original: It is proved to be able to control cost effectively and outperform BO methods for numerical cost-related hyperparameter search.
Revised: It is proved to be able to control cost effectively and outperform BO methods for numerical cost-related hyperparameter search.

Original: • BOwLS (Gao et al., 2020) – the baseline of an existing approach of combining local search with BO, i.e., using BO to propose start points for local search.
Revised: • BOwLS (Gao et al., 2020) – the baseline of an existing approach of combining local search with BO, i.e., using BO to propose start points for local search.

Original: • ASHA (Li et al., 2020), i.e., asynchronous successive halving – a state-of-the-art HPO method that uses multi-ﬁdelity optimization and supports parallel tuning.
Revised: • ASHA (Li et al., 2020), i.e., asynchronous successive halving – a state-of-the-art HPO method that uses multi-ﬁdelity optimization and supports parallel tuning.

Original: Initialization setting.
Revised: Initialization setting.

Original: For the local search method used in our framework, a low-cost initialization is needed to realize its unique advantages in controlling the cost.
Revised: For the local search method used in our framework, a low-cost initialization is needed to realize its unique advantages in controlling the cost.

Original: It is implemented via setting a lowcost initial value for each of the controlled dimensions.
Revised: It is implemented via setting a lowcost initial value for each of the controlled dimensions.

Original: For example, among the hyperparameters tuned in LightGBM (shown in Table 2 of Appendix B), three hyperparameters, including ‘tree num’, ’leaf num’ and ‘min child weight’ have initial values corresponding to the min or max values in their range, depending on whether they have a positive or negative correlation with the evaluation cost.
Revised: For example, among the hyperparameters tuned in LightGBM (shown in Table 2 of Appendix B), three hyperparameters, including ‘tree num’, ’leaf num’ and ‘min child weight’ have initial values corresponding to the min or max values in their range, depending on whether they have a positive or negative correlation with the evaluation cost.

Original: It does not require the loss of the initial conﬁguration to be low.
Revised: It does not require the loss of the initial conﬁguration to be low.

Original: Only one single low-cost initial value for each controlled dimension needs to be speciﬁed as input.
Revised: Only one single low-cost initial value for each controlled dimension needs to be speciﬁed as input.

Original: To ensure a fair evaluation, we use the same low-cost initial point (if controlled dimensions exist) as the starting point of all the baselines.
Revised: To ensure a fair evaluation, we use the same low-cost initial point (if controlled dimensions exist) as the starting point of all the baselines.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. T UNING XGB OOST AND L IGHT GBM
Revised: 4. T UNING XGB OOST AND L IGHT GBM

Original: We tune a set of 9-dimensional hyperparameters (all numerical) in LightGBM and 11-dimensional hyperparameters (9 numerical and 2 categorical) in XGBoost.
Revised: We tune a set of 9-dimensional hyperparameters (all numerical) in LightGBM and 11-dimensional hyperparameters (9 numerical and 2 categorical) in XGBoost.

Original: A detailed description of the search space can be found in Appendix B.
Revised: A detailed description of the search space can be found in Appendix B.

Original: In this section, we omit the multi-ﬁdelity baseline method, i.e.,
Revised: In this section, we omit the multi-ﬁdelity baseline method, i.e.,

Original: ASHA, because we do not ﬁnd a good ‘ﬁdelity’ dimension that works well in tuning LightGBM and XGBoost (ref.
Revised: ASHA, because we do not ﬁnd a good ‘ﬁdelity’ dimension that works well in tuning LightGBM and XGBoost (ref.

Original: additional results in Appendix B).
Revised: additional results in Appendix B).

Original: We perform the evaluation on 37 out of the 39 datasets from the AutoML benchmark (‘Robert’ and ‘Dionis’ are excluded due to out-ofmemory error and extremely long training time).
Revised: We perform the evaluation on 37 out of the 39 datasets from the AutoML benchmark (‘Robert’ and ‘Dionis’ are excluded due to out-ofmemory error and extremely long training time).

Original: The input budget B (in terms of CPU time) is set to be 4 hours for the 3 largest datasets among the 37 datasets, and 1 hour for the rest.
Revised: The input budget B (in terms of CPU time) is set to be 4 hours for the 3 largest datasets among the 37 datasets, and 1 hour for the rest.

Original: From the performance curves shown in Figure 3, we observe that BO tends to perform well on small datasets, e.g., ‘credit’ in Figure 3(a), where the 1h budget is sufﬁcient.
Revised: From the performance curves shown in Figure 3, we observe that BO tends to perform well on small datasets, e.g., ‘credit’ in Figure 3(a), where the 1h budget is sufﬁcient.

Original: Under the same budget, it may perform badly on large datasets, e.g., ‘volkert’ in Figure 3(c), as it may try conﬁgurations which consume a very large portion of the budget at the early stage of the search.
Revised: Under the same budget, it may perform badly on large datasets, e.g., ‘volkert’ in Figure 3(c), as it may try conﬁgurations which consume a very large portion of the budget at the early stage of the search.

Original: On the medium size dataset, e.g., ‘KDDCup09’ in Figure 3(b), the local search method is more efﬁcient than BO in the early stage but is outperformed in the later stage.
Revised: On the medium size dataset, e.g., ‘KDDCup09’ in Figure 3(b), the local search method is more efﬁcient than BO in the early stage but is outperformed in the later stage.

Original: BlendSearch performs similarly to the better one between
Revised: BlendSearch performs similarly to the better one between

Original: LS and BO in the early stage, and surpasses both of them in the later stage.
Revised: LS and BO in the early stage, and surpasses both of them in the later stage.

Original: We also observe that
Revised: We also observe that

Original: BOwLS performs similarly with LS (sometimes worse).
Revised: BOwLS performs similarly with LS (sometimes worse).

Original: This is because BOwLS needs to wait until a local search converges before proposing a new one.
Revised: This is because BOwLS needs to wait until a local search converges before proposing a new one.

Original: The aggregated result over all the test cases in
Revised: The aggregated result over all the test cases in

Original: Figure 4(a) & (b) is consistent with these observations.
Revised: Figure 4(a) & (b) is consistent with these observations.

Original: The interplay between local and global search in BlendSearch .
Revised: The interplay between local and global search in BlendSearch .

Original: We investigated the dynamics of local and global search thread selection in BlendSearch .
Revised: We investigated the dynamics of local and global search thread selection in BlendSearch .

Original: As a case study, we show the results in tuning XGBoost on two datasets in Figure 1(b) and Figure 5(a).
Revised: As a case study, we show the results in tuning XGBoost on two datasets in Figure 1(b) and Figure 5(a).

Original: The result in Figure 1(b) shows
Revised: The result in Figure 1(b) shows

Original: [Figure]
Revised: [Figure]

Original: In this ﬁgure (and Figure 1(b), which shows the result for dataset Volkert ), each marker corresponds to one time of search thread selection in BlendSearch .
Revised: In this ﬁgure (and Figure 1(b), which shows the result for dataset Volkert ), each marker corresponds to one time of search thread selection in BlendSearch .

Original: The diamonds correspond to the global search thread, and the solid circles correspond to the local search threads (different colors for different threads)
Revised: The diamonds correspond to the global search thread, and the solid circles correspond to the local search threads (different colors for different threads)

Original: [Figure]
Revised: [Figure]

Original: that the global search indeed plays a role after 1000s which contributes to BlendSearch ’s better performance comparing to LS.
Revised: that the global search indeed plays a role after 1000s which contributes to BlendSearch ’s better performance comparing to LS.

Original: The result in Figure 5(a) shows that although the ﬁrst point suggested by the global search (at around 200s) does not yield signiﬁcantly better performance immediately, the induced new local search thread (the thread in green circle) lead to a signiﬁcantly better performance soon.
Revised: The result in Figure 5(a) shows that although the ﬁrst point suggested by the global search (at around 200s) does not yield signiﬁcantly better performance immediately, the induced new local search thread (the thread in green circle) lead to a signiﬁcantly better performance soon.

Original: These two ﬁgures together indicate that global search is not only responsible for directly ﬁnding better conﬁgurations, but also for creating local search threads that can achieve yet better performance.
Revised: These two ﬁgures together indicate that global search is not only responsible for directly ﬁnding better conﬁgurations, but also for creating local search threads that can achieve yet better performance.

Original: We show two statistics about the overall interplay between global and local search in BlendSearch on all the datasets evaluated for tuning XGBoost in Figure 6.
Revised: We show two statistics about the overall interplay between global and local search in BlendSearch on all the datasets evaluated for tuning XGBoost in Figure 6.

Original: We provide additional experiments for tuning XGBoost and LightGBM in Appendix B, including a comparison with ASHA under different settings of ﬁdelity, and an empirical study about the effect of low-cost initialization.
Revised: We provide additional experiments for tuning XGBoost and LightGBM in Appendix B, including a comparison with ASHA under different settings of ﬁdelity, and an empirical study about the effect of low-cost initialization.

Original: Takeaway.
Revised: Takeaway.

Original: (1) BlendSearch is able to overcome the limitations of BO and LS and at the same time, inherit their advantages.
Revised: (1) BlendSearch is able to overcome the limitations of BO and LS and at the same time, inherit their advantages.

Original: By blending BO and LS, BlendSearch is able to outperform both on this large collection of datasets over time.
Revised: By blending BO and LS, BlendSearch is able to outperform both on this large collection of datasets over time.

Original: The interplay between global and local search indeed contribute to BlendSearch ’s good performance.
Revised: The interplay between global and local search indeed contribute to BlendSearch ’s good performance.

Original: 4. T RANSFORMER - BASED NLP MODEL FINE - TUNING
Revised: 4. T RANSFORMER - BASED NLP MODEL FINE - TUNING

Original: This section presents an application of economical hyperparameter optimization to an NLP model ﬁne tuning task used in a large software company.
Revised: This section presents an application of economical hyperparameter optimization to an NLP model ﬁne tuning task used in a large software company.

Original: It starts from a large transformer-based model
Revised: It starts from a large transformer-based model

Original: Turing-NLRv2 with 24 transformer layers, each with 16 attention heads and hidden-dimensiontotaling 340M parameters.
Revised: Turing-NLRv2 with 24 transformer layers, each with 16 attention heads and hidden-dimensiontotaling 340M parameters.

Original: It is pretrained on English Wikipedia (2,500M words) and BookCorpus (800M words), and uses byte-pair encoding 3 .
Revised: It is pretrained on English Wikipedia (2,500M words) and BookCorpus (800M words), and uses byte-pair encoding 3 .

Original: This pre-trained model is then ﬁne-tuned (Dai &
Revised: This pre-trained model is then ﬁne-tuned (Dai &

Original: Le, 2015) for use in multiple production scenarios, including sequence classiﬁcation, named entity recognition and question answering.
Revised: Le, 2015) for use in multiple production scenarios, including sequence classiﬁcation, named entity recognition and question answering.

Original: The ﬁne-tuning procedure is performed for a dozen separate tasks, and is repeated on a regular cadence, typically every few weeks.
Revised: The ﬁne-tuning procedure is performed for a dozen separate tasks, and is repeated on a regular cadence, typically every few weeks.

Original: We focus our experiment on a single sequence classiﬁcation ﬁne-tuning task where the objective is to label a document (consisting of one or more sentences) with one of ﬁve possible classes.
Revised: We focus our experiment on a single sequence classiﬁcation ﬁne-tuning task where the objective is to label a document (consisting of one or more sentences) with one of ﬁve possible classes.

Original: For ﬁne-tuning this model we introduce a classiﬁcation layer with 1024 × 5 = 5120 additional weight parameters, randomly initialized.
Revised: For ﬁne-tuning this model we introduce a classiﬁcation layer with 1024 × 5 = 5120 additional weight parameters, randomly initialized.

Original: The dataset used for training consists of 52K labeled examples, which we split 80/20 for training/validation.
Revised: The dataset used for training consists of 52K labeled examples, which we split 80/20 for training/validation.

Original: The objective to maximize is the f1-score obtained on the validation set of 10.4K labeled documents.
Revised: The objective to maximize is the f1-score obtained on the validation set of 10.4K labeled documents.

Original: Selecting hyperparameters for ﬁne-tuning this model has been a manual process that typically takes a data scientist a few days.
Revised: Selecting hyperparameters for ﬁne-tuning this model has been a manual process that typically takes a data scientist a few days.

Original: [Table]
Revised: [Table]

Original: In our experiment we use a 6-dimensional search space (4 numerical and 2 categorical).
Revised: In our experiment we use a 6-dimensional search space (4 numerical and 2 categorical).

Original: A detailed description of all the hyperparameters tuned, including their ranges, can be found in Table 5 in
Revised: A detailed description of all the hyperparameters tuned, including their ranges, can be found in Table 5 in

Original: Appendix B. We compare to ASHA and let it use 16 VMs with 4 NVIDIA Tesla V100 GPUs on each VM.
Revised: Appendix B. We compare to ASHA and let it use 16 VMs with 4 NVIDIA Tesla V100 GPUs on each VM.

Original: We run ASHA with 16 concurrent jobs for 6 hours in wallclock time.
Revised: We run ASHA with 16 concurrent jobs for 6 hours in wallclock time.

Original: That amounts to 4 × 16 × 6 = 384 GPU hours of hardware cost.
Revised: That amounts to 4 × 16 × 6 = 384 GPU hours of hardware cost.

Original: We run BlendSearch on a single VM of the same conﬁguration for 3 hours, which uses 12 GPU hours in total.
Revised: We run BlendSearch on a single VM of the same conﬁguration for 3 hours, which uses 12 GPU hours in total.

Original: For comparison, we include ASHAusing the same single VM.
Revised: For comparison, we include ASHAusing the same single VM.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: Takeaway.
Revised: Takeaway.

Original: Not only BlendSearch ﬁnds a more accurate model than both ASHA and manual effort, it does so faster and consumes only 3% of the resources of ASHA-16.
Revised: Not only BlendSearch ﬁnds a more accurate model than both ASHA and manual effort, it does so faster and consumes only 3% of the resources of ASHA-16.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: To investigate the effectiveness of several important modules of our framework (colored in orange in Figure 2), we perform an ablation study in tuning XGBoost on a random subset of the datasets (one third of the datasets mentioned in Section 4.1).
Revised: To investigate the effectiveness of several important modules of our framework (colored in orange in Figure 2), we perform an ablation study in tuning XGBoost on a random subset of the datasets (one third of the datasets mentioned in Section 4.1).

Original: We show the aggregated rank of different variants of our method in Figure 4(c).
Revised: We show the aggregated rank of different variants of our method in Figure 4(c).

Original: Speciﬁcally, we study the following three modules of the framework.
Revised: Speciﬁcally, we study the following three modules of the framework.

Original: (1) Priority metric.
Revised: (1) Priority metric.

Original: ‘BS-Priority:RoundRobin’ uses the round-robin policy in the search thread selector.
Revised: ‘BS-Priority:RoundRobin’ uses the round-robin policy in the search thread selector.

Original: (2) Conﬁg validator.
Revised: (2) Conﬁg validator.

Original: ‘BS-w/o-ConﬁgValidator’ skips the validity check of the proposed conﬁguration.
Revised: ‘BS-w/o-ConﬁgValidator’ skips the validity check of the proposed conﬁguration.

Original: (3) Create new thread condition.
Revised: (3) Create new thread condition.

Original: In ‘BS-CreateCond:Always’ and ‘BS-CreatCond:BestOrFirst’, the following two conditions are used as the condition for creating new local search threads respectively: always create, and create a new thread only when the loss is better than all the existing search threads’ loss or there is no local search thread yet.
Revised: In ‘BS-CreateCond:Always’ and ‘BS-CreatCond:BestOrFirst’, the following two conditions are used as the condition for creating new local search threads respectively: always create, and create a new thread only when the loss is better than all the existing search threads’ loss or there is no local search thread yet.

Original: From this ablation study, we have the following observations: (1) Making round-robin selection is worse than doing selection using our designed priority metric.
Revised: From this ablation study, we have the following observations: (1) Making round-robin selection is worse than doing selection using our designed priority metric.

Original: Round-robin’s relative performance becomes worse in the later stage of the search because it cannot avoid bad-performing search threads.
Revised: Round-robin’s relative performance becomes worse in the later stage of the search because it cannot avoid bad-performing search threads.

Original: (2) The conﬁg validator is also vital in our framework.
Revised: (2) The conﬁg validator is also vital in our framework.

Original: (3) Overall, the conditions for creating new threads has a smaller impact on our method comparing to the other designs studied.
Revised: (3) Overall, the conditions for creating new threads has a smaller impact on our method comparing to the other designs studied.

Original: ‘Better than half’ condition used by default tends to perform the best.
Revised: ‘Better than half’ condition used by default tends to perform the best.

Original:  E XTENSION AND F UTURE WORK
Revised:  E XTENSION AND F UTURE WORK

Original: In the low-resource scenario which is targeted by this paper, each single trial is not resource-saturated if we spend all resources in it.
Revised: In the low-resource scenario which is targeted by this paper, each single trial is not resource-saturated if we spend all resources in it.

Original: So we do not recommend parallel trials in this low-resource scenario.
Revised: So we do not recommend parallel trials in this low-resource scenario.

Original: In the case where more resources than the maximum resource each single trial can consume are available, our framework can be extended by running the trials from different search threads on multiple workers.
Revised: In the case where more resources than the maximum resource each single trial can consume are available, our framework can be extended by running the trials from different search threads on multiple workers.

Original: For example, if there are additional workers available, we can keep invoking the search thread selector (but skip the local search threads that have O ( d ) trials running).
Revised: For example, if there are additional workers available, we can keep invoking the search thread selector (but skip the local search threads that have O ( d ) trials running).

Original: Our design of having multiple independent local search threads naturally allows efﬁcient asynchronous parallel trials.
Revised: Our design of having multiple independent local search threads naturally allows efﬁcient asynchronous parallel trials.

Original: The design of utilizing existing global optimization methods allows existing easyto-parallelize global optimization (such as random search or batch versions of BO) to be plugged in.
Revised: The design of utilizing existing global optimization methods allows existing easyto-parallelize global optimization (such as random search or batch versions of BO) to be plugged in.

Original: The prioritization of search threads is still useful as long as the maximal concurrent number of trials divided by the number of search threads is smaller than O ( d ) .
Revised: The prioritization of search threads is still useful as long as the maximal concurrent number of trials divided by the number of search threads is smaller than O ( d ) .

Original: Since our method can be used together with multi-ﬁdelity pruning methods, it can naturally inherit the asynchronous resource scheduling when used in the parallel setting.
Revised: Since our method can be used together with multi-ﬁdelity pruning methods, it can naturally inherit the asynchronous resource scheduling when used in the parallel setting.

Original: Parallelization is now supported in the latest version of
Revised: Parallelization is now supported in the latest version of

Original: BlendSearch ’s implementation.
Revised: BlendSearch ’s implementation.

Original: In this work, we show the effectiveness of BlendSearch through extensive empirical study.
Revised: In this work, we show the effectiveness of BlendSearch through extensive empirical study.

Original: As future work, it is worth studying the theoretical properties of BlendSearch , including theoretical guarantees about its convergence rate and total resource consumption. 
Revised: As future work, it is worth studying the theoretical properties of BlendSearch , including theoretical guarantees about its convergence rate and total resource consumption. A M ORE DETAILS ABOUT B LEND S EARCH We provide detailed pseudocode of the sub-algorithms used in our framework. Choice of L .

Revised: ∆ used in constructing the ‘admissible’ region F .

Original: R 
Revised: R .

Revised: We normalize each numeric hyperparameter into [0,1].

Revised: The setting of L .

Revised: ∆ is decided by the local search method.

Revised: For the local search method we chose, L .

Revised: ∆ is a constant corresponding to the initial stepsize (ﬁxed as 0.1).

Revised: Deﬁnition of ReachableInOneStep in Alg 7 .

Revised: We deﬁne local search thread S 1 to be reachable by S 2 if the distance between their incumbents is no larger than the maximal distance between the next proposal of S 2 and the incumbent of S 2 .

Revised: In the local search method we used, the incumbent is the currently best conﬁg, and the maximal distance is equal to the stepsize.

Revised: B M ORE DETAILS ABOUT EXPERIMENTS AND ADDITIONAL RESULTS B. E XPERIMENT SETUP

Revised: Settings of BO and LS.

Revised: For BO, we use implementation from Optuna 2.0.0 ( https://optuna. readthedocs.io/en/stable/index.html ) with default settings for TPE sampler.

Revised: For

Revised: LS, we follow the implementation guidelines from Wu et al.

Revised: After a local search thread is created from a particular starting point, we ﬁx the categorical dimensions and only search for numerical dimensions in that local search thread.

Revised: A local search thread S is considered to have converged (corresponding to S. converged () in Algorithm 7) once the stepsize of the local search thread is smaller than a lower bound introduced by Wu et al.

Revised: Experiments in tuning XGBoost and LightGBM.

Revised: The XGBoost and LightGBM experiments are performed in a server with Intel Xeon E5-2690 v4 2.6GHz, and 256GB RAM.

Revised: A full list of hyperparameters tuned and their ranges can be found in Table 3 and Table 2.

Revised: The search space for numerical hyperparameters aligns with the search space used in (Wu et al., 2021).

Revised: On the same fold, the same random seed is used for LS, BO and BS.

Revised: Experiments on different folds use different random seeds.

Revised: Experiments in NLP model ﬁne tuning.

Revised: For ASHA, we set min and max epochs as 1 and 16, and reduction factor 4.

Revised: Inputs: l, F .

Revised: Outputs: |F .

Revised: S

Revised: |

Revised: = 1 or l ≤ Median ( { S.l 1 st } S ∈F .

Revised: S \ S 0 )

Revised: Inputs: S, F 1: if S. converged () then 2:

Revised: F .

Revised: S ← F .

Revised: S \ S , 3:

Revised: F .

Revised: R .x min i ← F .

Revised: R .x min i − L .

Revised: ∆ , and F .

Revised: R .x max

Revised: i ← F .

Revised: R .x max i + L .

Revised: ∆ , ∀ i ∈ D (cid:48) 4: else 5: for ∀ S (cid:48) ∈ F .

Revised: S \ S do 6: if S ∈ S (cid:48) .

Revised: ReachableInOneStep () & S (cid:48) .l

Revised: <

Revised: S.l then 7:

Revised: F .

Revised: S ← F .

Revised: S \ S 8: break 9: else if S (cid:48)

Revised: ∈ S. ReachableInOneStep () & S.l < S (cid:48) .l

Revised: then F .

Revised: S ← F .

Revised: S \ S (cid:48)

Revised: Result aggregation details.

Revised: Aggregated rank in Figure 4(a)&(c) and 12(c) is calculated as follows: (1) per dataset per fold, the method is ranked based on the loss on validation set at each second (x-axis), starting from when there is at least one ﬁnished conﬁg evaluation in any method; (2) the rank is then averaged across datasets per fold; (3) we ﬁnally compute the average rank (line) and conﬁdence interval (shaded area) across 10 folds.

Revised: Scaled loss in Figure 4(b) is calculated similarly.

Revised: Per dataset per fold, min-max scaling is applied on each method using the maximum and minimum loss along the whole performance curve across all methods.

Revised: B. A DDITIONAL EXPERIMENTAL RESULTS ON L IGHT GBM AND XGB OOST

Revised: More performance curves on LightGBM and XGBoost.

Revised: The performance curves for tuning

Revised: LightGBM on 3 representative datasets with an 1h budget are shown in Figure 7.

Revised: We observe that the performance of LS is quite good (comparing to BO), especially on large datasets.

Revised: This result is consistent with the results reported in (Wu et al., 2021), where all the hyperparameters for tuning are numerical.

Revised: In our experiment of XGBoost tuning, we include categorical hyperparameters.

Revised: LS performs worse in this case because the introduction of categorical hyperparameters ampliﬁes the local search method’s limitation of being trapped in local optima.

Revised: The observations about BlendSearch for LightGBM are similar to XGBoost tuning.

Revised: The performance curves on the three large datasets with a 4h budget are shown in Figure 8 and 9, where similar conclusions can be drawn.

Revised: Multi-ﬁdelity.

Revised: We compare BO and BlendSearch with the multi-ﬁdelity baseline ASHA for tuning LightGBM and XGBoost in Figure 10.

Revised: In this experiment, we tried two choices of ﬁdelity dimensions with ASHA, including number of iterations and sample size (the sample size begins with 10K, so small datasets are excluded) respectively.

Revised: The results show that the multi-ﬁdelity baseline overall perform no better than BO and are signiﬁcantly worse than BlendSearch .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 0.2, 0.4] random batch size categorical { 16, 32 } epochs int [1, 16] learning rate scheduler categorical { Warmup linear decay polynomial,

Revised: Warmup linear, Warmup linear decay exponential } random

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Ablation study on the low-cost initialization.

Revised: In this work, we use low-cost initialization for the controlled dimensions of the hyperparameters.

Revised: Although such information is fairly easy to obtain, we investigate our method’s robustness when no controlled dimension is provided.

Revised: We test

Revised: BlendSearch in a controlled dimension agnostic setting: there are still hyperparameters with heterogeneous cost, but the controlled dimensions and a low-cost initial point is not speciﬁed as input.

Revised: In such scenarios

Revised: [Figure]

Revised: BlendSearch will use random initialization and the conﬁg validator always returns ‘yes’.

Revised: We compare BlendSearch with local search and BO under such a setting using the same random initial point.

Revised: In Figure 11 we report the results including the aggregated rank and scaled loss on LightGBM across half of the datasets mentioned in Section 4.1 in Figure 11(a) & (b)

Revised: [Figure]

Revised: The results show that even if BlendSearch is agnostic to the controlled dimensions and a random initialization is used, it is still able to outperform both the local search method and BO.

Revised: B. T UNING D EEP T ABLES .

Revised: In this experiment, we tune 9-dimensional hyperparameters (5 numerical and 4 categorical as detailed in Table 4) in DeepTables.

Revised: Since the training of deep neural networks are more time-consuming than that of XGBoost, we run experiments for DeepTables on the datasets where they are worse than the best known performance in the benchmark, including ‘shuttle’,‘cnae’,‘mfeat’,‘vehicle’,‘phoneme’, ‘kc1’ .

Revised: All experiments for DeepTables are performed in a server with the same CPU, 110GB RAM, and one Tesla P100 GPU.

Revised: A full list of hyperparameters tuned and their ranges can be found in Table 4.

Revised: Recall that we mentioned multi-ﬁdelity pruning strategies could be incorporated into

Revised: BlendSearch in the conﬁg evaluator component.

Revised: In this experiment, we are particularly interested showing the performance of BlendSearch when combined with multi-ﬁdelity methods.

Revised: To this end, we include the three state-of-the-art multi-ﬁdelity methods, including BOHB (Falkner et al., 2018), ASHA (Li et al., 2020), and asynchronous HyperBand (Li et al., 2017; 2020) which are shown efﬁcient for tuning deep neural networks and the BlendSearch based on each of them.

Revised: We use the following libraries for baselines:

Revised: For BOHB, we use HpBandSter 0.7.

Revised: ( https://github.com/automl/HpBandSter ).

Revised: For ASHA and asynchronous HyperBand, we use implementations from Optuna 2.0.0.

Revised: In all the methods compared, including both existing methods and variants of BlendSearch , the number of training epochs is used as the ﬁdelity dimension, with maximum epochs set to be 1024, reduction factor set to be 3, and minimum epochs 4.

Revised: For ASHA, we set the minimum early stopping rate to be 4 (we adopted this setting as it yields better performance comparing to the default setting, i.e., 0).

Revised: The number of training epochs is used as the ﬁdelity dimension.

Revised: BlendSearch incorporates existing multi-ﬁdelity methods in the following way: Each conﬁg, either proposed by global search or local search, uses the same schedule to increase the ﬁdelity and check its pruning condition.

Revised: For example, when ASHA (Li et al., 2020), i.e., asynchronous successive halving, with a reduction factor of η , is used as the pruning strategy, after each conﬁg is evaluated by a certain ﬁdelity, it is compared with other conﬁgs already evaluated by the same ﬁdelity.

Revised: The conﬁg will be pruned if its loss is ranked in the worst 1 /η .

Revised: Otherwise, the ﬁdelity is multiplied by η .

Revised: In addition to the original pruning conditions speciﬁed by the multi-ﬁdelity method, a conﬁguration will also be pruned at a particular ﬁdelity level where no pruning is performed yet, and the conﬁguration does not yield superior performance (comparing to the currently-best performance) when evaluated at that ﬁdelity level.

Revised: We present the performance of all compared methods for tuning DeepTables in Figure 12.

Revised: Figure 12(a) shows the learning curves on dataset cane with budget 1h.

Revised: Figure 12(b) and (c) show the aggregated rank and loss on all the 6 datasets within budget 1h.

Revised: The performance of multi-ﬁdelity methods are signiﬁcantly improved when used in our BlendSearch framework.