Original: M ODEL - BASED A SYNCHRONOUS H YPERPARAMETER AND N EURAL A RCHITECTURE S EARCH
Revised: M ODEL - BASED A SYNCHRONOUS H YPERPARAMETER AND N EURAL A RCHITECTURE S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce an asynchronous multi-ﬁdelity method for model-based hyperparameter and neural architecture search that combines the strengths of asynchronous Successive Halving and Gaussian process-based Bayesian optimization.
Revised: We introduce an asynchronous multi-ﬁdelity method for model-based hyperparameter and neural architecture search that combines the strengths of asynchronous Successive Halving and Gaussian process-based Bayesian optimization.

Original: Our method relies on a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations.
Revised: Our method relies on a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations.

Original: We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classiﬁcation and language modelling and report substantial speed-ups over current state-of-the-art methods.
Revised: We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classiﬁcation and language modelling and report substantial speed-ups over current state-of-the-art methods.

Original: Our new method, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication
Revised: Our new method, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of hyperparameter and neural architecture search (HNAS) is to automate the process of ﬁnding the right architecture or hyperparameters x (cid:63) ∈ arg min x ∈X f ( x ) of a deep neural network by minimizing the validation loss f ( x ) , observed through noise: y i =
Revised: The goal of hyperparameter and neural architecture search (HNAS) is to automate the process of ﬁnding the right architecture or hyperparameters x (cid:63) ∈ arg min x ∈X f ( x ) of a deep neural network by minimizing the validation loss f ( x ) , observed through noise: y i =

Original: f ( x i ) + (cid:15) i , (cid:15) i ∼ N (0 , σ 2 ) , i = 1 , . . . , n .
Revised: f ( x i ) + (cid:15) i , (cid:15) i ∼ N (0 , σ 2 ) , i = 1 , . . . , n .

Original: Bayesian optimization (BO) is an effective model-based approach for solving expensive black-box optimization problems (Jones et al., 1998; Shahriari et al., 2016).
Revised: Bayesian optimization (BO) is an effective model-based approach for solving expensive black-box optimization problems (Jones et al., 1998; Shahriari et al., 2016).

Original: It constructs a probabilistic surrogate model of the loss function p ( f | D ) based on previous evaluations D
Revised: It constructs a probabilistic surrogate model of the loss function p ( f | D ) based on previous evaluations D

Original: = { ( x i , y i ) } ni
Revised: = { ( x i , y i ) } ni

Original: =1 .
Revised: =1 .

Original: Searching for the global minimum of f is then driven by trading off exploration in regions of uncertainty and exploitation in regions where the global optimum is likely to reside.
Revised: Searching for the global minimum of f is then driven by trading off exploration in regions of uncertainty and exploitation in regions where the global optimum is likely to reside.

Original: However, for HNAS problems, standard BO needs to be augmented in order to remain competitive.
Revised: However, for HNAS problems, standard BO needs to be augmented in order to remain competitive.

Original: For example, training runs for unpromising conﬁgurations x can be stopped early, but serve as low-ﬁdelity approximations of f ( x ) (Swersky et al., 2014; Domhan et al., 2015; Klein et al., 2017b).
Revised: For example, training runs for unpromising conﬁgurations x can be stopped early, but serve as low-ﬁdelity approximations of f ( x ) (Swersky et al., 2014; Domhan et al., 2015; Klein et al., 2017b).

Original: Further, evaluations of f can be executed in parallel to reduce the wall-clock time required for ﬁnding a good solution.
Revised: Further, evaluations of f can be executed in parallel to reduce the wall-clock time required for ﬁnding a good solution.

Original: Several methods for multi-ﬁdelity and/or distributed BO have been proposed (Kandasamy et al., 2017; 2016;
Revised: Several methods for multi-ﬁdelity and/or distributed BO have been proposed (Kandasamy et al., 2017; 2016;

Original: Takeno et al., 2020), but they rely on rather complicated approximations to either select the ﬁdelity level or to compute an information theoretic acquisition function to determine the next candidate.
Revised: Takeno et al., 2020), but they rely on rather complicated approximations to either select the ﬁdelity level or to compute an information theoretic acquisition function to determine the next candidate.

Original: In this work we aim to adopt the desiderata of Falkner et al.
Revised: In this work we aim to adopt the desiderata of Falkner et al.

Original: (2018), namely, that of simplicity, which often leads to more robust methods in practice.
Revised: (2018), namely, that of simplicity, which often leads to more robust methods in practice.

Original: A simple, easily parallelizable multi-ﬁdelity scheduling algorithm is successive halving (SH)
Revised: A simple, easily parallelizable multi-ﬁdelity scheduling algorithm is successive halving (SH)

Original: (Karnin et al., 2013; Jamieson & Talwalkar, 2016), which iteratively eliminates poorly performing neural networks over time.
Revised: (Karnin et al., 2013; Jamieson & Talwalkar, 2016), which iteratively eliminates poorly performing neural networks over time.

Original: Hyperband (Li et al., 2017) iterates over multiple rounds of SH with varying ratios between the number of conﬁgurations and the minimum amount of resources spent per conﬁguration.
Revised: Hyperband (Li et al., 2017) iterates over multiple rounds of SH with varying ratios between the number of conﬁgurations and the minimum amount of resources spent per conﬁguration.

Original: Falkner et al.
Revised: Falkner et al.

Original: (2018) introduced a hybrid model, called BOHB, that uses a probabilistic model to guide the search while retaining the efﬁcient any-time performance of Hyperband.
Revised: (2018) introduced a hybrid model, called BOHB, that uses a probabilistic model to guide the search while retaining the efﬁcient any-time performance of Hyperband.

Original: However, both
Revised: However, both

Original: SH and BOHB can be bottlenecked due to their synchronous nature: stopping decisions are done only after synchronizing all training jobs at certain resource levels (called rungs ).
Revised: SH and BOHB can be bottlenecked due to their synchronous nature: stopping decisions are done only after synchronizing all training jobs at certain resource levels (called rungs ).

Original: This approach is wasteful when the evaluation of some conﬁgurations take longer than others, as it is often the case when training neural networks (Ying et al., 2019), and can substantially delay progress towards high-performing conﬁgurations (see example shown in Figure 1).
Revised: This approach is wasteful when the evaluation of some conﬁgurations take longer than others, as it is often the case when training neural networks (Ying et al., 2019), and can substantially delay progress towards high-performing conﬁgurations (see example shown in Figure 1).

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: (2018) proposed ASHA, which adapts successive halving to the asynchronous parallel case.
Revised: (2018) proposed ASHA, which adapts successive halving to the asynchronous parallel case.

Original: Even though ASHA only relies on the random sampling of new conﬁgurations, it has been shown to outperform synchronous SH and BOHB.
Revised: Even though ASHA only relies on the random sampling of new conﬁgurations, it has been shown to outperform synchronous SH and BOHB.

Original: In this work, we augment ASHA with a
Revised: In this work, we augment ASHA with a

Original: Gaussian process (GP) surrogate, which jointly models the performance across conﬁgurations and
Revised: Gaussian process (GP) surrogate, which jointly models the performance across conﬁgurations and

Original: [Figure]
Revised: [Figure]

Original: rungs to improve the already strong performance of ASHA.
Revised: rungs to improve the already strong performance of ASHA.

Original: The asynchronous nature further requires to handle pending evaluations in principled way to obtain an efﬁcient model-based searcher[Figure]
Revised: The asynchronous nature further requires to handle pending evaluations in principled way to obtain an efﬁcient model-based searcher

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: Dealing with hyperparameter optimization (HPO) in general, and HNAS for neural networks in particular, we would like to make the most efﬁcient use of a given parallel computation budget (e g. number of compute instances) in order to ﬁnd high-accuracy solutions in the shortest wall-clock time possible.
Revised: Dealing with hyperparameter optimization (HPO) in general and HNAS for neural networks in particular, we would like to make the most efﬁcient use of a given parallel computation budget (e g., number of compute instances) in order to ﬁnd high-accuracy solutions in the shortest wall-clock time possible.

Original: Besides exploiting low-ﬁdelity approximations, we demonstrate that asynchronous parallel scheduling is a decisive factor in cost-efﬁcient search.
Revised: Besides exploiting low-ﬁdelity approximations, we demonstrate that asynchronous parallel scheduling is a decisive factor in cost-efﬁcient search.

Original: We further handle pending evaluations through fantasizing (Snoek et al., 2012), which is critical for asynchronous searchers.
Revised: We further handle pending evaluations through fantasizing (Snoek et al., 2012), which is critical for asynchronous searchers.

Original: Our novel combination of asynchronous SH with multi-ﬁdelity BO, dubbed MOdel Based aSynchronous mulTi ﬁdelity optimizER (M O B STER ) substantially outperforms either of them in isolation.
Revised: Our novel combination of asynchronous SH with multi-ﬁdelity BO, dubbed MOdel Based aSynchronous mulTi ﬁdelity optimizER (M O B STER ) substantially outperforms either of them in isolation.

Original: More speciﬁcally:
Revised: More speciﬁcally:

Original: • We clarify differences between existing asynchronous SH extensions: ASHA, as described by Li et al.
Revised: • We clarify differences between existing asynchronous SH extensions: ASHA, as described by Li et al.

Original: (2018) and an arguably simpler stopping rule variant, related to the median rule (Golovin et al., 2017) and ﬁrst implemented in Ray Tune (Liaw et al., 2018).
Revised: (2018) and an arguably simpler stopping rule variant, related to the median rule (Golovin et al., 2017) and ﬁrst implemented in Ray Tune (Liaw et al., 2018).

Original: Although their difference seem subtle, they lead to substantially different behaviour in practice.
Revised: Although their difference seem subtle, they lead to substantially different behaviour in practice.

Original: • We present an extensive ablation study, comparing and analysing different components for asynchronous HNAS.
Revised: • We present an extensive ablation study, comparing and analysing different components for asynchronous HNAS.

Original: While these individual components are not novel, one of our main contributions is to show that, by systematically combining them to form M O B STER , we obtain a reliable and more efﬁcient method than other recently proposed approaches.
Revised: While these individual components are not novel, one of our main contributions is to show that, by systematically combining them to form M O B STER , we obtain a reliable and more efﬁcient method than other recently proposed approaches.

Original: Due to limited space, we present a detailed description of the technical nuances and complexities of model-based asynchronous multi-ﬁdelity HNAS in the supplemental material.
Revised: Due to limited space, we present a detailed description of the technical nuances and complexities of model-based asynchronous multi-ﬁdelity HNAS in Appendix A.3.

Original: • On a variety of neural network benchmarks, we show that M O B STER is more efﬁcient in terms of wall-clock time than other state-of-the-art algorithms.
Revised: • On a variety of neural network benchmarks, we show that M O B STER is more efﬁcient in terms of wall-clock time than other state-of-the-art algorithms.

Original: Unlike BOHB, it does not suffer from substantial synchronization overheads when evaluations are expensive.
Revised: Unlike BOHB, it does not suffer from substantial synchronization overheads when evaluations are expensive.

Original: As a result, we can achieve the same performance in the same amount of wall-clock time but with often just half the computational resources compared to random-sampling based ASHA.
Revised: As a result, we can achieve the same performance in the same amount of wall-clock time but with often just half the computational resources compared to random-sampling based ASHA.

Original: Next, we relate our work to approaches recently published in the literature.
Revised: Next, we relate our work to approaches recently published in the literature.

Original: In Section 2, we review synchronous SH, as well as two asynchronous extensions.
Revised: In Section 2, we review synchronous SH, as well as two asynchronous extensions.

Original: Our novel method is presented in Section 3.
Revised: Our novel method is presented in Section 3.

Original: We present empirical evaluations for the HNAS of various neural architecture types in Section 4, and ﬁnish with conclusions and future work in Section 5.
Revised: We present empirical evaluations for the HNAS of various neural architecture types in Section 4, and ﬁnish with conclusions and future work in Section 5.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: A range of prior Gaussian process (GP) based BO work exploits multiple ﬁdelities of the objective (Kennedy & O’Hagan, 2000; Klein et al., 2017a;
Revised: A range of prior Gaussian process (GP) based BO work exploits multiple ﬁdelities of the objective (Kennedy & O’Hagan, 2000; Klein et al., 2017a;

Original: Poloczek et al., 2018; Cutajar et al., 2018).
Revised: Poloczek et al., 2018; Cutajar et al., 2018).

Original: A joint GP model across conﬁgurations and tasks was used by Swersky et al.
Revised: A joint GP model across conﬁgurations and tasks was used by Swersky et al.

Original: (2013), allowing for trade-offs between cheap auxiliaries and the expensive target.
Revised: (2013), allowing for trade-offs between cheap auxiliaries and the expensive target.

Original: Klein et al.
Revised: Klein et al.

Original: (2017a) presented a continuous multi-ﬁdelity BO method, where training set size is an input.
Revised: (2017a) presented a continuous multi-ﬁdelity BO method, where training set size is an input.

Original: It relies on a complicated and expensive acquisition function, and experiments with asynchronous scheduling are not provided.
Revised: It relies on a complicated and expensive acquisition function, and experiments with asynchronous scheduling are not provided.

Original: Kandasamy et al.
Revised: Kandasamy et al.

Original: (2017) presented BOCA, a Bayesian optimization method that exploits general continuous ﬁdelities of the objective function.
Revised: (2017) presented BOCA, a Bayesian optimization method that exploits general continuous ﬁdelities of the objective function.

Original: While BOCA uses a model similar to ours, they employ a different strategy to select the ﬁdelity, which appears to work less efﬁciently in practice, as we show in our experiments.
Revised: While BOCA uses a model similar to ours, they employ a different strategy to select the ﬁdelity, which appears to work less efﬁciently in practice, as we show in our experiments.

Original: The “freeze-thaw” approach (Swersky et al., 2014) allows for asynchronous parallel evaluations, yet its scheduling is quite different from asynchronous SH.
Revised: The “freeze-thaw” approach (Swersky et al., 2014) allows for asynchronous parallel evaluations, yet its scheduling is quite different from asynchronous SH.

Original: They propose an exponential decay kernel, a reﬁnement of which we use here.
Revised: They propose an exponential decay kernel, a reﬁnement of which we use here.

Original: No results for deep neural network tuning were presented, and no implementation is publicly available.
Revised: No results for deep neural network tuning were presented, and no implementation is publicly available.

Original: Previous work on asynchronous BO has demonstrated performance gains over synchronous BO methods (Alvin et al., 2019; Kandasamy et al., 2016), yet this work did not exploit multiple ﬁdelities of the objective.
Revised: Previous work on asynchronous BO has demonstrated performance gains over synchronous BO methods (Alvin et al., 2019; Kandasamy et al., 2016), yet this work did not exploit multiple ﬁdelities of the objective.

Original: Concurrently to our work, Takeno et al.
Revised: Concurrently to our work, Takeno et al.

Original: (2020) proposed an asynchronous multi-ﬁdelity BO method.
Revised: (2020) proposed an asynchronous multi-ﬁdelity BO method.

Original: However, it is based on an information-theoretic acquisition function that is substantially more complex than our method.
Revised: However, it is based on an information-theoretic acquisition function that is substantially more complex than our method.

Original: Since no implementation is publicly available it is omitted from our comparisons.
Revised: Since no implementation is publicly available it is omitted from our comparisons.

Original: Falkner et al.
Revised: Falkner et al.

Original: (2018) combine synchronous Hyperband with TPE (Bergstra et al., 2011) as a model.
Revised: (2018) combine synchronous Hyperband with TPE (Bergstra et al., 2011) as a model.

Original: Their method, called BOHB, combines early speed-ups of Hyperband with fast convergence later on, and shows competitive performance for NAS (Ying et al., 2019; Dong & Yang, 2020).
Revised: Their method, called BOHB, combines early speed-ups of Hyperband with fast convergence later on, and shows competitive performance for NAS (Ying et al., 2019; Dong & Yang, 2020).

Original: However, compared to our approach, it ﬁts independent kernel density estimator at each rung level, which prevents the interpolation to higher rung levels.
Revised: However, compared to our approach, it ﬁts independent kernel density estimator at each rung level, which prevents the interpolation to higher rung levels.

Original: Apart from this, kernel density estimators do not permit fantasizing the potential outcome of a pending conﬁguration, which makes it unclear how to extend BOHB to the asynchronous setting, and are notoriously sensitive to their bandwidth parameters.
Revised: Apart from this, kernel density estimators do not permit fantasizing the potential outcome of a pending conﬁguration, which makes it unclear how to extend BOHB to the asynchronous setting, and are notoriously sensitive to their bandwidth parameters.

Original:  S YNCHRONOUS AND A SYNCHRONOUS M ULTI -F IDELITY S CHEDULING
Revised:  S YNCHRONOUS AND A SYNCHRONOUS M ULTI -F IDELITY S CHEDULING

Original: Multi-ﬁdelity HNAS considers an objective f ( x , r ) , indexed by the resource level r ∈ { r min , . . .
Revised: Multi-ﬁdelity HNAS considers an objective f ( x , r ) , indexed by the resource level r ∈ { r min , . . .

Original: , r max } .
Revised: , r max } .

Original: The target function of interest is f ( x ) =
Revised: The target function of interest is f ( x ) =

Original: f ( x , r max ) , while evaluations of f ( x , r ) for r < r max are cheaper, lower-ﬁdelity evaluations that may be correlated with f ( x ) .
Revised: f ( x , r max ) , while evaluations of f ( x , r ) for r < r max are cheaper, lower-ﬁdelity evaluations that may be correlated with f ( x ) .

Original: Here, we consider the number of training epochs ( i.e. , full sweeps over the data) as the resource r , though other choices are possible, such as training subset ratios (Klein et al., 2017a).
Revised: Here, we consider the number of training epochs ( i.e. , full sweeps over the data) as the resource r , though other choices are possible, such as training subset ratios (Klein et al., 2017a).

Original: Two kinds of decisions are made in multi-ﬁdelity HNAS: choosing the conﬁgurations to evaluate and scheduling their evaluations.
Revised: Two kinds of decisions are made in multi-ﬁdelity HNAS: choosing the conﬁgurations to evaluate and scheduling their evaluations.

Original: In this work, conﬁgurations are chosen non-uniformly based on the GP surrogate model that accounts for pending and actual evaluations.
Revised: In this work, conﬁgurations are chosen non-uniformly based on the GP surrogate model that accounts for pending and actual evaluations.

Original: Scheduling decisions are stop/go decisions that happen at certain resource levels called rungs .
Revised: Scheduling decisions are stop/go decisions that happen at certain resource levels called rungs .

Original: We make use of successive halving (SH) for its simplicity and strong empirical performance (Jamieson & Talwalkar, 2016; Li et al., 2018).
Revised: We make use of successive halving (SH) for its simplicity and strong empirical performance (Jamieson & Talwalkar, 2016; Li et al., 2018).

Original: Let η ∈ { 2 , 3 , 4 } be the halving constant, r max and r min the maximum and minimum resource level for an evaluation.
Revised: Let η ∈ { 2 , 3 , 4 } be the halving constant, r max and r min the maximum and minimum resource level for an evaluation.

Original: We assume, for simplicity, that r max /r min =
Revised: We assume, for simplicity, that r max /r min =

Original: η K , where K ∈ N .
Revised: η K , where K ∈ N .

Original: The full set of rungs is
Revised: The full set of rungs is

Original: R = { r min η k | k = 0 , . . .
Revised: R = { r min η k | k = 0 , . . .

Original: , K } .
Revised: , K } .

Original: In the sequel, the term “rung” will be overloaded, denoting both the resource level at which stop/go decisions are made and the list of conﬁgurations that reached it.
Revised: In the sequel, the term “rung” will be overloaded, denoting both the resource level at which stop/go decisions are made and the list of conﬁgurations that reached it.

Original: In synchronous SH , each rung has an a priori ﬁxed size ( i.e. , number of slots for conﬁgurations evaluated until the rung level), the size ratio between successive rungs being η − 1 (e.g., n ( r ) =
Revised: In synchronous SH , each rung has an a priori ﬁxed size ( i.e. , number of slots for conﬁgurations evaluated until the rung level), the size ratio between successive rungs being η − 1 (e.g., n ( r ) =

Original: r max /r for r ∈ R ).
Revised: r max /r for r ∈ R ).

Original: A round of the algorithm starts with evaluating n ( r min ) conﬁgurations to the lowest rung r = r min , making use of parallel computation if available.
Revised: A round of the algorithm starts with evaluating n ( r min ) conﬁgurations to the lowest rung r = r min , making use of parallel computation if available.

Original: Once all of them ﬁnish, the η − 1 fraction of top performing conﬁgurations are promoted to the next rung while the others are terminated.
Revised: Once all of them ﬁnish, the η − 1 fraction of top performing conﬁgurations are promoted to the next rung while the others are terminated.

Original: Each rung is a synchronization point for many training tasks: it has to be populated entirely before any conﬁguration can be promoted (see Figure 1 left for a visualization).
Revised: Each rung is a synchronization point for many training tasks: it has to be populated entirely before any conﬁguration can be promoted (see Figure 1 left for a visualization).

Original: Synchronous scheduling makes less efﬁcient use of parallel computation than asynchronous scheduling.
Revised: Synchronous scheduling makes less efﬁcient use of parallel computation than asynchronous scheduling.

Original: Conﬁgurations in the same rung may require quite different compute times, for example, if we search over hyperparameters that control the network size.
Revised: Conﬁgurations in the same rung may require quite different compute times, for example, if we search over hyperparameters that control the network size.

Original: Hence, some workers may just be idle at a synchronization point.
Revised: Hence, some workers may just be idle at a synchronization point.

Original: Evaluations at larger resource levels are observed earlier with asynchronous scheduling.
Revised: Evaluations at larger resource levels are observed earlier with asynchronous scheduling.

Original: However, the risk is to continue mediocre conﬁgurations because they were selected earlier.
Revised: However, the risk is to continue mediocre conﬁgurations because they were selected earlier.

Original: Figure 1 illustrates the differences between asynchronous and synchronous scheduling.
Revised: Figure 1 illustrates the differences between asynchronous and synchronous scheduling.

Original: We distinguish two variants of asynchronous SH proposed in prior work.
Revised: We distinguish two variants of asynchronous SH proposed in prior work.

Original: During the optimization process, for any hyperparameter and architecture conﬁguration x ∈ X that is currently being evaluated, a decision needs to be made once it reaches the next rung r ∈ R .
Revised: During the optimization process, for any hyperparameter and architecture conﬁguration x ∈ X that is currently being evaluated, a decision needs to be made once it reaches the next rung r ∈ R .

Original: Given the new observed data point y at level r , the binary predicate continue ( x , r, y ) evaluates to true iff y is in the top η − 1 fraction of records at the rung.
Revised: Given the new observed data point y at level r , the binary predicate continue ( x , r, y ) evaluates to true iff y is in the top η − 1 fraction of records at the rung.

Original: Stopping variant.
Revised: Stopping variant.

Original: This is a simple extension of the median stopping rule (Golovin et al., 2017), which is implemented in Ray Tune (Liaw et al., 2018).
Revised: This is a simple extension of the median stopping rule (Golovin et al., 2017), which is implemented in Ray Tune (Liaw et al., 2018).

Original: As soon as a job for x reaches a rung r , if continue ( x , r, y ) is true, it continues towards the next rung level.
Revised: As soon as a job for x reaches a rung r , if continue ( x , r, y ) is true, it continues towards the next rung level.

Original: Otherwise, it is stopped , and the worker becomes free to pick up a novel evaluation.
Revised: Otherwise, it is stopped , and the worker becomes free to pick up a novel evaluation.

Original: As long as fewer than η conﬁgurations have reached a rung, the job is always continued.
Revised: As long as fewer than η conﬁgurations have reached a rung, the job is always continued.

Original: Promotion variant.
Revised: Promotion variant.

Original: This variant of asynchronous SH was presented by Li et al.
Revised: This variant of asynchronous SH was presented by Li et al.

Original: and called
Revised: and called

Original: ASHA .
Revised: ASHA .

Original: Note that, in the remaining text, we will refer to general asynchronous SH as ASHA and indicate the both variants with promotion or stopping based ASHA.
Revised: Note that, in the remaining text, we will refer to general asynchronous SH as ASHA and indicate the both variants with promotion or stopping based ASHA.

Original: Once a job for some x reaches a rung r , it is paused there, and the worker is released.
Revised: Once a job for some x reaches a rung r , it is paused there, and the worker is released.

Original: The evaluation of x can be promoted ( i.e. , continued to the next rung) later on.
Revised: The evaluation of x can be promoted ( i.e. , continued to the next rung) later on.

Original: When a worker becomes available, rungs are scanned in descending order, running a job to promote the ﬁrst paused x for which continue ( x , r, y ) is true.
Revised: When a worker becomes available, rungs are scanned in descending order, running a job to promote the ﬁrst paused x for which continue ( x , r, y ) is true.

Original: When no such paused conﬁguration exists, a new conﬁguration evaluation is started.
Revised: When no such paused conﬁguration exists, a new conﬁguration evaluation is started.

Original: With fewer than η metrics recorded at a rung, no promotions happen there.
Revised: With fewer than η metrics recorded at a rung, no promotions happen there.

Original: [Figure]
Revised: [Figure]

Original: The stopping and promotion variants can exhibit a rather different behaviour (see Figure 2), as also demonstrated in our experiments.
Revised: The stopping and promotion variants can exhibit a rather different behaviour (see Figure 2), as also demonstrated in our experiments.

Original: Initially, the stopping variant gives most conﬁgurations the beneﬁt of doubt, while the promotion variant pauses evaluations until they can be compared against a sufﬁcient number of competitors.
Revised: Initially, the stopping variant gives most conﬁgurations the beneﬁt of doubt, while the promotion variant pauses evaluations until they can be compared against a sufﬁcient number of competitors.

Original: Finally, note that asynchronous SH can be generalized to asynchronous Hyperband in much the same way as in the synchronous case, as shown by Li et al.
Revised: Finally, note that asynchronous SH can be generalized to asynchronous Hyperband in much the same way as in the synchronous case, as shown by Li et al.

Original: However, it was reported that asynchronous SH typically outperforms the Hyperband for expensive tuning problems, which we also show in our experiments in Section 4.1.
Revised: However, it was reported that asynchronous SH typically outperforms the Hyperband for expensive tuning problems, which we also show in our experiments in Section 4.1.

Original: After some initial exploration, both with random and model-based variants, we conﬁrmed this observation.
Revised: After some initial exploration, both with random and model-based variants, we conﬁrmed this observation.

Original: In the following, we will thus restrict our attention to SH scheduling.
Revised: In the following, we will thus restrict our attention to SH scheduling.

Original:  M ODEL - BASED A SYNCHRONOUS M ULTI -F IDELITY O PTIMIZER
Revised:  M ODEL - BASED A SYNCHRONOUS M ULTI -F IDELITY O PTIMIZER

Original: We now describe our new model-based method that is able to make asynchronous decisions across different ﬁdelities of the objective function and in presence of pending evaluations.
Revised: We now describe our new model-based method that is able to make asynchronous decisions across different ﬁdelities of the objective function and in presence of pending evaluations.

Original: Motivated by the fact that in standard sequential settings, Gaussian process (GP) based BO outperforms random search (Shahriari et al., 2016), we equip asynchronous SH with decisions based on a GP surrogate model .
Revised: Motivated by the fact that in standard sequential settings, Gaussian process (GP) based BO outperforms random search (Shahriari et al., 2016), we equip asynchronous SH with decisions based on a GP surrogate model .

Original: Following a range of prior work on non-parallel multi-ﬁdelity BO (Swersky et al., 2013; Klein et al., 2017a; Poloczek et al., 2018;
Revised: Following a range of prior work on non-parallel multi-ﬁdelity BO (Swersky et al., 2013; Klein et al., 2017a; Poloczek et al., 2018;

Original: Swersky et al., 2014), we place a GP prior over f ( x , r ) to jointly model not only the auto-correlation within each ﬁdelity, but also the cross-correlations between ﬁdelities,
Revised: Swersky et al., 2014), we place a GP prior over f ( x , r ) to jointly model not only the auto-correlation within each ﬁdelity, but also the cross-correlations between ﬁdelities,

Original: [Equation]
Revised: [Equation]

Original: This is in contrast to synchronous BOHB
Revised: This is in contrast to synchronous BOHB

Original: (Falkner et al., 2018), which just employs an independent
Revised: (Falkner et al., 2018), which just employs an independent

Original: TPE (Bergstra et al., 2011) model over x for each rung level r ∈ R , preventing it from transfering information from lower to higher rungs.
Revised: TPE (Bergstra et al., 2011) model over x for each rung level r ∈ R , preventing it from transfering information from lower to higher rungs.

Original: Pending evaluations and fantasizing.
Revised: Pending evaluations and fantasizing.

Original: Since multiple conﬁgurations are evaluated in parallel, we have to deal with pending evaluations (for which results have not yet been obtained) when proposing new conﬁgurations.
Revised: Since multiple conﬁgurations are evaluated in parallel, we have to deal with pending evaluations (for which results have not yet been obtained) when proposing new conﬁgurations.

Original: Failing to do so runs the risk of choosing the same (or very similar) conﬁgurations over and over, which would result in a poor sampling efﬁciency.
Revised: Failing to do so runs the risk of choosing the same (or very similar) conﬁgurations over and over, which would result in a poor sampling efﬁciency.

Original: We do so by fantasizing the outcomes of pending evaluations, as proposed by Snoek et al.
Revised: We do so by fantasizing the outcomes of pending evaluations, as proposed by Snoek et al.

Original: In a nutshell, we marginalize the acquisition function over the GP predictive posterior for pending conﬁgurations.
Revised: In a nutshell, we marginalize the acquisition function over the GP predictive posterior for pending conﬁgurations.

Original: In practice, we approximate this by averaging the acquisition function over sampled function values.
Revised: In practice, we approximate this by averaging the acquisition function over sampled function values.

Original: In the context of synchronous scheduling, dealing with pending evaluations is also known as batch
Revised: In the context of synchronous scheduling, dealing with pending evaluations is also known as batch

Original: HPO , a problem which has seen a lot of prior attention (Ginsbourger et al., 2010; Desautels et al., 2014; González et al., 2016; Alvin et al., 2019).
Revised: HPO , a problem which has seen a lot of prior attention (Ginsbourger et al., 2010; Desautels et al., 2014; González et al., 2016; Alvin et al., 2019).

Original: Surprisingly, most of these works ignore fantasizing, claiming it is too expensive to do (Ginsbourger et al., 2011), and propose heuristics like local penalties or artiﬁcial randomization by Thompson sampling (Hernandez-Lobato et al., 2017; Kandasamy et al., 2016).
Revised: Surprisingly, most of these works ignore fantasizing, claiming it is too expensive to do (Ginsbourger et al., 2011), and propose heuristics like local penalties or artiﬁcial randomization by Thompson sampling (Hernandez-Lobato et al., 2017; Kandasamy et al., 2016).

Original: Quite in contrast to such claims, we ﬁnd that, as long as kernel hyperparameters remain constant, fantasizing comes almost for free.
Revised: Quite in contrast to such claims, we ﬁnd that, as long as kernel hyperparameters remain constant, fantasizing comes almost for free.

Original: For GP regression, the posterior covariance matrix does not depend on the function values (only on the inputs), which means that fantasizing does not require additional cubic-complexity computations.
Revised: For GP regression, the posterior covariance matrix does not depend on the function values (only on the inputs), which means that fantasizing does not require additional cubic-complexity computations.

Original: Details are given in the supplemental material.
Revised: Details are given in Appendix A.1.

Original: Lastly, it is important to underscore the unique challenges posed by pending evaluations and fantasizing in the asynchronous SH case, in particular, those not encountered in the synchronous case.
Revised: Lastly, it is important to underscore the unique challenges posed by pending evaluations and fantasizing in the asynchronous SH case, in particular, those not encountered in the synchronous case.

Original: We elaborate on these challenges in the supplemental material , and summarize our asynchronous algorithm for overcoming them in concise pseudocode.
Revised: We elaborate on these challenges in Appendix A.3 , and summarize our asynchronous algorithm for overcoming them in concise pseudocode.

Original: Kernel choice.
Revised: Kernel choice.

Original: We implemented and compared a range of different surrogate models for f ( x , r ) : a joint Matérn 5 / 2 kernel with automatic relevance determination (ARD) (Snoek et al., 2012) over ( x , r ) , with and without warping of r ; the additive exponential decay model of Swersky et al.
Revised: We implemented and compared a range of different surrogate models for f ( x , r ) : a joint Matérn 5 / 2 kernel with automatic relevance determination (ARD) (Snoek et al., 2012) over ( x , r ) , with and without warping of r ; the additive exponential decay model of Swersky et al.

Original: (2014); a novel non-additive extension of the latter, relaxing underlying assumptions which may not hold in practice.
Revised: (2014); a novel non-additive extension of the latter, relaxing underlying assumptions which may not hold in practice.

Original: In all cases, the base kernel on x alone is Matérn 5 / 2 ARD.
Revised: In all cases, the base kernel on x alone is Matérn 5 / 2 ARD.

Original: Details for our new surrogate model are provided in the supplemental material.
Revised: Details for our new surrogate model are provided in Appendix A.2.

Original: Acquisition strategy.
Revised: Acquisition strategy.

Original: We follow synchronous BOHB (Falkner et al., 2018) in choosing x by optimizing the expected improvement (EI) acquisition function (Shahriari et al., 2016) over the posterior on f ( · , r acq ) , where r acq is ﬁxed for each decision.
Revised: We follow synchronous BOHB (Falkner et al., 2018) in choosing x by optimizing the expected improvement (EI) acquisition function (Shahriari et al., 2016) over the posterior on f ( · , r acq ) , where r acq is ﬁxed for each decision.

Original: We choose r acq ∈ R as the largest rung for which at least L metric values have been observed, where L is typically set to the input dimensionality ( i.e. , number of hyperparameters).
Revised: We choose r acq ∈ R as the largest rung for which at least L metric values have been observed, where L is typically set to the input dimensionality ( i.e. , number of hyperparameters).

Original: After some initial period, r acq eventually converges to r max .
Revised: After some initial period, r acq eventually converges to r max .

Original: This strategy does not take into account that the choice of x may have different downstream stopping or promotion consequences, interacting with currently pending evaluations.
Revised: This strategy does not take into account that the choice of x may have different downstream stopping or promotion consequences, interacting with currently pending evaluations.

Original: We leave the exploration of alternative acquisition strategies as an important direction for future work.
Revised: We leave the exploration of alternative acquisition strategies as an important direction for future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare M O B STER to a range of state-of-the-art algorithms.
Revised: We compare M O B STER to a range of state-of-the-art algorithms.

Original: For synchronous Hyperband and
Revised: For synchronous Hyperband and

Original: BOHB, we used the HpBandSter package.
Revised: BOHB, we used the HpBandSter package.

Original: 1 , which uses a simple mechanism to reduce the amount of idling workers: if there are more free workers than slots in the rung, left-over workers are used to immediately start the next bracket.
Revised: 1 , which uses a simple mechanism to reduce the amount of idling workers: if there are more free workers than slots in the rung, left-over workers are used to immediately start the next bracket.

Original: There is no open-source implementation of Swersky et al.
Revised: There is no open-source implementation of Swersky et al.

Original: (2014) and Takeno et al.
Revised: (2014) and Takeno et al.

Original: (2020) for which the implementation complexity is considerable, which is why we omitted them here.
Revised: (2020) for which the implementation complexity is considerable, which is why we omitted them here.

Original: We implemented M O B STER , as well as all other asynchronous techniques, in AutoGluon .
Revised: We implemented M O B STER , as well as all other asynchronous techniques, in AutoGluon .

Original: 2 , which will all be open-sourced along with this paper.
Revised: 2 , which will all be open-sourced along with this paper.

Original: A number of worker nodes run training jobs, publishing metrics after each epoch.
Revised: A number of worker nodes run training jobs, publishing metrics after each epoch.

Original: One of the workers doubles as master, concurrently running the HP optimization and scheduling.
Revised: One of the workers doubles as master, concurrently running the HP optimization and scheduling.

Original: We also compare against Dragonﬂy (Kandasamy et al., 2019).
Revised: We also compare against Dragonﬂy (Kandasamy et al., 2019).

Original: While AutoGluon allows to parallelize both across instances and processes on a single instance, Dragonﬂy only supports the latter.
Revised: While AutoGluon allows to parallelize both across instances and processes on a single instance, Dragonﬂy only supports the latter.

Original: Hence, for a fair comparison, we run our method only on a single instance if we compare against
Revised: Hence, for a fair comparison, we run our method only on a single instance if we compare against

Original: Dragonﬂy.
Revised: Dragonﬂy.

Original: For all other experiments we parallelize across multiple instances.
Revised: For all other experiments we parallelize across multiple instances.

Original: For all methods using SH the halving rate was η = 3 .
Revised: For all methods using SH the halving rate was η = 3 .

Original: We report the immediate regret r t =
Revised: We report the immediate regret r t =

Original: y t − y (cid:63) after wall-clock time t , where y (cid:63) is the best observed validation error across all methods, runs and time steps on this dataset.
Revised: y t − y (cid:63) after wall-clock time t , where y (cid:63) is the best observed validation error across all methods, runs and time steps on this dataset.

Original: Using the regret instead of just the validation error gives a more detailed picture of progress towards the global optimum.
Revised: Using the regret instead of just the validation error gives a more detailed picture of progress towards the global optimum.

Original: For all classiﬁcation datasets we cap the regret at a lower bound of 10 − 3 , which corresponds to a 0 .
Revised: For all classiﬁcation datasets we cap the regret at a lower bound of 10 − 3 , which corresponds to a 0 .

Original: 1% difference in classiﬁcation accuracy to the global optimum.
Revised: 1% difference in classiﬁcation accuracy to the global optimum.

Original: We consider lower values to be not signiﬁcant in practice and attributed them to the intrinsic noise of the objective function.
Revised: We consider lower values to be not signiﬁcant in practice and attributed them to the intrinsic noise of the objective function.

Original: For each method we report mean and the standard error of the mean across multiple runs with different seeds of the random number generator.
Revised: For each method we report mean and the standard error of the mean across multiple runs with different seeds of the random number generator.

Original: We run experiments on the following set of benchmarks (for further details regarding the search spaces see the supplemental material. All experiments were conducted on AWS EC2, using m4.xlarge instances for CPU- and g4dn.xlarge instances for GPU-based computations, respectively.
Revised: We run experiments on the following set of benchmarks (for further details regarding the search spaces see Appendix B. All experiments were conducted on AWS EC2, using m4.xlarge instances for CPU- and g4dn.xlarge instances for GPU-based computations, respectively.

Original: Tabular Datasets: First, we optimized the hyperparameters of a multi-layer perceptron (MLP) on the tabular classiﬁcation dataset electricity gathered from OpenML (Vanschoren et al., 2014).
Revised: Tabular Datasets: First, we optimized the hyperparameters of a multi-layer perceptron (MLP) on the tabular classiﬁcation dataset electricity gathered from OpenML (Vanschoren et al., 2014).

Original: The
Revised: The

Original: MLP has two hidden layers with ReLU activations.
Revised: MLP has two hidden layers with ReLU activations.

Original: We tuned 9 hyperparameters: the learning rate of
Revised: We tuned 9 hyperparameters: the learning rate of

Original: ADAM (Kingma & Ba, 2015), batch size, weight decay, and, for each layer, dropout rate, number of units and the scale of a uniform distribution to initialize weights.
Revised: ADAM (Kingma & Ba, 2015), batch size, weight decay, and, for each layer, dropout rate, number of units and the scale of a uniform distribution to initialize weights.

Original: The number of epochs r varies between r min = 1 and r max = 81 .
Revised: The number of epochs r varies between r min = 1 and r max = 81 .

Original: Neural Architecture Search: To evaluate our approach in a NAS setting, we used the NASBench datasets from Dong & Yang (2020) which consists of a ofﬂine evaluated grid of neural network architectures for three different datasets: CIFAR-10, CIFAR-100 (Krizhevsky, 2009) and Imagenet16 (Chrabaszcz et al., 2017).
Revised: Neural Architecture Search: To evaluate our approach in a NAS setting, we used the NASBench datasets from Dong & Yang (2020) which consists of a ofﬂine evaluated grid of neural network architectures for three different datasets: CIFAR-10, CIFAR-100 (Krizhevsky, 2009) and Imagenet16 (Chrabaszcz et al., 2017).

Original: Categorical hyperparameters were one-hot encoded.
Revised: Categorical hyperparameters were one-hot encoded.

Original: The minimum and maximum number of epochs were r min = 1 and r max = 200 .
Revised: The minimum and maximum number of epochs were r min = 1 and r max = 200 .

Original: To simulate realistic dynamics of asynchronous NAS, we put each worker to sleep for the time per epoch each real evaluation took.
Revised: To simulate realistic dynamics of asynchronous NAS, we put each worker to sleep for the time per epoch each real evaluation took.

Original: This does not speed up experiments, but decreases cost, since no GPU instances are required.
Revised: This does not speed up experiments, but decreases cost, since no GPU instances are required.

Original: Compared to the other benchmarks which contains integer and continuous hyperparameters, this benchmark is purely discrete.
Revised: Compared to the other benchmarks which contains integer and continuous hyperparameters, this benchmark is purely discrete.

Original: Image Classiﬁcation: Next, we tuned ResNet (He et al., 2016) on the image classiﬁcation dataset
Revised: Image Classiﬁcation: Next, we tuned ResNet (He et al., 2016) on the image classiﬁcation dataset

Original: CIFAR-10 (Krizhevsky, 2009), the batch size, weight decay, initial learning rate and momentum.
Revised: CIFAR-10 (Krizhevsky, 2009), the batch size, weight decay, initial learning rate and momentum.

Original: We used 40000 images for training, 10000 for validation, and applied standard data augmentation (cropping; horizontal ﬂips).
Revised: We used 40000 images for training, 10000 for validation, and applied standard data augmentation (cropping; horizontal ﬂips).

Original: SGD is used for training, the learning rate is multiplied by 0 .
Revised: SGD is used for training, the learning rate is multiplied by 0 .

Original: 1 after 25 epochs.
Revised: 1 after 25 epochs.

Original: We used r min = 1 and r max = 27 .
Revised: We used r min = 1 and r max = 27 .

Original: As our goal is to evaluate our approach, we omitted additional tricks-of-the-trade required to obtain state-of-the-art accuracy (Yang et al., 2020).
Revised: As our goal is to evaluate our approach, we omitted additional tricks-of-the-trade required to obtain state-of-the-art accuracy (Yang et al., 2020).

Original: Language Modelling:
Revised: Language Modelling:

Original: Lastly, we considered an LSTM (Hochreiter & Schmidhuber, 1997), applied to language modelling on the WikiText-2 dataset (Merity et al., 2017) with the default training/validation/test split.
Revised: Lastly, we considered an LSTM (Hochreiter & Schmidhuber, 1997), applied to language modelling on the WikiText-2 dataset (Merity et al., 2017) with the default training/validation/test split.

Original: We optimized the initial learning rate of SGD, batch size, dropout rate, gradient clipping range, and decay factor multiplied to the learning rate if the validation error has not improved upon the previous epoch.
Revised: We optimized the initial learning rate of SGD, batch size, dropout rate, gradient clipping range, and decay factor multiplied to the learning rate if the validation error has not improved upon the previous epoch.

Original: The LSTM consisted of two layers with 1500 hidden units each.
Revised: The LSTM consisted of two layers with 1500 hidden units each.

Original: For the word embeddings, we also used 1500 hidden units and tied them with the output.
Revised: For the word embeddings, we also used 1500 hidden units and tied them with the output.

Original: Epochs vary between r min = 1 and r max = 81 .
Revised: Epochs vary between r min = 1 and r max = 81 .

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: We start with an in-depth analysis of our proposed method.
Revised: We start with an in-depth analysis of our proposed method.

Original: If not explicitly stated otherwise, we report the immediate regret over wall-clock time of the MLP benchmark on the electricity dataset for 4 workers and the NASBench201 CIFAR-10 dataset for 8 workers.
Revised: If not explicitly stated otherwise, we report the immediate regret over wall-clock time of the MLP benchmark on the electricity dataset for 4 workers and the NASBench201 CIFAR-10 dataset for 8 workers.

Original: These two benchmarks cover continuous as well as discrete hyperparameter optimization problems.
Revised: These two benchmarks cover continuous as well as discrete hyperparameter optimization problems.

Original: Successive Halving vs Hyperband: First, we compare asynchronous scheduling based on SH versus Hyperband.
Revised: Successive Halving vs Hyperband: First, we compare asynchronous scheduling based on SH versus Hyperband.

Original: We run different instances of SH, where we vary the minimum rung level r min in the same way as Hyperband.
Revised: We run different instances of SH, where we vary the minimum rung level r min in the same way as Hyperband.

Original: Figure 3 shows the results for both promotion and stopping based scheduling, combined with sampling from our model or random.
Revised: Figure 3 shows the results for both promotion and stopping based scheduling, combined with sampling from our model or random.

Original: Similar to results reported by Li et al.
Revised: Similar to results reported by Li et al.

Original: (2018), the performance of SH is decaying with an increasing r min both for our model-based
Revised: (2018), the performance of SH is decaying with an increasing r min both for our model-based

Original: [Figure]
Revised: [Figure]

Original: As comparison we also include asynchronous random search (RS)
Revised: As comparison we also include asynchronous random search (RS)

Original: [Figure]
Revised: [Figure]

Original: and random sampling.
Revised: and random sampling.

Original: Also, the most aggressive bracket, where r min = 1 epoch, works as good or even better than Hyperband, most distinctly for the promotion based scheduling.
Revised: Also, the most aggressive bracket, where r min = 1 epoch, works as good or even better than Hyperband, most distinctly for the promotion based scheduling.

Original: For the remaining experiments we therefore use SH with a minimum rung level of r min = 1 epoch.
Revised: For the remaining experiments we therefore use SH with a minimum rung level of r min = 1 epoch.

Original: Kernels: We analyse the inﬂuence of the choice of kernel and mean function for f ( x , r ) on the performance of the stopping and promotion variant of asynchronous BOHB.
Revised: Kernels: We analyse the inﬂuence of the choice of kernel and mean function for f ( x , r ) on the performance of the stopping and promotion variant of asynchronous BOHB.

Original: We compare different variants of the exponential decay model (see the supplemental material for more details): ﬁxing δ = 0 as in Swersky et al.
Revised: We compare different variants of the exponential decay model (see Appendix A.2. for more details): ﬁxing δ = 0 as in Swersky et al.

Original: (2014), δ = 1 , or learning δ with the other GP hyperparameters.
Revised: (2014), δ = 1 , or learning δ with the other GP hyperparameters.

Original: We use a Matérn 5 /2 ARD kernel for the k X model between conﬁgurations x Snoek et al.
Revised: We use a Matérn 5 / ARD kernel for the k X model between conﬁgurations x Snoek et al.

Original: As an additional baseline, we use a Matérn 5 / 2 ARD kernel and constant mean function on inputs ( x , r ) .
Revised: As an additional baseline, we use a Matérn 5 / 2 ARD kernel and constant mean function on inputs ( x , r ) .

Original: In Figure 4 we compare the various types of kernel.
Revised: In Figure 4 we compare the various types of kernel.

Original: Maybe somewhat surprisingly, there is no signiﬁcant difference between the surrogate models and the precise model for inter- or extrapolation along r seems less important.
Revised: Maybe somewhat surprisingly, there is no signiﬁcant difference between the surrogate models and the precise model for inter- or extrapolation along r seems less important.

Original: For the remaining experiments, we use the exponential decay kernel with learned δ as the most ﬂexible choice which also obtains slightly better performance.
Revised: For the remaining experiments, we use the exponential decay kernel with learned δ as the most ﬂexible choice which also obtains slightly better performance.

Original: M O B STER vs ASHA: .
Revised: M O B STER vs ASHA: .

Original: Next, we show the beneﬁt of sampling from our model instead of uniformly at random for both scheduling variants (see Figure 4).
Revised: Next, we show the beneﬁt of sampling from our model instead of uniformly at random for both scheduling variants (see Figure 4).

Original: Initially, M O B STER (stopping / promotion) performs similar to its random-based counterpart ASHA (stopping / promotion).
Revised: Initially, M O B STER (stopping / promotion) performs similar to its random-based counterpart ASHA (stopping / promotion).

Original: However, with enough datapoints, it rapidly approaches the global optimum, which is in line with typical improvements of
Revised: However, with enough datapoints, it rapidly approaches the global optimum, which is in line with typical improvements of

Original: BO over random search (Falkner et al., 2018).
Revised: BO over random search (Falkner et al., 2018).

Original: [Figure]
Revised: [Figure]

Original: Scaling with the number of workers: As seen in Figure 4 right, the substantial speed-ups of
Revised: Scaling with the number of workers: As seen in Figure 4 right, the substantial speed-ups of

Original: M O B STER over ASHA are observed consistently with 2, 4, 8, and 16 workers on the MLP benchmark.
Revised: M O B STER over ASHA are observed consistently with 2, 4, 8, and 16 workers on the MLP benchmark.

Original: In fact, in most cases M O B STER converges to a high-accuracy solution in about the same time as
Revised: In fact, in most cases M O B STER converges to a high-accuracy solution in about the same time as

Original: ASHA needs with twice the number of workers: using a joint surrogate model can save half the resources compared to random search .
Revised: ASHA needs with twice the number of workers: using a joint surrogate model can save half the resources compared to random search .

Original: Stopping versus promotion scheduling: .
Revised: Stopping versus promotion scheduling: .

Original: Next, we focus on the two variants of asynchronous scheduling (stopping and promotion) described in Section 2.
Revised: Next, we focus on the two variants of asynchronous scheduling (stopping and promotion) described in Section 2.

Original: The stopping rule achieves a good performance faster than the promotion variant (Figure 3, right) up to 4000 seconds on the NASBench benchmark and up to 500 seconds on the MLP benchmark.
Revised: The stopping rule achieves a good performance faster than the promotion variant (Figure 3, right) up to 4000 seconds on the NASBench benchmark and up to 500 seconds on the MLP benchmark.

Original: Recall from Section 2 that for the stopping rule, the initial η conﬁgurations are not stopped, while the promotion variant pauses all initial conﬁgurations at the lowest rung level of their bracket.
Revised: Recall from Section 2 that for the stopping rule, the initial η conﬁgurations are not stopped, while the promotion variant pauses all initial conﬁgurations at the lowest rung level of their bracket.

Original: However, at least for the MLP benchmark, after this initial phase the promotion-based scheduling achieves slightly better performance.
Revised: However, at least for the MLP benchmark, after this initial phase the promotion-based scheduling achieves slightly better performance.

Original: 4. C OMPARISON TO S TATE - OF - THE -A RT M ETHODS
Revised: 4. C OMPARISON TO S TATE - OF - THE -A RT M ETHODS

Original: [Figure]
Revised: [Figure]

Original: We now compare M O B STER against a range of state-of-the-art algorithms: ASHA (Li et al., 2018), asynchronous BO (Snoek et al., 2012)
Revised: We now compare M O B STER against a range of state-of-the-art algorithms: ASHA (Li et al., 2018), asynchronous BO (Snoek et al., 2012)

Original: (A-BO), asynchronous random search (Bergstra et al., 2011)
Revised: (A-BO), asynchronous random search (Bergstra et al., 2011)

Original: (A-RS), synchronous BOHB
Revised: (A-RS), synchronous BOHB

Original: (Falkner et al., 2018)
Revised: (Falkner et al., 2018)

Original: (S-BOHB) , synchronous Hyperband (Li et al., 2017) (S-HB), asynchronous BO with Thompson Sampling (Kandasamy et al., 2016) (Dragonﬂy) and BOCA (Kandasamy et al., 2017)
Revised: (S-BOHB) , synchronous Hyperband (Li et al., 2017) (S-HB), asynchronous BO with Thompson Sampling (Kandasamy et al., 2016) (Dragonﬂy) and BOCA (Kandasamy et al., 2017)

Original: (Dragonﬂy-MF).
Revised: (Dragonﬂy-MF).

Original: MLP and NASBench201 Figure 5 provides results for the MLP and the NASBench201 benchmark, both for the single-instance (left) and multi-instance setting (right) (see Appendix for results on the other two NASBench201 benchmarks).
Revised: MLP and NASBench201 Figure 5 provides results for the MLP and the NASBench201 benchmark, both for the single-instance (left) and multi-instance setting (right) (see Appendix for results on the other two NASBench201 benchmarks).

Original: M O B STER outperforms both Dragonﬂy and Dragonﬂy-MF.
Revised: M O B STER outperforms both Dragonﬂy and Dragonﬂy-MF.

Original: Thereby, parallelizing across instances provides an additional speed-up compared to the singleinstance setting.
Revised: Thereby, parallelizing across instances provides an additional speed-up compared to the singleinstance setting.

Original: Somewhat surprisingly Dragonﬂy-MF is slower in terms of convergence than
Revised: Somewhat surprisingly Dragonﬂy-MF is slower in terms of convergence than

Original: Dragonﬂy on the NASBench201 benchmark, which indicates that it cannot model the performance across ﬁdelities well.
Revised: Dragonﬂy on the NASBench201 benchmark, which indicates that it cannot model the performance across ﬁdelities well.

Original: For the multi-instance scenario, ASHA (stopping and promotion) performs better than synchronous Hyperband and BOHB on the discrete NASBench201 benchmarks.
Revised: For the multi-instance scenario, ASHA (stopping and promotion) performs better than synchronous Hyperband and BOHB on the discrete NASBench201 benchmarks.

Original: But, due to its random sampling, it often does not converge to a sensible performance on the continuous
Revised: But, due to its random sampling, it often does not converge to a sensible performance on the continuous

Original: MLP problem.
Revised: MLP problem.

Original: We attributed S-BOHB poor performance on the NASBench201 datasets to its kernel density estimator which seems to struggle with discrete spaces.
Revised: We attributed S-BOHB poor performance on the NASBench201 datasets to its kernel density estimator which seems to struggle with discrete spaces.

Original: As described above, the stopping variant of M O B STER works better in the beginning, but the promotion variant achieves a better ﬁnal performance.
Revised: As described above, the stopping variant of M O B STER works better in the beginning, but the promotion variant achieves a better ﬁnal performance.

Original: However, both variants of M O B STER outperform all other baselines.
Revised: However, both variants of M O B STER outperform all other baselines.

Original: ResNet on CIFAR-10.
Revised: ResNet on CIFAR-10.

Original: Based on the previous results we limit our comparison to ASHA as well as
Revised: Based on the previous results we limit our comparison to ASHA as well as

Original: S-HB and S-BOHB.
Revised: S-HB and S-BOHB.

Original: M O B STER achieves consistently a lower regret than all three baselines.
Revised: M O B STER achieves consistently a lower regret than all three baselines.

Original: LSTM on WikiText-2.
Revised: LSTM on WikiText-2.

Original: Figure 6 left shows results for LSTM tuning on the WikiText-2 dataset.
Revised: Figure 6 left shows results for LSTM tuning on the WikiText-2 dataset.

Original: To save computations, we limited the comparison to M O B STER , ASHA, and S-BOHB.
Revised: To save computations, we limited the comparison to M O B STER , ASHA, and S-BOHB.

Original: Note that a single run (out of 5 per method) takes roughly 5 GPU days.
Revised: Note that a single run (out of 5 per method) takes roughly 5 GPU days.

Original: Training a single neural network for the full 81 epochs takes roughly 12 hours, and S-BOHB is only able to execute a single bracket in the allocated time budget.
Revised: Training a single neural network for the full 81 epochs takes roughly 12 hours, and S-BOHB is only able to execute a single bracket in the allocated time budget.

Original: Due to limited data, it cannot ﬁt a surrogate model at the highest resource level r max , which may explain its overall poor performance.
Revised: Due to limited data, it cannot ﬁt a surrogate model at the highest resource level r max , which may explain its overall poor performance.

Original: Because of its asynchronous scheduling, both
Revised: Because of its asynchronous scheduling, both

Original: M O B STER and ASHA perform substantially better.
Revised: M O B STER and ASHA perform substantially better.

Original: After around 7000 seconds, M O B STER ’s model kicks in, allowing it to approach the optimum much faster than random-based ASHA.
Revised: After around 7000 seconds, M O B STER ’s model kicks in, allowing it to approach the optimum much faster than random-based ASHA.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present a novel method, that combines asynchronous scheduling with multi-ﬁdelity BO.
Revised: We present a novel method, that combines asynchronous scheduling with multi-ﬁdelity BO.

Original: Instead of sampling new conﬁgurations at random, they are chosen based on a joint Gaussian process surrogate model over resource levels, and asynchronous scheduling is accommodated by fantasizing the outcome of pending candidates.
Revised: Instead of sampling new conﬁgurations at random, they are chosen based on a joint Gaussian process surrogate model over resource levels, and asynchronous scheduling is accommodated by fantasizing the outcome of pending candidates.

Original: On a range of challenging benchmarks, we compare our proposal to prior synchronous and asynchronous HNAS state-of-the-art.
Revised: On a range of challenging benchmarks, we compare our proposal to prior synchronous and asynchronous HNAS state-of-the-art.

Original: M O B STER signiﬁcantly outperforms synchronous BOHB and ASHA.
Revised: M O B STER signiﬁcantly outperforms synchronous BOHB and ASHA.

Original: The former is slowed down by synchronization overhead, using its parallel resources inefﬁciently at higher resource levels.
Revised: The former is slowed down by synchronization overhead, using its parallel resources inefﬁciently at higher resource levels.

Original: ASHA on the other hand does not suffer from synchronization points, but compared to M O B STER , we often ﬁnd it to plateau before solutions with performance are 
Revised: ASHA on the other hand does not suffer from synchronization points, but compared to M O B STER , we often ﬁnd it to plateau before solutions with performance are A F URTHER D ETAILS ABOUT THE M ODEL

Revised: Here, we provide some additional details in order to support the main text.

Revised: A. P OSTERIOR R EPRESENTATION . F ANTASIZING

Revised: We employ a standard representation of the Gaussian process posterior.

Revised: Say that k

Revised: ( x , x (cid:48) ) is the kernel function, µ ( x ) the mean function, and the dataset D is given by X ∈ R n × d , y ∈ R n .

Revised: For notational simplicity, x contains the resource attribute r here.

Revised: The representation maintains L and p , where

Revised: [Equation]

Revised: Here, K =

Revised: [ k ( x i , x j )]

Revised: ∈ R n × n is the kernel matrix over the training inputs X , σ 2 is the noise variance, and L is the Cholesky factor (lower triangular, positive diagonal).

Revised: Recall that we use fantasizing (Snoek et al., 2012) to deal with pending feedback.

Revised: For M fantasy samples, we now have a target matrix Y ∈ R n × M .

Revised: For any observed conﬁguration x i , the corresponding row in Y is y i 1 TM .

Revised: Now, L remains the same, and P =

Revised: L − 1 ( Y − µ ( X ) 1 TM ) .

Revised: Importantly, the dominating computation of L is independent of the number M of fantasy samples.

Revised: Predictions on test points result in M different posterior means, but common posterior variances.

Revised: In practice, the surrogate model (kernel and mean function) comes with hyperparameters ( e.g. , noise variance σ 2 , length scales and variance of kernel k X , α, β, γ, δ ), which have to be adjusted.

Revised: All in all, recomputing the posterior representation involves (a) optimizing the hyperparameters on the observed dataset only, and (b) drawing fantasy targets, then computing the representation on the extended dataset, which includes pending conﬁgurations.

Revised: Here, (a) is substantially more expensive than (b), as it involves computing the representation many times during the optimization.

Revised: In our implementation, we delay computations of the posterior representation until the next conﬁguration has to be chosen.

Revised: Our algorithm is later formalized by concise pseudocode in appendix A.4.

Revised: Note that if running tasks report metric values, they are inserted into the dataset, replacing pending conﬁgurations entered previously.

Revised: Our implementation offers a number of strategies to skip the expensive part (a) of the computation.

Revised: Here, (a) is not skipped initially, until the dataset size passes a threshold.

Revised: After that, we can skip (a) for all but every k -th update.

Revised: Another strategy is to skip (a) as long as the number of full resource datapoints ( x i , r i = r max ) does not grow.

Revised: This ensures that we do not spend more time on surrogate model updates in the multi-ﬁdelity case than in standard BO with full resource evaluations only.

Revised: A. E XPONENTIAL D ECAY S URROGATE M ODEL

Revised: The model proposed by Swersky et al.

Revised: (2014) is based on the assumption f ( x , r ) =

Revised: f

Revised: ( x ) + e − λr , where λ ∼ p ( λ ) is drawn from a Gamma prior, and the asymptote f ( x ) ( i.e. , f ( x , r ) as r → ∞ ) from a GP prior.

Revised: They end up using a zero-mean GP prior with kernel function k

Revised: (( x , r ) , ( x (cid:48) , r (cid:48) ))

Revised: =

Revised: k X ( x , x (cid:48) )

Revised: + k R ( r, r (cid:48) ) and

Revised: [Equation]

Revised: This proposal has several shortcomings.

Revised: First, f ( x , r ) → f ( x ) + 1 as r → 0 , independent of what metric is used.

Revised: Second, the random function e − λr assumption implies a non-zero mean function, and a kernel taking this into account.

Revised: A better assumption would be f ( x , r ) =

Revised: f ( x ) + γe − λr , γ > 0 , which implies a mean function µ ( x , r ) =

Revised: µ X ( x ) + γκ ( r ) and a covariance function

Revised: [Equation]

Revised: One advantage of this additive model structure is that an implied conditional independence relation can be exploited in order to speed up inference computations (Swersky et al., 2014).

Revised: However, this model implies f ( x , r ) =

Revised: f ( x ) + γ as r → 0 : the metric for no training at all depends on x in the same way as the asymptote.

Revised: A more sensible assumption would be f ( x , r )

Revised: = γ as r → 0 , since without training, predictions should be random ( e.g. , γ = 12 for binary classiﬁcation).

Revised: We present a novel surrogate model which can incorporate this assumption.

Revised: It is based on

Revised: [Equation]

Revised: Some algebra (provided below) gives

Revised: [Equation]

Revised: For δ = 0 , we recover the additive model of Eq.

Revised: 1, while for δ = 1 , we have that f ( x , r ) =

Revised: γ as r → 0 , independent of x .

Revised: This model comes with hyperparameters α, β, γ > 0 and δ ∈ [0 , 1] .

Revised: We provide an empirical analysis of this model in Section 4.1 of the main paper.

Revised: D ERIVATION OF E XPONENTIAL D ECAY S URROGATE M ODEL

Revised: Recall that our novel surrogate model k

Revised: (( x , r ) , ( x (cid:48) , r (cid:48) )) and µ ( x , r ) is based on the random function assumption (2).

Revised: Since E [ e − λr ] = κ ( r ) , it is clear that E

Revised: [ f ( x , r )]

Revised: = γκ ( r ) + µ X ( x )(1 − δκ ( r ))

Revised: =

Revised: µ X ( x ) + ( γ − δµ X

Revised: ( x )) κ ( r ) .

Revised: Moreover,

Revised: [Equation]

Revised: A. F ANTASIZING , SYNCHRONOUS VS . ASYNCHRONOUS

Revised: Here we underscore the unique challenges posed by pending evaluations and fantasizing in the asynchronous SH case, in particular, those not encountered in the synchronous case.

Revised: Speciﬁcally, in the asynchronous case, we are offered far fewer guarantees about the algorithmic state at a given point in time, not least when a decision needs to be made about what conﬁguration to propose.

Revised: This is exacerbated by the fact that concurrent state updates are now possible, making it challenging to incorporate the most up-to-date information in the decision-making process

Revised: [Equation]

Revised: In short, to combine asynchronous SH and GP-based BO with fantasizing, there are many more moving parts to contend with than in the synchronous counterpart, and an appreciable amount of care and attention toward bookkeeping is required to make it work efﬁciently.

Revised: We illustrate this with a concrete example.

Revised: Consider an instantiation of SH with r min = 1 , r max = 27 with η = 3 .

Revised: Further, assume there are a total of 5 workers.

Revised: Synchronous.

Revised: A snapshot of the algorithmic state is illustrated in Figure 7, showing the base rung of the ﬁrst bracket being ﬁlled: so far 6 conﬁgurations have been proposed and 2 have been evaluated ( x 1 , x 3 ), while 4 are still pending ( x 2 , x 4 , x 5 , x 6 ).

Revised: This means there is now 1 worker idle, namely, worker.

Revised: Since synchronous SH dictates that we need to completely evaluate 27 conﬁgurations at this rung before deciding which 9 to promote to the next rung, we still have 21 conﬁgurations to propose and evaluate.

Revised: Hence, the scheduler will assign worker 5 to evaluate a new conﬁguration at the base rung.

Revised: Proposing the new conﬁguration in the presence of these pending evaluations is straightforward.

Revised: One a) updates the GP model with all completed evaluations, and then b) sample fantasy outcomes for the pending evaluations.

Revised: Note that one need only extrapolate to the current rung level, so little to no bookkeeping is required.

Revised: Thereafter, one simply c) optimizes the acquisition function, conditioned on observed and fantasized evaluations.

Revised: w o r k e

Revised: r

Revised: w

Revised: o r k e

Revised: r w o r k e

Revised: r w o r k e

Revised: r w o r k e r 3333

Revised: · · · 26n r observed pending next

Revised: [Figure]

Revised: n r

Revised: In the edge case where the conﬁgurations necessary to ﬁll the base rung have already been proposed but some are still pending evaluation, a situation might arise where a worker becomes idle but cannot begin evaluations at the next rung until all evaluations are complete at the current rung.

Revised: An example of this is illustrated in Figure 8.

Revised: A total of 23 conﬁgurations have been evaluated and only 4 conﬁgurations are still pending ( x 22 , x 24 , x 25 , x 27 ).

Revised: This leaves 1 worker idle.

Revised: A thoughtful implementation might utilize idle workers to begin ﬁlling the base rung of subsequent brackets.

Revised: Even in this case, proposing new conﬁgurations in the presence of these pending evaluations in previous brackets is still relatively straightforward, since they are all pending at the same rung level.

Revised: Asynchronous.

Revised: By contrast, in the asynchronous paradigm, there are no guarantees concerning the rungs at which conﬁgurations will be pending evaluation.

Revised: Furthermore, in an asynchronous computing environment, it is possible that during the scheduling procedure, i.e. deciding whether promote an existing conﬁguration to the next rung or to propose a new conﬁguration to ﬁll the base rung (which would then invoke conﬁguration proposal procedure), it is possible for important events to take place concurrently, not least of which being the completion of evaluations.

Revised: To simply ignore such events would be reckless, as this could risk wasting resources on evaluating suboptimal conﬁgurations proposed with stale data.

Revised: A snapshot of an example algorithmic state is illustrated in Figure 9.

Revised: This snapshot has captured a situation where there is 1 idle worker, and the scheduler has decided to utilize it to evaluate a new conﬁguration in order to ﬁll the base rung.

Revised: Now it needs to invoke the conﬁguration proposal procedure.

Revised: Unlike before, this procedure needs to account for evaluations that have been observed, now across every rung, and deal with evaluations that are still pending, now also possible at every rung.

Revised: Added to these complexities is the fact that additional evaluations have completed since the scheduling procedure began, namely, those of conﬁgurations x 4 and x 9 , at rungs 0 and 2, respectively.

Revised: w o r k e r w o r k e

Revised: r w o r k e

Revised: r w o r k e

Revised: r w o r k e

Revised: r

Revised:          n r observed observed new pending next

Revised: To respond to these added complexities and nuances, additional care needs to be taken with algorithmic bookkeeping.

Revised: In appendix A.4, we describe the space of possible algorithmic states and formalize our asynchronous algorithm for dealing with these added complexities in concise pseudocode.

Revised: A. A LGORITHM

Revised: First, we formalize some concepts.

Revised: A task is a unit of work for evaluating the blackbox function at a conﬁguration x to some speciﬁed rung r .

Revised: Then, it is the responsibility of the workers to asynchronously execute these units of work.

Revised: Each task contains contextual information that includes hyperparameter conﬁguration x , the rung r , the target value y and a status ﬂag indicating the status of the target value.

Revised: Speciﬁcally, the status ﬂag

Revised: status ∈ { observed , observed new , pending , fantasized }

Revised: indicates whether an actual evaluation has been completed, in which case,

Revised: • if it has already been incorporated in updating the surrogate model, then status = observed , • otherwise, status = observed new .

Revised: If the evaluation is currently pending, in which case

Revised: • y may have a fantasized value, and status = fantasized , or • no value at all, and status = pending .

Revised: Note that, in general, each hyperparameter conﬁguration x will have one or more associated tasks, each with different rungs r .

Revised: Let D † denote the dataset consisting of all the observed data,

Revised: [Equation]

Revised: task .

Revised: status = observed } , and D ‡ denote the extended dataset consisting of both observed and fantasized data,

Revised: [Equation]

Revised: [Table]

Revised: Our method is summarized in algorithm 1.

Revised: end end /

Revised: * maximize acquisition

Revised: , conditioned on both observed and fantasized data D ‡ */

Revised: 39 end

Revised: B C ONFIGURATION S PACES

Revised: Table 1 shows the conﬁguration space for the MLP benchmark, Table 2 for the residual network benchmark, Table 4 for NASBench201 and Table 3 for the LSTM benchmark.

Revised: If we use an exponential notation than we optimized the corresponding hyperparameter on a logarithmic scale.

Revised: We used a onehot encoding for all categorical hyperparameters and an integer encoding for ordinal hyperparameters.

Revised: C A DDITIONAL NASB ENCH 201 E XPERIMENTS

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: v

Revised: [Figure]

Revised: a t i

Revised: [Figure]

Revised: n

Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: t

Revised: i

Revised: [Figure]