Original: M ASSIVELY P ARALLEL H YPERPARAMETER T UNING
Revised: M ASSIVELY P ARALLEL H YPERPARAMETER T UNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern machine learning models are characterized by large hyperparameter search spaces and prohibitively expensive training costs.
Revised: Modern machine learning models are characterized by large hyperparameter search spaces and prohibitively expensive training costs.

Original: For such models, we cannot afford to train candidate models sequentially and wait months before ﬁnding a suitable hyperparameter conﬁguration.
Revised: For such models, we cannot afford to train candidate models sequentially and wait months before ﬁnding a suitable hyperparameter conﬁguration.

Original: Hence, we introduce the large-scale regime for parallel hyperparameter tuning, where we need to evaluate orders of magnitude more conﬁgurations than available parallel workers in a small multiple of the wall-clock time needed to train a single model.
Revised: Hence, we introduce the large-scale regime for parallel hyperparameter tuning, where we need to evaluate orders of magnitude more conﬁgurations than available parallel workers in a small multiple of the wall-clock time needed to train a single model.

Original: We propose a novel hyperparameter tuning algorithm for this setting that exploits both parallelism and aggressive earlystopping techniques, building on the insights of the Hyperband algorithm (Li et al., 2017).
Revised: We propose a novel hyperparameter tuning algorithm for this setting that exploits both parallelism and aggressive earlystopping techniques, building on the insights of the Hyperband algorithm (Li et al., 2017).

Original: Finally, we conduct a thorough empirical study of our algorithm on several benchmarks, including large-scale experiments with up to 500 workers.
Revised: Finally, we conduct a thorough empirical study of our algorithm on several benchmarks, including large-scale experiments with up to 500 workers.

Original: Our results show that our proposed algorithm ﬁnds good hyperparameter settings nearly an order of magnitude faster than random search.
Revised: Our results show that our proposed algorithm ﬁnds good hyperparameter settings nearly an order of magnitude faster than random search.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although machine learning models have recently achieved dramatic successes in a variety of practical applications, these models are highly sensitive to internal parameters, i.e., hyperparameters .
Revised: Although machine learning models have recently achieved dramatic successes in a variety of practical applications, these models are highly sensitive to internal parameters, i.e., hyperparameters .

Original: In this modern era of machine learning, three trends motivate a new approach to hyperparameter tuning: 1.
Revised: In this modern era of machine learning, three trends motivate a new approach to hyperparameter tuning: 1.

Original: High-dimensional hyperparameter spaces .
Revised: High-dimensional hyperparameter spaces .

Original: Machine learning models are becoming increasingly complex, as evidenced by modern neural networks with dozens of hyperparameters, e.g., number of layers, width of each layer, forms of regularization, types of nodes, type of activation nonlinearities, etc.
Revised: Machine learning models are becoming increasingly complex, as evidenced by modern neural networks with dozens of hyperparameters, e.g., number of layers, width of each layer, forms of regularization, types of nodes, type of activation nonlinearities, etc.

Original: For such complex models with hyperparameters that interact in unknown ways, a practitioner is forced to evaluate potentially thousands of different hyperparameter settings.
Revised: For such complex models with hyperparameters that interact in unknown ways, a practitioner is forced to evaluate potentially thousands of different hyperparameter settings.

Original: Increasing training times .
Revised: Increasing training times .

Original: As datasets grow larger and models become more complex, training a model has become dramatically more expensive, often taking days or weeks on specialized high-performance hardware.
Revised: As datasets grow larger and models become more complex, training a model has become dramatically more expensive, often taking days or weeks on specialized high-performance hardware.

Original: For instance, Google’s neural machine translation system takes one week to train using eight GPUs in parallel (Wu et al., 2016) and Google’s LSTM acoustic model (Sak et al., 2014) also takes a few weeks to train
Revised: For instance, Google’s neural machine translation system takes one week to train using eight GPUs in parallel (Wu et al., 2016) and Google’s LSTM acoustic model (Sak et al., 2014) also takes a few weeks to train

Original: This trend is particularly onerous in the context of hyperparameter tuning, as a new model must be trained to evaluate each putative hyperparameter conﬁguration.
Revised: This trend is particularly onerous in the context of hyperparameter tuning, as a new model must be trained to evaluate each putative hyperparameter conﬁguration.

Original: Rise of parallel computing .
Revised: Rise of parallel computing .

Original: The combination of a growing number of hyperparameters and longer training time per model preclude the use of sequential hyperparameter tuning methods; we simply cannot wait months or years to ﬁnd a suitable hyperparameter setting.
Revised: The combination of a growing number of hyperparameters and longer training time per model preclude the use of sequential hyperparameter tuning methods; we simply cannot wait months or years to ﬁnd a suitable hyperparameter setting.

Original: Thankfully, the expansion of cloud computing resources has made specialized hardware like GPUs and TPUs (Dean and Hölzle, 2017) widely accessible.
Revised: Thankfully, the expansion of cloud computing resources has made specialized hardware like GPUs and TPUs (Dean and Hölzle, 2017) widely accessible.

Original: Leveraging these parallel and distributed computational resources presents an attractive path forward to combat the increasingly challenging problem of hyperparameter optimization.
Revised: Leveraging these parallel and distributed computational resources presents an attractive path forward to combat the increasingly challenging problem of hyperparameter optimization.

Original: Our goal is to design a hyperparameter tuning algorithm that can effectively leverage parallel resources.
Revised: Our goal is to design a hyperparameter tuning algorithm that can effectively leverage parallel resources.

Original: This goal seems trivial with standard methods like random search, where we could train different conﬁgurations in an embarrasingly parallel fashion.
Revised: This goal seems trivial with standard methods like random search, where we could train different conﬁgurations in an embarrasingly parallel fashion.

Original: However, in practice, the number of putative conﬁgurations often dwarfs the number of available parallel resources.
Revised: However, in practice, the number of putative conﬁgurations often dwarfs the number of available parallel resources.

Original: Hence, we aim to tackle the following problem, which we refer to as the large-scale regime for hyperparameter optimization:
Revised: Hence, we aim to tackle the following problem, which we refer to as the large-scale regime for hyperparameter optimization:

Original: Evaluate orders of magnitude more hyperparameter conﬁgurations than available parallel workers in a small multiple of the wall-clock time needed to train a single model.
Revised: Evaluate orders of magnitude more hyperparameter conﬁgurations than available parallel workers in a small multiple of the wall-clock time needed to train a single model.

Original: Existing parallel hyperparameter tuning methods generally fall into two categories: (1) embarrassingly parallel methods that use a ﬁxed sampling scheme (i.e. random search and grid search); and (2) adaptive conﬁguration selection schemes using Bayesian optimization.
Revised: Existing parallel hyperparameter tuning methods generally fall into two categories: (1) embarrassingly parallel methods that use a ﬁxed sampling scheme (i.e. random search and grid search); and (2) adaptive conﬁguration selection schemes using Bayesian optimization.

Original: Methods in the ﬁrst category are ill suited for the large-scale regime because they can only evaluate conﬁgurations on the order of number of parallel workers.
Revised: Methods in the ﬁrst category are ill suited for the large-scale regime because they can only evaluate conﬁgurations on the order of number of parallel workers.

Original: Methods in the second category are general black-box optimization routines that are complex and computationally expensive.
Revised: Methods in the second category are general black-box optimization routines that are complex and computationally expensive.

Original: In contrast, our proposed method empirically exhibits state-of-the-art performance without these added layers of complexity.
Revised: In contrast, our proposed method empirically exhibits state-of-the-art performance without these added layers of complexity.

Original: Our main contribution in this work is a practical hyperparameter tuning algorithm for the largescale regime that exploits parallelism and aggressive early-stopping techniques.
Revised: Our main contribution in this work is a practical hyperparameter tuning algorithm for the largescale regime that exploits parallelism and aggressive early-stopping techniques.

Original: We build upon the
Revised: We build upon the

Original: Hyperband algorithm (Li et al., 2017), adapting it for the parallel setting and improving it in two signiﬁcant ways, as described in Section 3.1.
Revised: Hyperband algorithm (Li et al., 2017), adapting it for the parallel setting and improving it in two signiﬁcant ways, as described in Section 3.1.

Original: Furthermore, we conduct a thorough empirical study of our algorithm on multiple benchmarks, demonstrating success in the large-scale regime on tasks with up to 500 workers.
Revised: Furthermore, we conduct a thorough empirical study of our algorithm on multiple benchmarks, demonstrating success in the large-scale regime on tasks with up to 500 workers.

Original: Lastly, we present a small case study on the efﬁcient use of parallel computing resources for hyperparameter tuning.
Revised: Lastly, we present a small case study on the efﬁcient use of parallel computing resources for hyperparameter tuning.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We review prior work on hyperparameter tuning in both the sequential and parallel settings.
Revised: We review prior work on hyperparameter tuning in both the sequential and parallel settings.

Original: While there are many sequential algorithms that have associated implementations for the parallel setting, we categorize algorithms not speciﬁcally designed for the parallel setting as sequential methods.
Revised: While there are many sequential algorithms that have associated implementations for the parallel setting, we categorize algorithms not speciﬁcally designed for the parallel setting as sequential methods.

Original: 2. S EQUENTIAL M ETHODS
Revised: 2. S EQUENTIAL M ETHODS

Original: Existing hyperparameter tuning methods attempt to speed up the search for a good conﬁguration by either adaptively selecting conﬁgurations or adaptively evaluating conﬁgurations.
Revised: Existing hyperparameter tuning methods attempt to speed up the search for a good conﬁguration by either adaptively selecting conﬁgurations or adaptively evaluating conﬁgurations.

Original: Adaptive conﬁguration selection approaches attempt to identify promising regions of the hyperparameter search space from which to sample new conﬁgurations to try.
Revised: Adaptive conﬁguration selection approaches attempt to identify promising regions of the hyperparameter search space from which to sample new conﬁgurations to evaluate.

Original: Well-established conﬁguration selection approaches include SMAC, Spearmint, TPE, and GP-UCB (Hutter et al., 2011;Snoek et al., 2012;
Revised: Well-established conﬁguration selection approaches include SMAC, Spearmint, TPE, and GP-UCB (Hutter et al., 2011;

Original:  Bergstra et al., 2011; Srinivas et al., 2010), which have been shown to outperform random search in empirical studies.
Revised: Snoek et al., 2012; Bergstra et al., 2011; Srinivas et al., 2010), which have been shown to outperform random search in empirical studies.

Original: However, by relying on previous observations to inform which conﬁguration to evaluate next, these algorithms are inherently sequential and thus not suitable for the large-scale regime, where the number of updates to the posterior is limited.
Revised: However, by relying on previous observations to inform which conﬁguration to evaluate next, these algorithms are inherently sequential and thus not suitable for the large-scale regime, where the number of updates to the posterior is limited.

Original: Adaptive conﬁguration evaluation approaches attempt to early-stop poor conﬁgurations and allocate more training “resources” to promising conﬁgurations.
Revised: Adaptive conﬁguration evaluation approaches attempt to early-stop poor conﬁgurations and allocate more training “resources” to promising conﬁgurations.

Original: The use of resources can, for example, be measured as the of number of training iterations, cpu or wall-clock time, or the number of training examples.
Revised: The use of resources can, for example, be measured as the of number of training iterations, cpu or wall-clock time, or the number of training examples.

Original: Previous methods like György and Kocsis (2011); Agarwal et al.
Revised: Previous methods like György and Kocsis (2011); Agarwal et al.

Original: (2011); Sabharwal et al.
Revised: (2011); Sabharwal et al.

Original: (2016) provide theoretical guarantees under strong assumptions on the convergence behavior of intermediate losses.
Revised: (2016) provide theoretical guarantees under strong assumptions on the convergence behavior of intermediate losses.

Original: Krueger et al.
Revised: Krueger et al.

Original: (2015) relies on a heuristic early-stopping rule based on sequential analysis to terminate poor conﬁgurations.
Revised: (2015) relies on a heuristic early-stopping rule based on sequential analysis to terminate poor conﬁgurations.

Original: Hybrid Bayesian optimization approaches that combine adaptive conﬁguration selection and evaluation include Swersky et al.
Revised: Hybrid Bayesian optimization approaches that combine adaptive conﬁguration selection and evaluation include Swersky et al.

Original: (2013; 2014); Domhan et al.
Revised: (2013; 2014); Domhan et al.

Original: ; Klein et al.
Revised: ; Klein et al.

Original: (2017a).
Revised: (2017a).

Original: However, these methods scale poorly with number of conﬁgurations due to the cubic complexity of ﬁtting a Gaussian process (Snoek et al., 2015).
Revised: However, these methods scale poorly with number of conﬁgurations due to the cubic complexity of ﬁtting a Gaussian process (Snoek et al., 2015).

Original: Nonetheless, Klein et al. (2017a) recently introduced Fabolas and demonstrated state-of-the-art performance on several hyperparameter tuning tasks; we will provide a detailed comparison to Fabolas in Section 4.
Revised: Nonetheless, Klein et al. (2017a) recently introduced Fabolas and demonstrated state-of-the-art performance on several hyperparameter tuning tasks; we will provide a detailed comparison to Fabolas in Section 4.

Original: Lázaro-Gredilla et al.
Revised: Lázaro-Gredilla et al.

Original: ; Wilson et al.
Revised: ; Wilson et al.

Original: (2015) improve the scaling proﬁles of Gaussian processes by imposing additional structure on the model.
Revised: (2015) improve the scaling proﬁles of Gaussian processes by imposing additional structure on the model.

Original: Additionally, Bayesian deep neural networks, which scale linearly as a function of the number of observations, have been used to model the posterior (Snoek et al., 2015; Springenberg et al., 2016).
Revised: Additionally, Bayesian deep neural networks, which scale linearly as a function of the number of observations, have been used to model the posterior (Snoek et al., 2015; Springenberg et al., 2016).

Original: However, the improved computational tractibility comes at a price; for a ﬁxed number of evaluations, these methods generally underperform vanilla
Revised: However, the improved computational tractibility comes at a price; for a ﬁxed number of evaluations, these methods generally underperform vanilla

Original: Gaussian processes from an accuracy perspective.
Revised: Gaussian processes from an accuracy perspective.

Original: Li et al.
Revised: Li et al.

Original: (2016) proposed Hyperband, an adaptive conﬁguration evaluation approach which does not have the aforementioned drawbacks and showed that it achieves state-of-the-art performance on several empirical tasks.
Revised: (2016) proposed Hyperband, an adaptive conﬁguration evaluation approach which does not have the aforementioned drawbacks and showed that it achieves state-of-the-art performance on several empirical tasks.

Original: At ﬁrst glance, the Hyperband algorithm appears trivial to parallelize since all evaluated conﬁgurations are drawn independently.
Revised: At ﬁrst glance, the Hyperband algorithm appears trivial to parallelize since all evaluated conﬁgurations are drawn independently.

Original: However, the algorithm eliminates conﬁgurations at an exponential rate, severely limiting the degree of parallelism in later elimination rounds.
Revised: However, the algorithm eliminates conﬁgurations at an exponential rate, severely limiting the degree of parallelism in later elimination rounds.

Original: Additionally, these eliminations are performed at ﬁxed intervals, making the algorithm susceptible to stragglers.
Revised: Additionally, these eliminations are performed at ﬁxed intervals, making the algorithm susceptible to stragglers.

Original: Moreover, the divide-and-conquer strategy of running an instance of the algorithm on each machine does not reduce the latency of receiving an output from the algorithm.
Revised: Moreover, the divide-and-conquer strategy of running an instance of the algorithm on each machine does not reduce the latency of receiving an output from the algorithm.

Original: Our proposed algorithm adapts Hyperband for the large-scale regime by addressing these issues.
Revised: Our proposed algorithm adapts Hyperband for the large-scale regime by addressing these issues.

Original: Note that Hyperband uses randomly sampled hyperparameter conﬁgurations by default, but can be combined with any sampling mechanism.
Revised: Note that Hyperband uses randomly sampled hyperparameter conﬁgurations by default, but can be combined with any sampling mechanism.

Original: For example, Klein et al.
Revised: For example, Klein et al.

Original: (2017b)
Revised: (2017b)

Original: combined Bayesian neural networks with Hyperband by ﬁrst training a Bayesian neural network to predict learning curves and then using the model to select promising conﬁgurations to use as inputs to Hyperband.
Revised: combined Bayesian neural networks with Hyperband by ﬁrst training a Bayesian neural network to predict learning curves and then using the model to select promising conﬁgurations to use as inputs to Hyperband.

Original: 2. P ARALLEL M ETHODS
Revised: 2. P ARALLEL M ETHODS

Original: Grid search and random search are popular in practice because they are trivial to parallelize.
Revised: Grid search and random search are popular in practice because they are trivial to parallelize.

Original: Nonetheless, distributed versions of SMAC, Spearmint, and TPE (Hutter et al., 2011; Snoek et al., 2012;
Revised: Nonetheless, distributed versions of SMAC, Spearmint, and TPE (Hutter et al., 2011; Snoek et al., 2012;

Original: Bergstra et al., 2011) are sometimes used.
Revised: Bergstra et al., 2011) are sometimes used.

Original: These methods either act greedily (Spearmint) or rely on randomness to provide additional conﬁgurations to evaluate (SMAC, TPE).
Revised: These methods either act greedily (Spearmint) or rely on randomness to provide additional conﬁgurations to evaluate (SMAC, TPE).

Original: More sophisticated batch Bayesian optimization methods attempt to construct a batch of conﬁgurations by optimizing a joint objective (Shah and Ghahramani, 2015; González et al., 2016; Wu and Frazier, 2016).
Revised: More sophisticated batch Bayesian optimization methods attempt to construct a batch of conﬁgurations by optimizing a joint objective (Shah and Ghahramani, 2015; González et al., 2016; Wu and Frazier, 2016).

Original: However, these methods fair poorly in the large-scale regime since all conﬁgurations are trained to completion.
Revised: However, these methods fair poorly in the large-scale regime since all conﬁgurations are trained to completion.

Original: As for hybrid adaptive conﬁguration selection and evaluation methods, the hybrid sequential methods mentioned in Section 2.1 can be parallelized using constant liar (CL) type heuristics (Ginsbourger et al., 2010; González et al., 2016).
Revised: As for hybrid adaptive conﬁguration selection and evaluation methods, the hybrid sequential methods mentioned in Section 2.1 can be parallelized using constant liar (CL) type heuristics (Ginsbourger et al., 2010; González et al., 2016).

Original: These heuristics ﬁll in the objective value of pending conﬁgurations with a constant before proposing new conﬁgurations to try.
Revised: These heuristics ﬁll in the objective value of pending conﬁgurations with a constant before proposing new conﬁgurations to try.

Original: Note that assuming the Bayesian model is correct, algorithms parallelized using the CL heuristic are inferior to the sequential version, which uses a more accurate posterior, updated with true objective values, to propose new points.
Revised: Note that assuming the Bayesian model is correct, algorithms parallelized using the CL heuristic are inferior to the sequential version, which uses a more accurate posterior, updated with true objective values, to propose new points.

Original: Similar to the sequential method in Domhan et al.
Revised: Similar to the sequential method in Domhan et al.

Original: (2015), Rasley et al.
Revised: (2015), Rasley et al.

Original: (2017) uses parametric learning curves to perform early-stopping but adds a scheduling algorithm to effciently use parallel resources.
Revised: (2017) uses parametric learning curves to perform early-stopping but adds a scheduling algorithm to effciently use parallel resources.

Original: More recently Golovin et al.
Revised: More recently Golovin et al.

Original: (2017) introduced Vizier, a black-box optimization service with support for multiple tuning methods and early stopping options.
Revised: (2017) introduced Vizier, a black-box optimization service with support for multiple tuning methods and early stopping options.

Original: For succinctness, we will refer to Vizier’s default Bayesian optimization algorithm as “Vizier” with the understanding that it is simply one of methods available on the Vizier platform.
Revised: For succinctness, we will refer to Vizier’s default Bayesian optimization algorithm as “Vizier” with the understanding that it is simply one of methods available on the Vizier platform.

Original: We compare to Vizier in Section 4.3.
Revised: We compare to Vizier in Section 4.3.

Original: Another class of related methods focus on learning neural network architectures for a given task in a large-scale setting (Zoph and Le, 2017; Real et al., 2017).
Revised: Another class of related methods focus on learning neural network architectures for a given task in a large-scale setting (Zoph and Le, 2017; Real et al., 2017).

Original: Real et al.
Revised: Real et al.

Original: (2017) uses evolutionary algorithms to design neural network architectures for image classiﬁers.
Revised: (2017) uses evolutionary algorithms to design neural network architectures for image classiﬁers.

Original: A different approach is taken in Zoph and Le (2017), which uses reinforcement learning to learn a policy used to generates neural network architectures that perform well on a given task.
Revised: A different approach is taken in Zoph and Le (2017), which uses reinforcement learning to learn a policy used to generates neural network architectures that perform well on a given task.

Original: While these method match or even outperform human designed architectures, they are computationally expensive and require task dependent speciﬁcations on how architectures can be modiﬁed to arrive at new architectures.
Revised: While these method match or even outperform human designed architectures, they are computationally expensive and require task dependent speciﬁcations on how architectures can be modiﬁed to arrive at new architectures.

Original: We note that Zoph and Le (2017) achieves state-of-the-art performance on the Penn Treebank language modeling task, outperforming previous methods by a large margin; we consider a similar tuning task in Section 4.3.
Revised: We note that Zoph and Le (2017) achieves state-of-the-art performance on the Penn Treebank language modeling task, outperforming previous methods by a large margin; we consider a similar tuning task in Section 4.3.

Original: However, they explore a much larger search space and the proposed architecture still requires hyperparameter tuning of learning parameters, weight decay, and batchnorm epsilon.
Revised: However, they explore a much larger search space and the proposed architecture still requires hyperparameter tuning of learning parameters, weight decay, and batchnorm epsilon.

Original:  A LGORITHM
Revised:  A LGORITHM

Original: Hyperband (Li et al., 2017) calls the Successive Halving algorithm
Revised: Hyperband (Li et al., 2017) calls the Successive Halving algorithm

Original: (Karnin et al., 2013; Jamieson and
Revised: (Karnin et al., 2013; Jamieson and

Original: Talwalkar, 2015), with different early stopping rates as a subroutine.
Revised: Talwalkar, 2015), with different early stopping rates as a subroutine.

Original: Hence, we focus on adapting the synchronous Successive Halving algorithm (SHA) for the parallel setting.
Revised: Hence, we focus on adapting the synchronous Successive Halving algorithm (SHA) for the parallel setting.

Original: In Algorithm 1, we introduce a “conservative” asynchronous algorithm, which has the property of advancing all conﬁgurations that would have been advanced by synchronous SHA.
Revised: In Algorithm 1, we introduce a “conservative” asynchronous algorithm, which has the property of advancing all conﬁgurations that would have been advanced by synchronous SHA.

Original: Thus, the ﬁnal selected conﬁguration is guaranteed to be at least as good as the one returned by the synchronous algorithm, at the expense of a potentially larger total computational cost.
Revised: Thus, the ﬁnal selected conﬁguration is guaranteed to be at least as good as the one returned by the synchronous algorithm, at the expense of a potentially larger total computational cost.

Original: Algorithm 1 requires as input a minimum resource r , the reduction factor η > 1 , and the minimum early stopping rate s .
Revised: Algorithm 1 requires as input a minimum resource r , the reduction factor η > 1 , and the minimum early stopping rate s .

Original: We will refer to trials of SHA with different values of s as brackets and, within a bracket, we will refer to each round of promotion as a rung with the base rung numbered 0 and
Revised: We will refer to trials of SHA with different values of s as brackets and, within a bracket, we will refer to each round of promotion as a rung with the base rung numbered 0 and

Original: Algorithm
Revised: Algorithm

Original: Procedure get_job() //
Revised: Procedure get_job() //

Original: A configuration in a given rung is “promotable” if its validation loss places it in the top 1 /η fraction of completed configurations in its rung and it has not already been promoted.
Revised: A configuration in a given rung is “promotable” if its validation loss places it in the top 1 /η fraction of completed configurations in its rung and it has not already been promoted.

Original: Let θ k be a promotable conﬁguration θ in the highest rung k (i.e. trained the longest).
Revised: Let θ k be a promotable conﬁguration θ in the highest rung k (i.e. trained the longest).

Original: if θ k exists then Promote θ k to rung k + 1 .
Revised: if θ k exists then Promote θ k to rung k + 1 .

Original: return else end increasing.
Revised: return else end increasing.

Original: For a given η , only the top 1 /η fraction of conﬁgurations are promoted to the next rung.
Revised: For a given η , only the top 1 /η fraction of conﬁgurations are promoted to the next rung.

Original: For a given s , a minimum resource of rη s will be allocated to each conﬁguration.
Revised: For a given s , a minimum resource of rη s will be allocated to each conﬁguration.

Original: Hence, lower s corresponds to more aggressive early-stopping, with s = 0 prescribing a minimum resource of r .
Revised: Hence, lower s corresponds to more aggressive early-stopping, with s = 0 prescribing a minimum resource of r .

Original: Another component of the algorithm is the run_then_return_val_loss ( θ, r ) subroutine, which returns the validation loss after training the model with the hyperparameter setting θ and resource allocation r .
Revised: Another component of the algorithm is the run_then_return_val_loss ( θ, r ) subroutine, which returns the validation loss after training the model with the hyperparameter setting θ and resource allocation r .

Original: The subroutine is asynchronous and the code execution of async_SHA continues after the job is passed to the worker.
Revised: The subroutine is asynchronous and the code execution of async_SHA continues after the job is passed to the worker.

Original: Asynchronous SHA promotes conﬁgurations to the next rung whenever possible instead of waiting for a rung to complete before proceeding to the next rung.
Revised: Asynchronous SHA promotes conﬁgurations to the next rung whenever possible instead of waiting for a rung to complete before proceeding to the next rung.

Original: Additionally, if no promotions are possible, the asynchronous algorithm simply adds a conﬁguration to the base rung, so that more conﬁgurations can be promoted to the upper rungs.
Revised: Additionally, if no promotions are possible, the asynchronous algorithm simply adds a conﬁguration to the base rung, so that more conﬁgurations can be promoted to the upper rungs.

Original: This promotion scheme is laid out in the get_job subroutine.
Revised: This promotion scheme is laid out in the get_job subroutine.

Original: From the promotion scheme, it is clear that Algorithm 1 is asynchronous because a new job can be requested at any time and does not require waiting for other jobs to complete.
Revised: From the promotion scheme, it is clear that Algorithm 1 is asynchronous because a new job can be requested at any time and does not require waiting for other jobs to complete.

Original: Thus, no training job or advancement decision is blocked due to stragglers.
Revised: Thus, no training job or advancement decision is blocked due to stragglers.

Original: Of course, this also means we may prematurely advance some conﬁgurations, but the probability of incorrect promotions decreases as more conﬁgurations are evaluated.
Revised: Of course, this also means we may prematurely advance some conﬁgurations, but the probability of incorrect promotions decreases as more conﬁgurations are evaluated.

Original: Note there is no bound on the number of rungs in a bracket and, therefore, no bound on the maximum resource that can be allocated to a given conﬁguration.
Revised: Note there is no bound on the number of rungs in a bracket and, therefore, no bound on the maximum resource that can be allocated to a given conﬁguration.

Original: However, an upper bound on the maximum resource is often desired (i.e. limit the number of epochs of stochastic gradient descent when training a neural network to prevent overﬁtting (Hardt et al., 2016)).
Revised: However, an upper bound on the maximum resource is often desired (i.e. limit the number of epochs of stochastic gradient descent when training a neural network to prevent overﬁtting (Hardt et al., 2016)).

Original: Algorithm 1 can be easily adapted to enforce a maximum resource per conﬁguration by limiting the number of rungs.
Revised: Algorithm 1 can be easily adapted to enforce a maximum resource per conﬁguration by limiting the number of rungs.

Original: Speciﬁcally, for a given maximum resource R and bracket s , limit the number of rungs to log η ( R/r ) − s + 1 .
Revised: Speciﬁcally, for a given maximum resource R and bracket s , limit the number of rungs to log η ( R/r ) − s + 1 .

Original: Finally, since Hyperband simply runs multiple brackets of SHA, we can parallelize Hyperband by taking the minimum across multiple brackets of asynchronous SHA.
Revised: Finally, since Hyperband simply runs multiple brackets of SHA, we can parallelize Hyperband by taking the minimum across multiple brackets of asynchronous SHA.

Original: In practice, if prior knowledge is available about the task, a user can run a speciﬁc bracket with a target early-stopping rate instead of running all possible brackets as is done by Hyperband.
Revised: In practice, if prior knowledge is available about the task, a user can run a speciﬁc bracket with a target early-stopping rate instead of running all possible brackets as is done by Hyperband.

Original: In our experience, early-stopping is helpful for most neural network type tasks and brackets with minimum resources of R/ 64 or even R/are reasonable for a given maximum resource per conﬁguration of R .
Revised: In our experience, early-stopping is helpful for most neural network type tasks and brackets with minimum resources of R/ 64 or even R/are reasonable for a given maximum resource per conﬁguration of R .

Original: 3. A LGORITHM D ISCUSSION
Revised: 3. A LGORITHM D ISCUSSION

Original: Asynchronous SHA improves upon the synchronous version of the algorithm in two important ways
Revised: Asynchronous SHA improves upon the synchronous version of the algorithm in two important ways

Original: Li et al.
Revised: Li et al.

Original: (2017) has two versions of Hyperband, one for the ﬁnite horizon (bounded resource per conﬁguration) and another for the inﬁnite horizon setting.
Revised: (2017) has two versions of Hyperband, one for the ﬁnite horizon (bounded resource per conﬁguration) and another for the inﬁnite horizon setting.

Original: Asynchronous SHA consolidates the two settings into one algorithm.
Revised: Asynchronous SHA consolidates the two settings into one algorithm.

Original: For the ﬁnite horizon setting, simply limit the number of rungs per bracket.
Revised: For the ﬁnite horizon setting, simply limit the number of rungs per bracket.

Original: For the inﬁnite horizon setting, the maximum resource per conﬁguration increases naturally as conﬁgurations are promoted to higher rungs.
Revised: For the inﬁnite horizon setting, the maximum resource per conﬁguration increases naturally as conﬁgurations are promoted to higher rungs.

Original: In contrast, the synchronous version of the algorithm relies on the doubling trick and reruns brackets with larger budgets to increase the maximum resource.
Revised: In contrast, the synchronous version of the algorithm relies on the doubling trick and reruns brackets with larger budgets to increase the maximum resource.

Original: The algorithm analyzed in Li et al.
Revised: The algorithm analyzed in Li et al.

Original: (2017) differs from the algorithm used in practice.
Revised: (2017) differs from the algorithm used in practice.

Original: The theoretical analysis relies on the budget per bracket increasing so that the number of evaluated conﬁgurations n will eventually be large enough to ﬁnd a sufﬁciently good conﬁguration.
Revised: The theoretical analysis relies on the budget per bracket increasing so that the number of evaluated conﬁgurations n will eventually be large enough to ﬁnd a sufﬁciently good conﬁguration.

Original: In contrast, the algorithm recommended for use in practice runs a bracket with a ﬁxed budget multiple times so that the waiting time for a conﬁguration trained on the maximum resource does not double with each outer loop.
Revised: In contrast, the algorithm recommended for use in practice runs a bracket with a ﬁxed budget multiple times so that the waiting time for a conﬁguration trained on the maximum resource does not double with each outer loop.

Original: Asynchronous SHA naturally grows the number of conﬁgurations and trains a conﬁguration on the maximum resource whenever possible, so we do not need to rerun brackets or wait exponentially longer for conﬁgurations trained to completion.
Revised: Asynchronous SHA naturally grows the number of conﬁgurations and trains a conﬁguration on the maximum resource whenever possible, so we do not need to rerun brackets or wait exponentially longer for conﬁgurations trained to completion.

Original: A complete theoretical treatment of asynchronous SHA is out of the scope of our paper and reserved for future work.
Revised: A complete theoretical treatment of asynchronous SHA is out of the scope of our paper and reserved for future work.

Original: However, we conjecture that the theoretical guarantees for synchronous SHA will also apply for our proposed algorithm.
Revised: However, we conjecture that the theoretical guarantees for synchronous SHA will also apply for our proposed algorithm.

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: In this section, we ﬁrst present experiments in the sequential setting to provide a point of comparison for asynchronous SHA and to compare Hyperband with a recent Bayesian optimization method combining adaptive conﬁguration selection and evaluation method.
Revised: In this section, we ﬁrst present experiments in the sequential setting to provide a point of comparison for asynchronous SHA and to compare Hyperband with a recent Bayesian optimization method combining adaptive conﬁguration selection and evaluation method.

Original: Next, we show that asynchronous
Revised: Next, we show that asynchronous

Original: SHA achieves comparable performance as the serial version of the algorithm and is not hurt too much by incorrect promotions.
Revised: SHA achieves comparable performance as the serial version of the algorithm and is not hurt too much by incorrect promotions.

Original: Finally, we present large-scale experiments using asynchronous SHA on a language modeling task and a speech acoustic modeling task.
Revised: Finally, we present large-scale experiments using asynchronous SHA on a language modeling task and a speech acoustic modeling task.

Original: 4. C OMPARISON WITH F ABOLAS IN S EQUENTIAL S ETTING
Revised: 4. C OMPARISON WITH F ABOLAS IN S EQUENTIAL S ETTING

Original: Klein et al.
Revised: Klein et al.

Original: (2017a) showed that Fabolas can be over an order of magnitude faster than existing
Revised: (2017a) showed that Fabolas can be over an order of magnitude faster than existing

Original: Bayesian optimization methods.
Revised: Bayesian optimization methods.

Original: Additionally, the empirical studies presented in Klein et al.
Revised: Additionally, the empirical studies presented in Klein et al.

Original: (2017a) suggest that Fabolas is faster than Hyperband at ﬁnding a good conﬁguration.
Revised: (2017a) suggest that Fabolas is faster than Hyperband at ﬁnding a good conﬁguration.

Original: We conducted our own experiments to compare Fabolas with Hyperband on the following tasks: (1) tuning an SVM using the same search space as Klein et al.
Revised: We conducted our own experiments to compare Fabolas with Hyperband on the following tasks: (1) tuning an SVM using the same search space as Klein et al.

Original: (2017a)
Revised: (2017a)

Original: , (2) tuning a convolutional neural network (CNN) with the same search space as Li et al.
Revised: , (2) tuning a convolutional neural network (CNN) with the same search space as Li et al.

Original: (2017) on CIFAR-10 (Krizhevsky, 2009), and (3) tuning a
Revised: (2017) on CIFAR-10 (Krizhevsky, 2009), and (3) tuning a

Original: CNN on SVHN (Netzer et al., 2011) with varying number of layers, batch size, and number of ﬁlters (see Appendix A.1 for more details).
Revised: CNN on SVHN (Netzer et al., 2011) with varying number of layers, batch size, and number of ﬁlters (see Appendix A.1 for more details).

Original: In the case of the SVM task, the allocated resource is number of training datapoints, while for the CNN tasks, the allocated resource is the number of training iterations.
Revised: In the case of the SVM task, the allocated resource is number of training datapoints, while for the CNN tasks, the allocated resource is the number of training iterations.

Original: We ﬁnd that after adjusting for the accounting scheme used to record the performance of Hyperband, Hyperband actually outperforms Fabolas.
Revised: We ﬁnd that after adjusting for the accounting scheme used to record the performance of Hyperband, Hyperband actually outperforms Fabolas.

Original: Additionally, as mentioned in Section 2.1, while Fabolas can be parallelized using the constant liar heuristic, the distributed version of the algorithm would at best achieve similar performance to the sequential algorithm.
Revised: Additionally, as mentioned in Section 2.1, while Fabolas can be parallelized using the constant liar heuristic, the distributed version of the algorithm would at best achieve similar performance to the sequential algorithm.

Original: Hence, we will focus on evaluating asynchronous SHA in the next two sections.
Revised: Hence, we will focus on evaluating asynchronous SHA in the next two sections.

Original: For these sequential experiments, we use the same evaluation framework as Klein et al.
Revised: For these sequential experiments, we use the same evaluation framework as Klein et al.

Original: (2017a), where the best conﬁguration, also known as the incumbent , is recorded through time and the test error is calculated in an ofﬂine validation step.
Revised: (2017a), where the best conﬁguration, also known as the incumbent , is recorded through time and the test error is calculated in an ofﬂine validation step.

Original: Following Klein et al.
Revised: Following Klein et al.

Original: (2017a), the incumbent for
Revised: (2017a), the incumbent for

Original: Hyperband is taken to be the conﬁguration with the lowest validation loss and the incumbent for
Revised: Hyperband is taken to be the conﬁguration with the lowest validation loss and the incumbent for

Original: Fabolas is the conﬁguration with the lowest predicted validation loss on the full dataset.
Revised: Fabolas is the conﬁguration with the lowest predicted validation loss on the full dataset.

Original: We use this framework to reproduce the results for Hyperband, Fabolas, and random search on the SVM task from Klein et al.
Revised: We use this framework to reproduce the results for Hyperband, Fabolas, and random search on the SVM task from Klein et al.

Original: (2017a)
Revised: (2017a)

Original: and apply the same methodology to the two CNN tuning tasks.
Revised: and apply the same methodology to the two CNN tuning tasks.

Original: We will refer to Hyperband as evaluated in Klein et al.
Revised: We will refer to Hyperband as evaluated in Klein et al.

Original: (2017a)
Revised: (2017a)

Original: as “Hyperband (by bracket),” since the incumbent is recorded after the completion of each SHA bracket.
Revised: as “Hyperband (by bracket),” since the incumbent is recorded after the completion of each SHA bracket.

Original: We also evaluate the performance
Revised: We also evaluate the performance

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: of Hyperband
Revised: of Hyperband

Original: when recording the incumbent after the completion of each rung of SHA instead of each bracket to make use of intermediate validation losses; we will refer to Hyperband using this accounting scheme as “Hyperband (by rung).”
Revised: when recording the incumbent after the completion of each rung of SHA instead of each bracket to make use of intermediate validation losses; we will refer to Hyperband using this accounting scheme as “Hyperband (by rung).”

Original: In Figure 1, we show the performance of Hyperband, Fabolas, and random search.
Revised: In Figure 1, we show the performance of Hyperband, Fabolas, and random search.

Original: Our results show that Hyperband (by rung) is competitive with Fabolas at ﬁnding a good conﬁguration and will often ﬁnd a better conﬁguration than Fabolas with less variance.
Revised: Our results show that Hyperband (by rung) is competitive with Fabolas at ﬁnding a good conﬁguration and will often ﬁnd a better conﬁguration than Fabolas with less variance.

Original: 4. L IMITED -S CALE D ISTRIBUTED E XPERIMENTS
Revised: 4. L IMITED -S CALE D ISTRIBUTED E XPERIMENTS

Original: We evaluate asynchronous SHA on tasks (2) and (3) described in Section 4.1 to see how the asynchronous algorithm compares.
Revised: We evaluate asynchronous SHA on tasks (2) and (3) described in Section 4.1 to see how the asynchronous algorithm compares.

Original: For each experiment, we ran asynchronous SHA with 25 workers for 3 × the time needed to train a single model.

Original: For both tasks, we use asynchronous SHA with a bounded maximum resource (number of iterations) per conﬁguration R and η = 4 .
Revised: For both tasks, we use asynchronous SHA with a bounded maximum resource (number of iterations) per conﬁguration R and η = 4 .

Original: The minimum resource r is set to Rη − 4 for 5 possible brackets of asynchronous SHA.
Revised: The minimum resource r is set to Rη − 4 for 5 possible brackets.

Revised: For each experiment, asynchronous SHA is run with 25 workers for 3 × the time needed to train a single model.

Original: Figure 2 shows the average test error of brackets of asynchronous SHA with different early-stopping rates.
Revised: Figure 2 shows the average test error of brackets of asynchronous SHA with different early-stopping rates.

Original: For the CIFAR-10 task, it took Hyperband over 200 minutes in the sequential setting to ﬁnd a good conﬁguration (Fig 1), while it took Bracket 0 slightly over 40 minutes in the parallel setting.
Revised: For the CIFAR-10 task, it took Hyperband over 200 minutes in the sequential setting to ﬁnd a good conﬁguration (Figure 1), while it took Bracket 0 slightly over 40 minutes in the parallel setting.

Original: For the SVHN task, it took Hyperband 300 minutes to ﬁnd a conﬁguration with a test error below 5% (Fig 1), whereas Bracket 0 and Bracket 1 in the parallel setting found one in less than 40 minutes.
Revised: For the SVHN task, it took Hyperband 300 minutes to ﬁnd a conﬁguration with a test error below 5% (Figure 1), whereas Bracket 0 and Bracket 1 in the parallel setting found one in less than 40 minutes.

Original: On both tasks, asynchronous SHA succeeded in the large-scale regime; Bracket 0 evaluated over one thousand conﬁgurations in 40 minutes with just 25 workers and found a good conﬁguration in approximately the time needed to train a single model.
Revised: On both tasks, asynchronous SHA succeeded in the large-scale regime; Bracket 0 evaluated over one thousand conﬁgurations in 40 minutes with just 25 workers and found a good conﬁguration in approximately the time needed to train a single model.

Original: We note that the speedup is less than the number of machines due to the moderate difﬁculty of the problem.
Revised: We note that the speedup is less than the number of machines due to the moderate difﬁculty of the problem.

Original: We explore harder tasks with a much higher degree of parallelism in the next section.
Revised: We explore harder tasks with a much higher degree of parallelism in the next section.

Original: 4. L ARGE -S CALE E XPERIMENTS
Revised: 4. L ARGE -S CALE E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: We conduct empirical studies on two large-scale benchmarks that take on the run.
Revised: We conduct empirical studies on two large-scale benchmarks that take on the order of weeks to run

Original: Both benchmarks show that asynchronous SHA is well suited for the large-scale regime and can successfully ﬁnd good hyperparameter settings under resource and time constraints.
Revised: Both benchmarks show that asynchronous SHA is well suited for the large-scale regime and can successfully ﬁnd good hyperparameter settings under resource and time constraints.

Original: We consider brackets of asynchronous SHA with η = 4 and a bounded resource per conﬁguration R , with time ( R ) representing the average time to train a single model.
Revised: We consider brackets of asynchronous SHA with η = 4 and a bounded resource per conﬁguration R , with time ( R ) representing the average time to train a single model.

Original: We compare to Vizier with and without the performance curve early stopping rule (Golovin et al., 2017).
Revised: We compare to Vizier with and without the performance curve early stopping rule (Golovin et al., 2017).

Original: For both tasks, the resources allocated to different conﬁgurations is the number of training records, which translates into the number of training iterations after accounting for certain hyperparameters.
Revised: For both tasks, the resources allocated to different conﬁgurations is the number of training records, which translates into the number of training iterations after accounting for certain hyperparameters.

Original: Finally, due to the computational burden of these experiments, we perform only a single trial of each tuner.
Revised: Finally, due to the computational burden of these experiments, we perform only a single trial of each tuner.

Original: Our results on these tasks are qualitatively similar to the results in Section 4.2, which are averaged over multiple trials.
Revised: Our results on these tasks are qualitatively similar to the results in Section 4.2, which are averaged over multiple trials.

Original: For the ﬁrst task, we tune a one layer LSTM language model for next word prediction on the Penn
Revised: For the ﬁrst task, we tune a one layer LSTM language model for next word prediction on the Penn

Original: Treebank (PTB) dataset (Marcus et al., 1993).
Revised: Treebank (PTB) dataset (Marcus et al., 1993).

Original: The benchmark has 10 hyperparameters that control both the model architecture as well as the optimization routine (see Appendix A.2).
Revised: The benchmark has 10 hyperparameters that control both the model architecture as well as the optimization routine (see Appendix A.2).

Original: Each tuner is given 500 workers and 10 × time ( R ) .
Revised: Each tuner is given 500 workers and 10 × time ( R ) .

Original: We set the minimum resource to r =
Revised: We set the minimum resource to r =

Original: Rη − 4 , leading to 5 possible brackets of SHA.
Revised: Rη − 4 , leading to 5 possible brackets of SHA.

Original: The results in Figure 3 show that the brackets with more aggressive early stopping rates outperform random search.
Revised: The results in Figure 3 show that the brackets with more aggressive early stopping rates outperform random search.

Original: Bracket 0 found a good conﬁguration for this task in 3 × time ( R ) .
Revised: Bracket 0 found a good conﬁguration for this task in 3 × time ( R ) .

Original: By that time, random search (Bracket 4) had evaluated 1.5k conﬁgurations, compared to 1.8k for Vizier, 2.3k for Vizier with early stopping, and 52k for Bracket 0, the bracket with the most aggressive early stopping rate.
Revised: By that time, random search (Bracket 4) had evaluated 1.5k conﬁgurations, compared to 1.8k for Vizier, 2.3k for Vizier with early stopping, and 52k for Bracket 0, the bracket with the most aggressive early stopping rate.

Original: Vizier  underperforms Bracket 0 initially but eventually matches the performance of the best bracket (Bracket 1).
Revised: Vizier initially underperforms Bracket 0, but eventually matches the performance of the best bracket (Bracket 1).

Original: Vizier with early-stopping matches default Vizier up until 4 × time ( R ) but then overtakes it.
Revised: Vizier with early-stopping matches default Vizier up until 4 × time ( R ) but then overtakes it.

Original: The acoustic modeling task trains an LSTM on a small collection of 250 thousand spoken recordings of anonymized and aggregated typed search queries.
Revised: The acoustic modeling task trains an LSTM on a small collection of 250 thousand spoken recordings of anonymized and aggregated typed search queries.

Original: The search space we consider has 8 hyperpa- rameters and includes models with up to 5 LSTM layers (see Appendix A.2 for details).
Revised: The search space we consider has 8 hyperpa- rameters and includes models with up to 5 LSTM layers (see Appendix A.2 for details).

Original: Our model uses the LSTM cell introduced in Sak et al.
Revised: Our model uses the LSTM cell introduced in Sak et al.

Original: (2014) with a recurrent projection layer.
Revised: (2014) with a recurrent projection layer.

Original: Each tuning method is given 20 workers and allowed to run for 7 × time ( R ) .
Revised: Each tuning method is given 20 workers and allowed to run for 7 × time ( R ) .

Original: Note that the search space includes models that take nearly the entire time allowance to train.
Revised: Note that the search space includes models that take nearly the entire time allowance to train.

Original: The minimum resource for SHA is set to r =
Revised: The minimum resource for SHA is set to r =

Original: Rη − 3 for a total of 4 possible brackets.
Revised: Rη − 3 for a total of 4 possible brackets.

Original: The results in Figure 3 show a similar relative ordering of the brackets of SHA.
Revised: The results in Figure 3 show a similar relative ordering of the brackets of SHA.

Original: 2 Bracket 0 found a conﬁguration with error rate below 30 % in 2 × time ( R ) and evaluated over one thousand conﬁgurations in that time with just 20 workers.
Revised: 2 Bracket 0 found a conﬁguration with error rate below 30 % in 2 × time ( R ) and evaluated over one thousand conﬁgurations in that time with just 20 workers.

Original: While we have demonstrated state-of-the-art performance for asynchronous SHA, we hypothesize that even better results relative to Bayesian methods are possible in settings with more parallel resources and wall-clock times restricted to roughly the training time of a single model.
Revised: While we have demonstrated state-of-the-art performance for asynchronous SHA, we hypothesize that even better results relative to Bayesian methods are possible in settings with more parallel resources and wall-clock times restricted to roughly the training time of a single model.

Original: Asynchronous SHA is massively scalable because it faces no degradation in performance when compensating for time with more parallel resources.
Revised: Asynchronous SHA is massively scalable because it faces no degradation in performance when compensating for time with more parallel resources.

Original: In contrast, Bayesian methods are hampered in wall-clock time constrained settings, since they require updates to the posterior to effectively select new conﬁgurations.
Revised: In contrast, Bayesian methods are hampered in wall-clock time constrained settings, since they require updates to the posterior to effectively select new conﬁgurations.

Original: 4. P RACTICAL C ONSIDERATIONS FOR H YPERPARAMETER T UNING IN P ARALLEL S ETTINGS
Revised: 4. P RACTICAL C ONSIDERATIONS FOR H YPERPARAMETER T UNING IN P ARALLEL S ETTINGS

Original: 22222222 7 # of GPUs per Model 22222222# o f M o d e l s E v a l u a t e d
Revised: 22222222 7 # of GPUs per Model 22222222# o f M o d e l s E v a l u a t e d

Original: AlexNet
Revised: AlexNet

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: f M o d
Revised: f M o d

Original: [Table]
Revised: [Table]

Original: l s E v a l u a t e d
Revised: l s E v a l u a t e d

Original: Inception
Revised: Inception

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In the CNN tuning tasks from from Section 4.2, we used one GPU to train each model.
Revised: In the CNN tuning tasks from from Section 4.2, we used one GPU to train each model.

Original: However, distributed resources can also be used for parallel training to speed up the time it takes to train a single model (Krizhevsky, 2014; Szegedy et al., 2014;
Revised: However, distributed resources can also be used for parallel training to speed up the time it takes to train a single model (Krizhevsky, 2014; Szegedy et al., 2014;

Original: You et al., 2017; You et al., 2017; Goyal et al., 2017).
Revised: You et al., 2017; You et al., 2017; Goyal et al., 2017).

Original: Unfortunately, the speedups do not scale linearly with the number of GPUs due to communication costs and other overheads.
Revised: Unfortunately, the speedups do not scale linearly with the number of GPUs due to communication costs and other overheads.

Original: Therefore, there is a tradeoff between using distributed resources to train a model faster versus to evaluate more conﬁgurations.
Revised: Therefore, there is a tradeoff between using distributed resources to train a model faster versus to evaluate more conﬁgurations.

Original: Qi et al.
Revised: Qi et al.

Original: (2017) introduced Paleo, a framework for estimating the cost of training neural networks in parallel given different speciﬁcations.
Revised: (2017) introduced Paleo, a framework for estimating the cost of training neural networks in parallel given different speciﬁcations.

Original: Qi et al.
Revised: Qi et al.

Original: (2017) demonstrated that the speedups computed using
Revised: (2017) demonstrated that the speedups computed using

Original: Paleo are fairly accurate when compared to the actual observed speedups for AlexNet (Krizhevsky, 2014) and Inception (et al., 2016).
Revised: Paleo are fairly accurate when compared to the actual observed speedups for AlexNet (Krizhevsky, 2014) and Inception (et al., 2016).

Original: Therefore, we use Paleo to estimate the training time of these two networks under strong scaling (i.e. ﬁxed batch size with increasing parallelism), Butterﬂy AllReduce communication scheme, standard hardware settings (namely Tesla K80 GPU and 20G Ethernet), and a batch size of 1024.
Revised: Therefore, we use Paleo to estimate the training time of these two networks under strong scaling (i e. ﬁxed batch size with increasing parallelism), Butterﬂy AllReduce communication scheme, standard hardware settings (namely Tesla K80 GPU and 20G Ethernet), and a batch size of 1024.

Original: We then use the estimates to produce the tradeoff curves in Figure 4.
Revised: We then use the estimates to produce the tradeoff curves in Figure 4.

Original: The ﬁgure shows a general trend of diminishing returns associated with using more GPUs to train a single model and suggest choosing a time budget and level of parallelism as close to to the upper left as possible (i.e. if time permits, use only one GPU per model to explore as many models as possible).
Revised: The ﬁgure shows a general trend of diminishing returns associated with using more GPUs to train a single model and suggest choosing a time budget and level of parallelism as close to to the upper left as possible (i.e. if time permits, use only one GPU per model to explore as many models as possible).

Original: A E XPERIMENT D ETAILS
Revised: A E XPERIMENT D ETAILS

Original: In the below sections, we detail the experimental setup for the benchmarks in Section 4.
Revised: In the below sections, we detail the experimental setup for the benchmarks in Section 4.

Original: The SVM task uses the same search space and experimental setup as Klein et al.
Revised: The SVM task uses the same search space and experimental setup as Klein et al.

Original: (2017a)
Revised: (2017a)

Original: and the cuda-convnet task is the same as that in Li et al.
Revised: and the cuda-convnet task is the same as that in Li et al.

Original: A. E XPERIMENTAL S ETUP FOR THE S MALL CNN A RCHITECTURE T UNING T ASK
Revised: A. E XPERIMENTAL S ETUP FOR THE S MALL CNN A RCHITECTURE T UNING T ASK

Original: [Table]
Revised: [Table]

Original: This benchmark tunes a multiple layer CNN network with the hyperparameters shown in Table 1.
Revised: This benchmark tunes a multiple layer CNN network with the hyperparameters shown in Table 1.

Original: The # of layers hyperparameter indicate the number of convolutional layers before two fully connected layers.
Revised: The # of layers hyperparameter indicate the number of convolutional layers before two fully connected layers.

Original: The # of ﬁlters indicates the # of ﬁlters in the CNN layers with the last CNN layer having 2 ×
Revised: The # of ﬁlters indicates the # of ﬁlters in the CNN layers with the last CNN layer having 2 ×

Original: # ﬁlters.
Revised: # ﬁlters.

Original: Weights are initialized randomly from a Gaussian distribution with the indicated standard deviation.
Revised: Weights are initialized randomly from a Gaussian distribution with the indicated standard deviation.

Original: There are three sets of weight init and l 2 penalty hyperparameters; weight init 1 and lpenalty 1 apply to the convolutional layers, weight init 2 and l 2 penalty 2 to the ﬁrst fully connected layer, and weight init 3 and l 2 penalty 3 to the last fully connected layer.
Revised: There are three sets of weight init and l 2 penalty hyperparameters; weight init 1 and lpenalty 1 apply to the convolutional layers, weight init 2 and l 2 penalty 2 to the ﬁrst fully connected layer, and weight init 3 and l 2 penalty 3 to the last fully connected layer.

Original: Finally, the learning rate hyperparameter controls the initial learning rate for SGD.
Revised: Finally, the learning rate hyperparameter controls the initial learning rate for SGD.

Original: All models use a ﬁxed learning rate schedule with the learning rate decreasing by a factor of 10 twice in equally spaced intervals over the training window.
Revised: All models use a ﬁxed learning rate schedule with the learning rate decreasing by a factor of 10 twice in equally spaced intervals over the training window.

Original: This benchmark is run on the SVHN dataset (Netzer et al., 2011) following
Revised: This benchmark is run on the SVHN dataset (Netzer et al., 2011) following

Original: Sermanet et al.
Revised: Sermanet et al.

Original: (2012) to create the train, validation, and test splits.
Revised: (2012) to create the train, validation, and test splits.

Original: A. E XPERIMENTAL S ETUP FOR L ARGE -S CALE B ENCHMARKS
Revised: A. E XPERIMENTAL S ETUP FOR L ARGE -S CALE B ENCHMARKS

Original: The LSTM models tuned for our two large-scale benchmarks are implemented in Tensorﬂow.
Revised: The LSTM models tuned for our two large-scale benchmarks are implemented in Tensorﬂow.

Original: There are a few sources of randomness that impact the relative performance of the tuning methods:
Revised: There are a few sources of randomness that impact the relative performance of the tuning methods:

Original: Randomly sampled hyperparameter conﬁgurations.
Revised: Randomly sampled hyperparameter conﬁgurations.

Original: Random weight initialization.
Revised: Random weight initialization.

Original: Random minibatches.
Revised: Random minibatches.

Original: We control for these sources to the best of our ability but randomness remain in part due to multithreaded data input queues and non-associative ﬂoating point operations.
Revised: We control for these sources to the best of our ability but randomness remain in part due to multithreaded data input queues and non-associative ﬂoating point operations.

Original: The hyperparameters for the LSTM tuning task on the Penn Tree Bank (PTB) dataset presented in
Revised: The hyperparameters for the LSTM tuning task on the Penn Tree Bank (PTB) dataset presented in

Original: Section 4.3 is shown in Table 2.
Revised: Section 4.3 is shown in Table 2.

Original: Note that all hyperparameters are tuned on a linear scale and sampled uniform over the speciﬁed range.
Revised: Note that all hyperparameters are tuned on a linear scale and sampled uniform over the speciﬁed range.

Original: The inputs to the LSTM layer are learned embeddings of the words in a sequence.
Revised: The inputs to the LSTM layer are learned embeddings of the words in a sequence.

Original: The number of hidden nodes hyperparameter refers to the number of nodes in the
Revised: The number of hidden nodes hyperparameter refers to the number of nodes in the

Original: LSTM.
Revised: LSTM.

Original: The learning rate is decayed by the decay rate after each interval of decay steps.
Revised: The learning rate is decayed by the decay rate after each interval of decay steps.

Original: Finally, the weight initialization range indicates the upper bound of the uniform distribution used to initialize all weights.
Revised: Finally, the weight initialization range indicates the upper bound of the uniform distribution used to initialize all weights.

Original: The other hyperparameters have their standard interpretations for neural networks.
Revised: The other hyperparameters have their standard interpretations for neural networks.

Original: The default training (929k words) and test (82k words) splits for PTB are used for training and evaluation (Marcus et al., 1993).
Revised: The default training (929k words) and test (82k words) splits for PTB are used for training and evaluation (Marcus et al., 1993).

Original: The hyperparameters for the acoustic model tuning task presented in Section 4.3 is shown in Table 3.
Revised: The hyperparameters for the acoustic model tuning task presented in Section 4.3 is shown in Table 3.

Original: The hyperparameters that overlap with the PTB task also have the same interpretation.
Revised: The hyperparameters that overlap with the PTB task also have the same interpretation.

Original: The only additional hyperparameter is the projection fraction, which indicates the size of the recurrent projection
Revised: The only additional hyperparameter is the projection fraction, which indicates the size of the recurrent projection

Original: [Table]
Revised: [Table]

Original: layer in the LSTM cell introduced by Sak et al.
Revised: layer in the LSTM cell introduced by Sak et al.

Original: (2014) as a fraction of the number of hidden units in the LSTM.
Revised: (2014) as a fraction of the number of hidden units in the LSTM.

Original: The model is trained on a dataset with 250k examples and evaluated on a test set with 2k examples.
Revised: The model is trained on a dataset with 250k examples and evaluated on a test set with 2k examples.

Original: [Table]
Revised: [Table]