M ASSIVELY P ARALLEL H YPERPARAMETER T UNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Modern learning models are characterized by large hyperparameter spaces.
In order to adequately explore these large spaces, we must evaluate a large number of conﬁgurations, typically orders of magnitude more conﬁgurations than available parallel workers.
Given the growing costs of model training, we would ideally like to perform this search in roughly the same wall-clock time needed to train a single model.
In this work, we tackle this challenge by introducing ASHA, a simple and robust hyperparameter tuning algorithm with solid theoretical underpinnings that exploits parallelism and aggressive early-stopping.
Our extensive empirical results show that ASHA outperforms state-of-the-art hyperparameter tuning methods; scales linearly with the number of workers in distributed settings; converges to a high quality conﬁguration in half the time taken by Vizier (Google’s internal hyperparameter tuning service) in an experiment with 500 workers; and beats the published result for a near state-of-the-art LSTM architecture in under 2 × the time to train a single model.
I NTRODUCTION
Although machine learning models have recently achieved dramatic successes in a variety of practical applications, these models are highly sensitive to internal parameters, i.e., hyperparameters .
In these modern learning regimes, three trends motivate a new approach to hyperparameter tuning:
High- dimensional hyperparameter spaces .
Machine learning models
are
becoming
increasingly complex, as evidenced by modern neural networks with dozens of hyperparameters.
For such complex models with hyperparameters that interact in unknown ways, a practitioner is forced to evaluate potentially thousands of different hyperparameter settings
Increasing training times .
As datasets grow larger and models become more complex, training a model has become dramatically more expensive, often taking days or weeks on specialized high-performance hardware.
This trend is particularly onerous in the context of hyperparameter tuning, as a new model must be trained to evaluate each candidate hyperparameter conﬁguration.
Rise of parallel computing .
The combination of a growing number of hyperparameters and longer training time per model precludes
evaluating conﬁgurations sequentially; we simply cannot wait months or years to ﬁnd a suitable hyperparameter setting.
Leveraging parallel and distributed computational resources presents a solution to the increasingly challenging problem of hyperparameter optimization.
Our goal is to design a hyperparameter tuning algorithm that can effectively leverage parallel resources.
This goal seems trivial with standard methods like random search, where we could train different conﬁgurations in an embarrassingly parallel fashion.
However, for high-dimensional search spaces, the number of candidate conﬁgurations required to ﬁnd a good conﬁguration often dwarfs the number of available parallel resources, and when possible, our goal is to:
Evaluate orders of magnitude more hyperparameter conﬁgurations than available parallel workers in
a small multiple of the wall-clock time needed to train a single model.
We denote this setting as the large-scale regime for hyperparameter optimization.
In this work we study this regime, and our contributions are as follows:
• We introduce the A synchronous S uccessive H alving A lgorithm (ASHA), a practical hyperparameter tuning method for the large-scale regime that exploits parallelism and aggressive early-stopping.
Our algorithm is inspired by the Successive Halving algorithm (SHA)
(Karnin et al., 2013; Jamieson & Talwalkar, 2015), a theoretically principled early-stopping method that allocates more resources to promising conﬁgurations.
As a side beneﬁt of adapting SHA for the parallel setting, we also improve upon the synchronous version of SHA, as described in Section 3.3.
• In the sequential setting, we compare SHA with Fabolas (Klein et al., 2017) and Population Based Tuning (PBT) (Jaderberg et al., 2017), two state-of-the-art methods that exploit partial training (Klein et al., 2017; Jaderberg et al., 2017).
Our results show that SHA slightly outperforms Fabolas and PBT, which when coupled with SHA’s simplicity and theoretical grounding, motivate the use of SHA in parallel settings.
We also verify that SHA and ASHA achieve similar results.
• In a distributed setting with 25 workers, we evaluate ASHA’s suitability for the large-scale regime.
We focus on tuning two CNNs on CIFAR-10, and observe that ASHA ﬁnds a good conﬁguration in approximately the time it takes to train a single model.
We also show that ASHA scales linearly with the number of workers, and that it again slightly exceeds the performance of PBT and BOHB (Falkner et al., 2018), a method combining Bayesian optimization with SHA.
• In a distributed setting with 500 workers, we compare ASHA to the default tuning algorithm used by Vizier (Golovin et al., 2017), Google’s internal hyperparameter tuning service.
We show that ASHA outperforms Vizier when tuning an LSTM model on the Penn Treebank dataset (PTB).
• Finally, we compare ASHA and PBT on a task using 16 GPUs to tune a near state-of-the-art language model (Merity et al., 2018), with a published test perplexity of 57.3 on PTB.
ASHA outperforms PBT and is able to ﬁnd a conﬁguration with a test perplexity of 56.3 in less than 2 × the time it takes to train a single model.
R ELATED W ORK
Sequential methods : There is a large body of work in this area, with existing methods focusing on adaptively evaluating conﬁgurations, adaptively selecting conﬁgurations, or taking a hybrid approach that combines both strategies.
SHA (Jamieson & Talwalkar, 2015) and Hyperband (Li et al., 2018) are empirically effective adaptive evaluation techniques that employ principled ‘early-stopping.’
SHA serves as the inner loop for Hyperband, with Hyperband automating the choice of the early-stopping rate by running different variants of SHA.
While the appropriate choice of early stopping rate is problem dependent, Li et al.
(2018)’s
empirical results show that aggressive early-stopping works well for a wide variety of tuning tasks.
Hence, we focus on adapting SHA to the parallel setting in
Section 3, though we also evaluate the corresponding asynchronous Hyperband method.
Fabolas (Klein et al., 2017), a hybrid Bayesian optimization method, has demonstrated state-of-theart performance on several tasks in comparison to Hyperband and other leading methods.
While
Fabolas, along with most other Bayesian optimization approaches, can be parallelized using a constant liar (CL) type heuristic (Ginsbourger et al., 2010; González et al., 2016), the parallel version will underperform the sequential version, since the latter uses a more accurate posterior to propose new points.
Hence, we compare to Fabolas in the sequential setting in Section 4.1, and demonstrate that under an appropriate experimental setup, SHA and Hyperband in fact slightly outperform Fabolas. More recently in parallel work, Falkner et al.
(2018) introduced BOHB, a hybrid method combining
Bayesian optimization with Hyperband.
They also propose a parallelization scheme for SHA that retains synchronized eliminations of underperforming conﬁgurations.
We discuss the drawbacks of this parallelization scheme in Section 3 and demonstrate that ASHA outperforms this version of parallel SHA as well as BOHB in Section 4.2.
Parallel methods : Jaderberg et al.
(2017) developed PBT, a state-of-the-art hybrid evolutionary approach that exploits partial training to iteratively increase the ﬁtness of a population of models.
Additionally, Golovin et al.
(2017) introduced Vizier, a black-box optimization service with support for multiple tuning methods and early-stopping options.
For succinctness, we will refer to Vizier’s default algorithm as “Vizier” with the understanding that it is simply one of methods available on the Vizier platform.
We compare to PBT and Vizier in Section 4.2 and Section 4.3, respectively.
We further note that PBT is more comparable to SHA than to Hyperband since both PBT and SHA require the user to set the early-stopping rate via internal hyperparameters.
Neural Architecture Search (NAS) methods : Another class of related methods are tailored for search spaces designed to tune the architectures of neural networks.
While earlier methods were computationally expensive (Zoph & Le, 2017; Real et al., 2017), more recent methods exploit reuse to reduce the costs (Pham et al., 2018; Bender et al., 2018).
However, these methods suffer from reproducibility issues (Pham et al., 2018; Pham & Guan, 2018), and recent work has demonstrated that appropriately tuning more traditional architectures can match or outperform the results from architecture search (Melis et al., 2018).
Relatedly, ASHA’s performance in Section 4.3.1 matches that of leading NAS methods on the PTB benchmark (Pham et al., 2018; Pham & Guan, 2018).
ASHA A LGORITHM
We start with an overview of SHA (Karnin et al., 2013; Jamieson & Talwalkar, 2015) and motivate the need to adapt it to the parallel setting.
Then we present ASHA and discuss how it addresses issues with synchronous SHA and improves upon the original algorithm.
3. S UCCESSIVE H ALVING (SHA)
Algorithm
1 Input: number of conﬁgurations n , minimum resource r , maximum resource R , reduction factor η , minimum early-stopping rate s 2 s max
=
(cid:98) log η
( R/r ) (cid:99) 3 assert n ≥ η s max − s so that at least one conﬁguration will be allocated R .
4 T = get_hyperparameter_configuration ( n ) 5 for i ∈ { 0 , . . .
, s max − s } do //
All configurations trained for a given i constitute a “rung.”
n i = (cid:98) nη − i (cid:99) r i =
rη i + s
L = run_then_return_val_loss ( θ, r i ) : θ ∈ T
T = top_k ( T, L, n i /η ) 11 end 12 return best conﬁguration in T
The idea behind SHA (Algorithm 1) is simple: allocate a small budget to each conﬁguration, evaluate all conﬁgurations and keep the top 1 /η , increase the budget per conﬁguration by a factor of η , and repeat until the maximum per-conﬁguration budget of R is reached (lines 5–11).
The resource allocated by SHA can be iterations of stochastic gradient descent, number of training examples, number of random features, etc.
bracket s rung i n i r i total budget    9  9       27        takes more
SHA requires the number of conﬁgurations n , a minimum resource r , a maximum resource
R , a reduction factor η ≥ 2 , and a minimum early-stopping rate s .
Additionally, the get_hyperparameter_configuration ( n ) subroutine returns n conﬁgurations sampled randomly from a given search space; and the run_then_return_val_loss ( θ, r ) subroutine returns the validation loss after training the model with the hyperparameter setting θ and for r resources.
For a given early-stopping rate s , a minimum resource of r 0 =
rη s will be allocated to each conﬁguration.
Hence, lower s corresponds to more aggressive early-stopping, with s = 0 prescribing a minimum resource of r .
We will refer to SHA with different values of s as brackets and, within a bracket, we will refer to each round of promotion as a rung with the base rung numbered 0 and increasing.
Figure 1(left) shows the rungs for bracket 0 for an example setting with n
= 9 , r = 1 , R = 9 , and η = 3 , while Figure 1(right) shows how resource allocations change for different brackets.
The naive way of parallelizing SHA is to distribute the training of the n/η k surviving conﬁgurations on each rung k as is done by Falkner et al.
and add brackets when there are no jobs available in existing brackets.
We will refer to this method as “synchronous” SHA. The efﬁcacy of this strategy is severely hampered by two issues: (1) SHA’s synchronous nature is sensitive to stragglers and dropped jobs as every conﬁguration within a rung must complete before proceeding to the next rung, and (2) the estimate of the top 1 /eta conﬁgurations for a given early-stopping rate does not improve as more brackets are run since promotions are performed independently for each bracket. We demonstrate in Appendix A.1 that ASHA is more robust than synchronous SHA to stragglers and dropped jobs using simulated workloads. Nonetheless, we compare synchronous SHA in Section 4.3 and show that ASHA performs better.
We could also consider the embarrassingly parallel approach of running multiple instances of SHA, one on each worker.
However, this strategy is not well suited for the large-scale regime, where we would like results in little more than the time to train one conﬁguration.
To see this, assume that training time for a conﬁguration scales linearly with the allocated resource and time ( R ) represents the time required to train a conﬁguration for the maximum resource R .
In general, for a given bracket s , the minimum time to return a conﬁguration trained to completion is (log η ( R/r ) − s +1)
× time ( R ) , where log η ( R/r ) − s + 1 counts the number of rungs.
For example, consider Bracket 0 in the toy example in Figure 1.
The time needed to return a fully trained conﬁguration is 3 × time ( R ) , since there are three rungs and each rung is allocated R resource.
Moreover, using this parallelization strategy on the task considered in Section 4.3.1 would have required 5 days to return an answer, compared to the 1 day needed for ASHA.
early
end do
// Check to see if there is a promotable config.
We now introduce ASHA as an effective technique to parallelize SHA, leveraging asynchrony to mitigate stragglers and maximize parallelism.
Intuitively, ASHA promotes conﬁgurations to the next rung whenever possible instead of waiting for a rung to complete before proceeding to the next rung.
Additionally, if no promotions are possible, ASHA simply adds a conﬁguration to the base rung, so that more conﬁgurations can be promoted to the upper rungs.
ASHA is formally deﬁned in Algorithm 2.
Given its asynchronous nature it does not require the user to pre-specify the number of conﬁgurations to evaluate, but it otherwise requires the same inputs as SHA.
Note that the run_then_return_val_loss subroutine in ASHA is asynchronous and the code execution continues after the job is passed to the worker.
ASHA’s promotion scheme is laid out in the get_job subroutine.
We compare the promotion schemes of SHA and ASHA in Figure 2. [Figure]
[-12 Chronological Jobs rungrungrung1 2 3 4 5 6 7 88 
 Successive Halving (Synchronous) B u dg e t-]color gradient corresponds to performance, with lighter bars having lower error; as in Figure 1(left), conﬁgurations 1, 6, and 8 are promoted to rung 1 and conﬁguration 8 is promoted to rung 2.
SHA must complete all jobs in a [-t e d t o J o b     12 Chronological Jobs rungrungrung1 2 4 5 7 
-]rung before moving the next rung.
In contrast, ASHA strives to maintain the simple rule that each rung should always have about 1 /η
= 1 / 3 of the conﬁgurations as the rung below it, while new conﬁgurations are added to the bottom rung as needed.
ASHA is well-suited for the large-scale regime, where wall-clock time is constrained to a small multiple of the time needed to train a single model.
For ease of comparison with SHA, assume training time scales linearly with the resource.
Consider the example of Bracket 0 shown in Figure 1, and assume we can run ASHA with 9 machines.
Then ASHA returns a fully trained conﬁguration in 13 / 9 × time ( R ) , since 9 machines are sufﬁcient to promote conﬁgurations to the next rung in the same time it takes to train a single conﬁguration in the rung.
Hence, the training time for a conﬁguration in rung 0 is 1 / 9 × time ( R ) , for rung 1 it is 1 / 3 × time ( R ) , and for rung 2 it is time ( R ) .
In general, η log η ( R ) − s machines are needed to advance a conﬁguration to the next rung in the same time it takes to train a single conﬁguration in the rung, and it takes η s + i − log η ( R ) × time ( R ) to train a conﬁguration in rung i .
Hence, ASHA can return a conﬁguration trained to completion in time
[Equation]
Moreover, when training is iterative, ASHA can return an answer in time ( R ) , since incrementally trained conﬁgurations can be checkpointed and resumed.
Finally, since Hyperband simply runs multiple SHA brackets, we can asynchronously parallelize
Hyperband by either running multiple brackets of ASHA or looping through brackets of ASHA sequentially as is done in the original Hyperband.
We employ the latter looping scheme for asynchronous Hyperband in the next section.
3. A LGORITHM D ISCUSSION
ASHA is able to remove the bottleneck associated with synchronous promotions at the cost of a small number of incorrect promotions.
By the law of large numbers, we expect to erroneously promote a vanishing fraction of conﬁgurations in each rung as the number of conﬁgurations grows.
Intuitively, in the ﬁrst rung with n evaluated conﬁgurations, the number of mispromoted conﬁgurations is roughly √ n , since the process resembles the convergence of an empirical cumulative distribution function to its expected value (c.f., the Dvoretzky-Kiefer-Wolfowitz inequality (Dvoretzky et al., 1956)).
For later rungs, although the conﬁgurations are no longer i.i.d.
since they were advanced based on the empirical CDF of the evaluation scores, we expect this dependence to be weak.
We further note that ASHA improves upon SHA in two ways.
First, Li et al.
(2018) discusses two SHA variants: ﬁnite horizon (bounded resource per conﬁguration, i.e., R is bounded) and inﬁnite horizon (unbounded resources per conﬁguration, i.e., R is unbounded).
ASHA consolidates these settings into one algorithm.
In Algorithm 2, we do not promote conﬁgurations that have been trained for R , thereby restricting the number of rungs.
However, this algorithm trivially generalizes to the inﬁnite horizon; we can remove this restriction so that the maximum resource per conﬁguration increases naturally as conﬁgurations are promoted to higher rungs.
In contrast, SHA does not naturally extend to the inﬁnite horizon setting, as it relies on the doubling trick and must rerun brackets with larger budgets to increase the maximum resource.
Additionally, SHA does not return an output until a single bracket completes.
In the ﬁnite horizon this means that there is a constant interval of ( # of rungs × time ( R )) between receiving outputs from SHA.
In the inﬁnite horizon this interval doubles between outputs.
In contrast, ASHA grows the bracket incrementally instead of in ﬁxed budget intervals.
To further reduce latency, ASHA uses intermediate losses to determine the current best performing conﬁguration, as opposed to only considering the ﬁnal SHA outputs.
E MPIRICAL E VALUATION
We ﬁrst present results in the sequential setting to justify our choice of focusing on SHA and to compare SHA to ASHA.
We next evaluate ASHA in parallel environments on three benchmark tasks.
4. S EQUENTIAL E XPERIMENTS
We benchmark Hyperband and SHA against PBT and Fabolas, and examine the relative performance of SHA versus ASHA and Hyperband versus asynchronous Hyperband.
As mentioned previously, asynchronous Hyperband loops through brackets of ASHA with different early-stopping rates, switching brackets when a budget corresponding to a hypothetical bracket of SHA would be depleted.
[Figure]
We compare ASHA against PBT and synchronous SHA on two benchmarks for CIFAR-10: (1) tuning a convolutional neural network (CNN) with the cuda-convnet architecture and the same search space as Li et al.
(2017); and (2) tuning a CNN architecture with varying number of layers, batch size, and [Figure] number of ﬁlters.
The details for the search spaces considered for each benchmark and the settings we used for each search method can be found in Appendix A.3.
The results on these two benchmarks are shown in Figure 3.
On benchmark 1, both Hyperband and SHA (both synchronous and asynchronous versions) outperform PBT by 3 × .
On benchmark 2, while all methods comfortably beat random search, SHA, ASHA and PBT performed similarly and slightly outperform Hyperband and asynchronous Hyperband.
This last observation (i) corroborates the results in Li et al.
(2017), which found that the brackets with the most aggressive early-stopping rates performed the best; and (ii) follows from the discussion in Section 2 noting that PBT is more similar in spirit to SHA than Hyperband, as PBT / SHA both require user-speciﬁed early-stopping rates (and are more aggressive in their early-stopping behavior in these experiments).
For both tasks, introducing asynchrony does not consequentially impact the performance of ASHA (relative to SHA) or asynchronous Hyperband (relative to Hyperband).
This not surprising; as discussed in Section 3.3, we expect the number of ASHA mispromotions to be square root in the number of conﬁgurations.
Finally, due to the nuanced nature of the evaluation framework used by Klein et al.
(2017) in comparing Fabolas with Hyperband, we present our results on 4 different benchmarks along with a detailed discussion of the experimental setup in Appendix A.2.
In summary, our results show that
Hyperband, speciﬁcally the ﬁrst bracket of SHA, tends to outperform Fabolas while also exhibiting lower variance across experimental trials.
4. L IMITED -S CALE D ISTRIBUTED E XPERIMENTS
We next compare ASHA to synchronous SHA, the parallelization scheme discussed in Section 3.1; PBT; and BOHB (synchronous SHA with Bayesian optimization as introduced by Falkner et al. on the same two tasks from Section 4.1.
For each experiment, we run each search method with 25 workers for 150 minutes.
We use the same setups for ASHA and PBT as in the previous section.
We run synchronous SHA and BOHB with default settings and the same η and early-stopping rate as ASHA.
Figure 4 shows the average test error across 5 trials for each search method.
On both tasks, ASHA succeeded in the large-scale regime.
For benchmark 1, ASHA evaluated over 1000 conﬁgurations in just over 40 minutes with 25 workers and found a good conﬁguration (error rate below 0.21) in approximately the time needed to train a single model, whereas it took ASHA nearly 400 minutes to do so in the sequential setting (Figure 3).
Notably, we only achieve a 10 × speedup on 25 workers due to the relative simplicity of this task, i.e., it only required evaluating a few hundred conﬁgurations before identifying a good one in the sequential setting.
In contrast, when considering the more difﬁcult search space in benchmark 2, we observe linear speedups with ASHA, as the roughly 700 minutes in the sequential setting (Figure 3) needed to ﬁnd a conﬁguration [Figure]
Compared to synchronous SHA and BOHB, ASHA ﬁnds a good conﬁguration twice as fast on benchmark 1 while BOHB ﬁnds a slightly better ﬁnal conﬁguration. On benchmark 2, ASHA performs signiﬁcantly better than synchronous SHA and BOHB due to the higher variance in training times between conﬁgurations (the average time required to train a conﬁguration on the maximum resource R is 30 minutes with a standard deviation of 27 minutes), which exacerbates the sensitivity of synchronous SHA to stragglers (see Appendix A.1). BOHB actually underperforms synchronous SHA on benchmark 2, demonstrating that adaptive sampling is not always helpful.
We further note that ASHA outperforms PBT on benchmark 1; in fact the minimum and maximum range for ASHA across 5 trials does not overlap with the average for PBT.
On benchmark 2, PBT slightly outperforms asynchronous Hyperband and performs comparably to ASHA.
However, note that the ranges for the searchers share large overlap and the result is likely not signiﬁcant. Overall,
ASHA outperforms PBT, BOHB and SHA on these two tasks. This improved performance, coupled with the fact that it is a more principled and general approach than either BOHB or PBT (e.g., agnostic to resource type and robust to hyperparameters that change the size of the model), further motivates its use for the large-scale regime.
[Equation] ODELS
We tune a one layer LSTM language model for next word prediction on the Penn Treebank dataset (Marcus et al., 1993). Our search space is constructed based off of the LSTMs considered in Zaremba et al.
(2014), with the largest model in our search space matching their large LSTM (see
Appendix A.5 for more details). Each tuner is given 500 workers and 6 × time ( R ) , i.e., 6 × the average time needed to train a single model.
For ASHA, we set η = 4 , r = R / 64 , and s = 0 ; asynchronous Hyperband loops through brackets s = 0 , 1 , 2 , 3 .
We compare to Vizier without the performance curve early-stopping rule (Golovin et al., 2017).
The results in Figure 5 show that ASHA and asynchronous Hyperband found good conﬁgurations for this task in 1 × time ( R ) .
Additionally, ASHA and asynchronous Hyperband are both about 3 × faster than Vizier at ﬁnding a conﬁguration with test perplexity below 80, despite being much simpler and easier to implement.
Furthermore, the best model found by ASHA achieved a test perplexity of 76.6, which is signiﬁcantly better than 78.4 reported for the large LSTM in Zaremba et al.
We also note that asynchronous Hyperband initially lags behind ASHA, but eventually catches up at around 1 .
5 × time ( R ) .
Notably, we observe that certain hyperparameter conﬁgurations in this benchmark induce perplexities that are orders of magnitude larger than the average case perplexity.
Model-based methods that make assumptions on the data distribution, such as Vizier, can degrade in performance without further care to adjust this signal.
We attempted to alleviate this by capping perplexity scores at 1000 but this still signiﬁcantly hampered the performance of Vizier.
We view robustness to these types of scenarios as an additional beneﬁt of ASHA and Hyperband.
4.3. T UNING M ODERN LSTM A RCHITECTURES
We next performed a follow up experiment on the Penn Treebank dataset focusing on models with improved perplexity.
Our starting point was the work of Merity et al.
(2018), which introduced a near state-of-the-art LSTM architecture with DropConnect, a new regularization scheme that can be easily combined with existing LSTM implementations in common deep learning frameworks.
We constructed a search space around their conﬁguration and ran ASHA and PBT, each with
GPUS on one p2.16xlarge instance on AWS. For ASHA, we used η = 4 , r = 1 epoch, R =epochs, and s = 0 .
For PBT, we used the same settings as the previous two sections except we set a maximum resource of 256 epochs and perform explore/exploit every 8 epochs. Figure 6 shows that while PBT performs better initially, ASHA soon catches up and ﬁnds ﬁnal conﬁguration with a validation perplexity of 60.6 and a test perplexity of 58.4 compared to 62. (validation) and 60.3 (test) for PBT.
We then trained the best conﬁguration found by ASHA for more epochs and reached valdiation and test perplexities of 60.2 and 58.1 respectively before ﬁne-tuning and 58.7 and 56.3 after ﬁne-tuning. For reference, Merity et al. (2018) reported validation and test perplexities respectively of 60.7 and 58.8 without ﬁne-tuning and 60.0 and 57.3 with ﬁne-tuning.
This demonstrates the effectiveness of ASHA in the large-scale regime for modern hyperparameter optimization problems.
C ONCLUSION
In this paper, we introduced ASHA and demonstrated its suitability for the large-scale regime of hyperparameter optimization.
Directions for future work include combining ASHA with adaptive selection methods and incorporating meta-learning to inform early-stopping.
These extensions overlap with those for synchronous SHA, and any progress made for the synchronous algorithm will likely also apply for ASHA.
A A PPENDIX
As part of our supplementary material, we (1) demonstrate the impact of stragglers and dropped jobs on synchronous SHA and ASHA, (2) present the comparison to Fabolas in the sequential setting, and (3) provide additional details for the empirical results shown in Section 4.
A. S IMULATED W ORKLOADS TO E VALUATE THE I MPACT OF S TRAGGLERS AND D ROPPED J OBS
For our simulated workloads, we run SHA with η = 4 , r = 1 , R = 256 , and n = 256 and ASHA with the same values and the maximum early-stopping rate s = 0 .
Note that BOHB is also susceptible to stragglers and dropped jobs since it uses synchronous SHA as its parallelization scheme but leverages
Bayesian optimization to perform adaptive sampling.
We assume that the expected training time for each job is the same as the allocated resource.
We simulated stragglers by multiplying the expected train time by (1 + | z | ) where z is drawn from a normal distribution with mean 0 and a speciﬁed standard deviation.
We simulated dropped jobs by assuming that there is a given p probability that a job will be dropped at each time unit, hence, for a job with a runtime of 256 units, the probability that it is not dropped is (1 − (1 − p ) 256 . For each combination of training standard deviation of drop probability, we simulate ASHA and synchronous SHA 25 times. Figure 7 shows that ASHA trains many more conﬁgurations to completion when the standard deviation is high; we hypothesis that this is one reason ASHA performs signiﬁcantly better than synchronous SHA and BOHB for the second benchmark in Section 4.2. Figure 8 shows that ASHA returns a conﬁguration trained for the maximum resource R much faster than synchronous SHA when there is high variability in training time (i.e., stragglers) and dropped jobs.
A. C OMPARISON WITH F ABOLAS IN S EQUENTIAL S ETTING
Klein et al.
(2017) showed that Fabolas can be over an order of magnitude faster than existing
Bayesian optimization methods.
Additionally, the empirical studies presented in Klein et al.
(2017) suggest that Fabolas is faster than Hyperband at ﬁnding a good conﬁguration.
We conducted our own experiments to compare Fabolas with Hyperband on the following tasks:
Tuning an SVM using the same search space as Klein et al.
Tuning a convolutional neural network (CNN) with the same search space as Li et al.
(2017) on CIFAR-10 (Krizhevsky, 2009).
Tuning a CNN on SVHN (Netzer et al., 2011) with varying number of layers, batch size, and number of ﬁlters (see Appendix A.4 for more details).
In the case of the SVM task, the allocated resource is number of training datapoints, while for the
CNN tasks, the allocated resource is the number of training iterations.
We note that Fabolas was speciﬁcally designed for data points as the resource, and hence, is not directly applicable to tasks (2) and (3).
However, freeze-thaw Bayesian optimization (Swersky et al., 2014), which was speciﬁcally designed for models that use iterations as the resource, is known to perform poorly on deep learning tasks (Domhan et al., 2015).
Hence, we believe Fabolas to be a reasonable competitor for tasks (2) and (3) as well, despite the aforementioned shortcoming.
We use the same evaluation framework as Klein et al.
(2017), where the best conﬁguration, also known as the incumbent , is recorded through time and the test error is calculated in an ofﬂine validation step. [Figure] [Figure] [Figure]
Following Klein et al.
(2017), the incumbent for Hyperband is taken to be the conﬁguration with the lowest validation loss and the incumbent for Fabolas is the conﬁguration with the lowest predicted validation loss on the full dataset.
Moreover, for these experiments, we set η = 4 for Hyperband.
Notably, when tracking the best performing conﬁguration for Hyperband, we consider two approaches.
We ﬁrst consider the approach proposed in Li et al.
and used by Klein et al.
(2017) in their evaluation of Hyperband.
In this variant, which we refer to as “Hyperband (by bracket),” the incumbent is recorded after the completion of each SHA bracket .
We also consider a second approach where we record the incumbent after the completion of each rung of SHA to make use of intermediate validation losses, similar to what we propose for ASHA (see discussion in Section 3.3 for details).
We will refer to Hyperband using this accounting scheme as “Hyperband (by rung).”
Interestingly, by leveraging these intermediate losses, we observe that Hyperband actually outperforms Fabolas.
In Figure 9, we show the performance of Hyperband, Fabolas, and random search.
Our results show that Hyperband (by rung) is competitive with Fabolas at ﬁnding a good conﬁguration and will often ﬁnd a better conﬁguration than Fabolas with less variance.
Note that Hyperband loops through the brackets of SHA, ordered by decreasing early-stopping rate; the ﬁrst bracket ﬁnishes when the test error for Hyperband (by bracket) drops.
Hence, most of the progress made by Hyperband comes from the bracket with the most aggressive early-stopping rate, i.e. bracket 0.
A. E XPERIMENTS IN S ECTION 4.1 AND S ECTION 4.
We use the usual train/validation/test splits for CIFAR-10, evaluate conﬁgurations on the validation set to inform algorithm decisions, and report test error.
These experiments were conducted using g2.2xlarge instances on Amazon AWS.
For both benchmark tasks, we run SHA with n = 256 , η = 4 , s = 0 , and set r = R/ 256 , where
R = 30000 iterations of stochastic gradient descent.
Hyperband loops through 5 brackets of SHA, moving from bracket s = 0 , r
=
R/ 256 to bracket s = 4 , r = R .
We run ASHA and asynchronous
Hyperband with the same settings as the synchronous versions.
We run PBT with a population size of 25, which is between the recommended 20–40 (Jaderberg et al., 2017).
Furthermore, to help PBT evolve from a good set of conﬁgurations, we randomly sample conﬁgurations until at least half of the population performs above random guessing.
We implement PBT with truncation selection for the exploit phase, where the bottom 20% of conﬁgurations are replaced with a uniformly sampled conﬁguration from the top 20% (both weights and hyperparameters are copied over).
Then, the inherited hyperparameters pass through an exploration phase where 3 / 4 of the time they are either perturbed by a factor of 1.2 or 0.8 (discrete hyperparameters are perturbed to two adjacent choices), and 1 / 4 of the time they are randomly resampled.
Conﬁgurations are considered for exploitation/exploration every 1000 iterations, for a total ofrounds of adaptation.
For the experiments in Section 4.2, to maintain 100% worker efﬁciently for
PBT while enforcing that all conﬁgurations are trained for within 2000 iterations of each other, we spawn new populations of 25 whenever a job is not available from existing populations.
[Table]
[Equation] [Equation] Vanilla PBT is not compatible with hyperparameters that change the architecture of the neural network, since inherited weights are no longer valid once those hyperparameters are perturbed.
To adapt PBT for the architecture tuning task, we ﬁx hyperparameters that affect the architecture in the explore stage.
Additionally, we restrict conﬁgurations to be trained within 2000 iterations of each other so a fair comparison is made to select conﬁgurations to exploit.
If we do not impose this restriction, PBT will be biased against conﬁgurations that take longer to train, since it will be comparing these conﬁgurations with those that have been trained for more iterations.
A. E XPERIMENTAL S ETUP FOR THE S MALL CNN A RCHITECTURE T UNING T ASK
This benchmark tunes a multiple layer CNN network with the hyperparameters shown in Table 1.
This search space was used for the small architecture task on SVHN (Section A.2) and CIFAR-
(Section 4.2).
The # of layers hyperparameter indicate the number of convolutional layers before two fully connected layers.
The # of ﬁlters indicates the # of ﬁlters in the CNN layers with the last
CNN layer having 2 × # ﬁlters.
Weights are initialized randomly from a Gaussian distribution with the indicated standard deviation.
There are three sets of weight init and l 2 penalty hyperparameters; weight init 1 and l 2 penalty 1 apply to the convolutional layers, weight init 2 and l 2 penalty 2 to the ﬁrst fully connected layer, and weight init 3 and l 2 penalty 3 to the last fully connected layer.
Finally, the learning rate hyperparameter controls the initial learning rate for SGD.
All models use a ﬁxed learning rate schedule with the learning rate decreasing by a factor of 10 twice in equally spaced intervals over the training window.
This benchmark is run on the SVHN dataset (Netzer et al., 2011) following Sermanet et al.
(2012) to create the train, validation, and test splits.
A. E XPERIMENTAL S ETUP FOR L ARGE -S CALE B ENCHMARKS
The hyperparameters for the LSTM tuning task comparing ASHA to Vizier on the Penn Tree Bank (PTB) dataset presented in Section 4.3 is shown in Table 2.
Note that all hyperparameters are tuned on a linear scale and sampled uniform over the speciﬁed range.
The inputs to the LSTM layer [Table] [Table] are embeddings of the words in a sequence.
The number of hidden nodes hyperparameter refers to the number of nodes in the LSTM.
The learning rate is decayed by the decay rate after each interval of decay steps.
Finally, the weight initialization range indicates the upper bound of the uniform distribution used to initialize all weights.
The other hyperparameters have their standard interpretations for neural networks.
The default training (929k words) and test (82k words) splits for
PTB are used for training and evaluation (Marcus et al., 1993).
We deﬁne resources as the number of training records, which translates into the number of training iterations after accounting for certain hyperparameters.
For the task tuning a modern LSTM architecture, we use the code provided by Merity et al.
and construct a search space around the hyperparameter setting that they used.
The hyperparameters that we considered along with their associated ranges are shown in Table 3.