A CCELERATING N EURAL A RCHITECTURE S EARCH USING P ERFORMANCE P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Methods for neural network hyperparameter optimization and meta-modeling are computationally expensive due to the need to train a large number of model conﬁgurations.
In this paper, we show that standard frequentist regression models can predict the ﬁnal performance of partially trained model conﬁgurations using features based on network architectures, hyperparameters, and time-series validation performance data.
We empirically show that our performance prediction models are much more effective than prominent Bayesian counterparts, are simpler to implement, and are faster to train.
Our models can predict ﬁnal performance in both visual classiﬁcation and language modeling domains, are effective for predicting performance of drastically varying model architectures, and can even generalize between model classes.
Using these prediction models, we also propose an early stopping method for hyperparameter optimization and meta-modeling, which obtains a speedup of a factor up to 6x in both hyperparameter optimization and meta-modeling.
Finally, we empirically show that our early stopping method can be seamlessly incorporated into both reinforcement learning-based architecture selection algorithms and bandit based search methods.
Through extensive experimentation, we empirically show our performance prediction models and early stopping algorithm are state-of-the-art in terms of prediction accuracy and speedup achieved while still identifying the optimal model conﬁgurations.
I NTRODUCTION
At present, signiﬁcant human expertise and labor is required for designing high-performing neural network architectures and successfully training them for different applications.
Ongoing research in two areas—meta-modeling and hyperparameter optimization—attempts to reduce the amount of human intervention required for these tasks.
Hyperparameter optimization methods (e g., Hutter et al.
; Snoek et al.
; Li et al.
(2017)) focus primarily on obtaining good optimization hyperparameter conﬁgurations for training human-designed networks, whereas meta-modeling algorithms (Bergstra et al., 2013; Verbancsics & Harguess, 2013; Baker et al., 2017; Zoph & Le, 2017) aim to design neural network architectures from scratch.
Both sets of algorithms require training a large number of neural network conﬁgurations for identifying the right set of hyperparameters or the right network architecture—and are hence computationally expensive.
When sampling many different model conﬁgurations, it is likely that many subpar conﬁgurations will be explored.
Human experts are quite adept at recognizing and terminating suboptimal model conﬁgurations by inspecting their partial learning curves.
In this paper we seek to emulate this behavior and automatically identify and terminate subpar model conﬁgurations in order to speedup both meta-modeling and hyperparameter optimization methods.
Our method parameterizes learning curve trajectories with simple features derived from model architectures, training hyperparameters, and early time-series measurements from the learning curve.
We use these features to train a set of frequentist regression models that predicts the ﬁnal validation accuracy of partially trained neural network conﬁgurations using a small training set of fully trained curves from both image classiﬁcation and language modeling domains.
We use these predictions and uncertainty estimates obtained from small model ensembles to construct a simple early stopping algorithm that can speedup both meta-modeling and hyperparameter optimization methods.
While there is some prior work on neural network performance prediction using Bayesian methods (Domhan et al., 2015; Klein et al., 2017), our proposed method is signiﬁcantly more accurate,
[Figure]
accessible, and efﬁcient.
We hope that our work leads to inclusion of neural network performance prediction and early stopping in the practical neural network training pipeline.
R ELATED W ORK
Neural Network Performance Prediction: There has been limited work on predicting neural network performance during the training process.
Domhan et al.
(2015) introduce a weighted probabilistic model for learning curves and utilize this model for speeding up hyperparameter search in small convolutional neural networks (CNNs) and fully-connected networks (FCNs).
Building on Domhan et al.
, Klein et al.
(2017) train Bayesian neural networks for predicting unobserved learning curves using a training set of fully and partially observed learning curves.
Both methods rely on expensive Markov chain Monte Carlo (MCMC) sampling procedures and handcrafted learning curve basis functions.
We also note that Swersky et al.
(2014) develop a Gaussian Process kernel for predicting individual learning curves, which they use to automatically stop and restart conﬁgurations.
Meta-modeling: We deﬁne meta-modeling as an algorithmic approach for designing neural network architectures from scratch.
The earliest meta-modeling approaches were based on genetic algorithms (Schaffer et al., 1992; Stanley & Miikkulainen, 2002; Verbancsics & Harguess, 2013) or
Bayesian optimization (Bergstra et al., 2013; Shahriari et al., 2016).
More recently, reinforcement learning methods have become popular.
Baker et al.
(2017) use Q-learning to design competitive
CNNs for image classiﬁcation.
Zoph & Le (2017) use policy gradients to design state-of-the-art
CNNs and Recurrent cell architectures.
Several methods for architecture search (Cortes et al., 2017;
Negrinho & Gordon, 2017; Zoph et al., 2017; Brock et al., 2017; Suganuma et al., 2017) have been proposed this year since the publication of Baker et al.
and Zoph & Le (2017).
Hyperparameter Optimization: We deﬁne hyperparameter optimization as an algorithmic approach for ﬁnding optimal values of design-independent hyperparameters such as learning rate and batch size, along with a limited search through the network design space.
Bayesian hyperparameter optimization methods include those based on sequential model-based optimization (SMAC)
(Hutter et al., 2011), Gaussian processes (GP) (Snoek et al., 2012), TPE (Bergstra et al., 2013), and neural networks Snoek et al.
However, random search or grid search is most commonly used in practical settings (Bergstra & Bengio, 2012).
Recently, Li et al.
(2017) introduced Hyperband, a multiarmed bandit-based efﬁcient random search technique that outperforms state-of-the-art Bayesian optimization methods.
N EURAL N ETWORK P ERFORMANCE P REDICTION
We ﬁrst describe our model for neural network performance prediction, followed by a description of the datasets used to evaluate our model, and ﬁnally present experimental results.
3. M ODELING L EARNING C URVES
Our goal is to model the validation accuracy y T of a neural network conﬁguration x ∈ X ⊂ R d at epoch T ∈
Z + using previous performance observations y ( t ) .
For each conﬁguration x trained for T epochs, we record a time-series y ( T ) =
y 1 , y 2 , . .
, y T of validation accuracies.
We train a population of n conﬁgurations, obtaining a set S =
{ ( x 1 , y 1 ( t )) , ( x 2 , y 2 ( t )) , . .
, ( x n , y n ( t )) } .
Note that this problem formulation is very similar to Klein et al.
We propose to use a set of features u x , derived from the neural network conﬁguration x , along with a subset of time-series accuracies y ( τ ) =
( y t ) t =1 , 2 ,...,τ (where 1 ≤ τ < T ) from S to train a regression model for estimating y T .
Our model predicts y T of a neural network conﬁguration using a feature set x f = { u x , y ( t ) 1– τ } .
For clarity, we train T − 1 regression models, where each successive model uses one more point of the time-series validation data.
As we shall see in subsequent sections, this use of sequential regression models (SRM) is more computationally and more precise than methods that train a single Bayesian model.
Features: We use features based on time-series (TS) validation accuracies, architecture parameters (AP), and hyperparameters (HP).
(1) TS: These include the validation accuracies y ( t ) 1– τ = ( y t ) t =1 , 2 ,...,τ (where 1 ≤ τ < T ), the ﬁrst-order differences of validation accuracies (i.e., y t (cid:48) =
( y t − y t − 1 ) ), and the second-order differences of validation accuracies (i.e., y t (cid:48)(cid:48) =
( y t (cid:48) − y (cid:48)
t − 1 ) ).
(2) AP: These include total number of weights and number of layers.
(3) HP: These include all hyperparameters used for training the neural networks, e.g., initial learning rate and learning rate decay (full list in Appendix Table 2).
3. D ATASETS AND T RAINING P ROCEDURES
We experiment with small and very deep CNNs (e.g., ResNet, Cuda-Convnet) trained on image classiﬁcation datasets and with LSTMs trained with Penn Treebank (PTB), a language modeling dataset.
Figure 1 shows example learning curves from three of the datasets considered in our experiments.
We provide brief summary of the datasets below.
Please see Appendix Section A for further details on the search space, preprocessing, hyperparameters and training settings of all datasets.
Datasets with Varying Architectures:
Deep Resnets (TinyImageNet): We sample 500 ResNet architectures and train them on the TinyIm - ageNet * dataset (containing 200 classes with 500 training images of 32 × 32 pixels for 140 epochs.
We vary depths
ﬁlter sizes
and
number of convolutional
ﬁlter
block outputs.
The network depths vary
between 14 and
Deep Resnets (CIFAR-10):
We sample 500 39-layer ResNet architectures from a search space similar to Zoph & Le (2017), varying kernel width, kernel height, and number of kernels.
We train these models for 50 epochs on CIFAR-10.
MetaQNN CNNs (CIFAR-10 and SVHN): We sample 1,000 model architectures from the search space detailed by Baker et al.
(2017), which allows for varying the numbers and orderings of convolution, pooling, and fully connected layers.
The models are between 1 and 12 layers for the
SVHN experiment and between 1 and 18 layers for the CIFAR-10 experiment.
Each architecture is trained on SVHN and CIFAR-10 datasets for 20 epochs.
LSTM (PTB): We sample 300 LSTM models and train them on the Penn Treebank dataset forepochs, evaluating perplexity on the validation set.
We vary number of LSTM cells and hidden layer inputs between 10-1400.
Datasets with Varying Hyperparameters:
Cuda-Convnet (CIFAR-10 and SVHN): We train Cuda-Convnet architecture (Krizhevsky, 2012) with varying values of initial learning rate, learning rate reduction step size, weight decay for convolutional and fully connected layers, and scale and power of local response normalization layers.
We train models with CIFAR-10 for 60 epochs and with SVHN for 12 epochs.
3. P REDICTION P ERFORMANCE
Choice of Regression Method: We now describe our results for performing ﬁnal neural network performance.
For all experiments, we train our SRMs on 100 randomly sampled neural network conﬁgurations.
We obtain the best performing method using random hyperparameter search over 3-fold cross-validation.
We then compute the regression performance over the remainder of the [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
and perplexity for LSTMs. dataset using the coefﬁcient of determination R 2 .
We repeat each experiment 10 times and report the results with standard errors.
We experiment with a few different frequentist regression models, including ordinary least squares (OLS), random forests, and ν -support vector machine regressions ( ν -SVR).
As seen in Table 1, ν -SVR with linear or RBF kernels perform the best on most datasets, though not by a large margin.
For the rest of this paper, we use ν -SVR RBF unless otherwise speciﬁed.
Ablation Study on Feature Sets: In Table 2, we compare the predictive ability of different feature sets, training SVR (RBF) with time-series (TS) features obtained from 25% of the learning curve, along with features of architecture parameters (AP), and hyperparameters (HP).
TS features explain the largest fraction of the variance in all cases.
For datasets with varying architectures, AP are more important that HP; and for hyperparameter search datasets, HP are more important than AP, which is expected.
AP features almost match TS on the ResNet (TinyImageNet) dataset, indicating that choice of architecture has a large inﬂuence on accuracy for ResNets.
Figure 2 shows the true vs. predicted performance for all test points in three datasets, trained with TS, AP, and HP features.
Generalization Between Depths: We also test to see whether SRMs can accurately predict the performance of out-of-distribution neural networks.
In particular, we train SVR (RBF) with 25% of TS, along with AP and HP features on ResNets (TinyImagenet) dataset, using 100 models with number of layers less than a threshold d and test on models with number of layers greater than d , averaging over 10 runs.
Value of d varies from 14 to 110.
For d = 32 , R 2 is 80 .
66 ± 3 . 8 .
For d = 62 ,
R 2 is 84 .
58 ± 2 . 7 .
3.3. C OMPARISON WITH E XISTING M ETHODS :
We now compare the neural network performance prediction ability of SRMs with three existing learning curve prediction methods: (1) Bayesian Neural Network (BNN) (Klein et al., 2017), (2) the learning curve extrapolation (LCE) method (Domhan et al., 2015), and (3) the last seen value (LastSeenValue) heuristic (Li et al., 2017).
When training the BNN, we not only present it with the subset of fully observed learning curves but also all other partially observed learning curves from the training set.
While we do not present the partially observed curves to the ν -SVR SRM for training, [Table] we felt this was a fair comparison as ν -SVR uses the entire partially observed learning curve during inference.
Methods (2) and (3) do not incorporate prior learning curves during training.
Figure 3 shows the R 2 obtained by each method for predicting the ﬁnal performance versus the percent of the learning curve used for training the model.
We see that in all neural network conﬁguration spaces and across all datasets, either one or both SRMs outperform the competing methods.
The
LastSeenValue heuristic only becomes viable when the conﬁgurations are near convergence, and its performance is worse than an SRM for very deep models.
We also ﬁnd that the SRMs outperform the
LCE method in all experiments, even after we remove a few extreme prediction outliers produced by LCE.
Finally, while BNN outperforms the LastSeenValue and LCE methods when only a few iterations have been observed, it does worse than our proposed method.
In summary, we show that our simple, frequentist SRMs outperforms existing Bayesian approaches on predicting neural network performance on modern, very deep models in computer vision and language modeling tasks.
Since most of our experiments perform stepwise learning rate decay; it is conceivable that the performance gap between SRMs and both LCE and BNN results from a lack of sharp jump in their basis functions.
We experimented with exponential learning rate decay (ELRD), which the basis functions in LCE are designed for.
We trained 630 random nets with ELRD, from the
MetaQNN-CIFAR10 nets.
Predicting from 25% of the learning curve, the R 2 is 0.95 for ν -SVR (RBF), 0.48 for LCE (with extreme outlier removal, negative without), and 0.31 for BNN.
This comparison illuminates another beneﬁt of our method: we do not require handcrafted basis functions to model new learning curve types.
Training and Inference Speed Comparison: Another advantage of our regression approach is speed.
SRMs are much faster to train and do inference in than proposed Bayesian methods (Domhan et al., 2015; Klein et al., 2017).
On 1 core of a Intel 6700k CPU, an ν -SVR (RBF) with 100 training points trains in 0.006 seconds, and each inference takes 0.00006 seconds.
In comparison, the LCE code takes 60 seconds and BNN code takes 0.024 seconds on the same hardware for each inference.
A PPLYING P ERFORMANCE P REDICTION F OR E ARLY S TOPPING
To speed up hyperparameter optimization and meta-modeling methods, we develop an algorithm to determine whether to continue training a partially trained model conﬁguration using our sequential regression models.
If we would like to sample N total neural network conﬁgurations, we begin by sampling and training n (cid:28) N conﬁgurations to create a training set S .
We then train a model f ( x f ) to predict y T .
Now, given the current best performance observed y BEST , we would like to terminate training a new conﬁguration x (cid:48) given its partial learning curve y (cid:48)
( t ) 1– τ if f ( x f (cid:48) )
=
ˆ y T ≤ y BEST
so as to not waste computational resources exploring a suboptimal conﬁguration.
However, in the case f ( x f ) has poor out-of-sample generalization, we may mistakenly terminate the optimal conﬁguration.
If we assume that our estimate can be modeled as a Gaussian perturbation of the true value ˆ y T ∼ N ( y T , σ ( x , τ )) , then we can ﬁnd the probability p (ˆ y T ≤ y BEST | σ ( x , τ )) =
Φ ( y BEST ; y T , σ ) , where Φ ( · ; µ, σ ) is the CDF of N ( µ, σ ) .
Note that in general the uncertainty will depend on both the conﬁguration and τ , the number of points observed from the learning curve.
Because frequentist models do not admit a natural estimate of uncertainty, we assume that σ is independent of x yet still dependent on τ and estimate it via Leave One Out Cross Validation.
[Figure]
Now that we can estimate the model uncertainty, given a new conﬁguration x (cid:48) and an observed learning curve y (cid:48) ( t ) 1– τ , we may set our termination criteria to be p (ˆ y T ≤ y BEST ) ≥ ∆ .
∆ balances the trade-off between increased speedups and risk of prematurely terminating good conﬁgurations.
In many cases, one may want several conﬁgurations that are close to optimal, for the purpose of ensembling.
We offer two modiﬁcations in this case.
First, one may relax the termination criterion to p (ˆ y T ≤ y BEST − δ ) ≥ ∆ , which will allow conﬁgurations within δ of optimal performance to complete training.
One can alternatively set the criterion based on the n th best conﬁguration observed, guaranteeing that with high probability the top n conﬁgurations will be fully trained.
4. E ARLY S TOPPING FOR M ETA - MODELING
Baker et al.
(2017) train a Q -learning agent to design convolutional neural networks.
In this method, the agent samples architectures from a large, ﬁnite space by traversing a path from input layer to termination layer.
However, the MetaQNN method uses 100 GPU-days to train 2700 neural architectures and the similar experiment by Zoph & Le (2017) utilized 10,000 GPU-days to train 12,800 models on CIFAR-10.
The amount of computing resources required for these approaches makes them prohibitively expensive for large datasets (e.g., Imagenet) and larger search spaces.
The main computational expense of reinforcement learning-based meta-modeling methods is training the neural network conﬁguration to T epochs (where T is typically a large number at which the network stabilizes to peak accuracy).
We now detail the performance of a ν -SVR (RBF) SRM in speeding up architecture search using sequential conﬁguration selection.
First, we take 1,000 random models from the MetaQNN (Baker et al., 2017) search space.
We simulate the MetaQNN algorithm by taking 10 random orderings of each set and running our early stopping algorithm.
We compare against the LCE early stopping algorithm (Domhan et al., 2015) as a baseline, which has a similar probability threshold termination criterion.
Our SRM trains off of the ﬁrst 100 fully observed curves, while the LCE model trains from each individual partial curve and can begin early termination immediately.
Despite this “burn in” time needed by an SRM, it is still able to signiﬁcantly outperform the LCE model (Figure 4).
In addition, ﬁtting the LCE model to a learning curve takes between 1-3 minutes on a modern CPU due to expensive MCMC sampling, and it is necessary to ﬁt a new LCE model each time a new point on the learning curve is observed.
Therefore, on a full meta-modeling experiment involving thousands
[Figure]
of neural network conﬁgurations, our method could be faster by several orders of magnitude as compared to LCE based on current implementations.
[Figure]
that the agent continues to ﬁnd better architectures.
We furthermore simulate early stopping for ResNets trained on CIFAR-10.
We found that only the probability threshold ∆ = 0 . 99 resulted in recovering the top model consistently.
However, even with such a conservative threshold, the search was sped up by a factor of 3.4 over the baseline.
While we do not have the computational resources to run the full experiment from Zoph & Le (2017), our method could provide similar gains in large scale architecture searches.
It is not enough, however, to simply simulate the speedup because meta-modeling algorithms typically use the observed performance in order to update an acquisition function to inform future sampling.
In the reinforcement learning setting, the performance is given to the agent as a reward, so we also empirically verify that substituting ˆ y T for y T does not cause the MetaQNN agent to converge to a subpar policy.
Replicating the MetaQNN experiment on
CIFAR-10 (see Figure 5), we ﬁnd that integrating early stopping with the Q -learning procedure does not disrupt learning and resulted in a speedup of 3.8x with ∆ = 0 . 99 .
The speedup is relatively low due to a conservative value of ∆ .
After training the top models to 300 epochs, we also ﬁnd that the resulting performance (just under 93%) is on par with original results of Baker et al.
4. E ARLY S TOPPING FOR H YPERPARAMETER O PTIMIZATION
Recently, Li et al.
(2017) introduced Hyperband, a random search technique based on multi-armed bandits that obtains state-of-the-art performance in hyperparameter optimization in a variety of settings.
The Hyperband algorithm trains a population of models with different hyperparameter conﬁgurations and iteratively discards models below a certain percentile in performance among the population until the computational budget is exhausted or satisfactory results are obtained.
[Figure]
C u m u l a t
i v e
M a x i m u m
V a li d a t i o n A cc
[Figure]
r
[Figure]
4.2. F AST H YPERBAND
We present a Fast Hyperband (f-Hyperband) algorithm based on our early stopping scheme.
During each iteration of successive halving, Hyperband trains n i conﬁgurations to r i epochs.
In f-Hyperband, we train an SRM to predict y r i and do early stopping within each iteration of successive halving.
We initialize f-Hyperband in exactly the same way as vanilla Hyperband, except once we have trained 100 models to r i iterations, we begin early stopping for all future successive halving iterations that train to r i iterations.
By doing this, we exhibit no initial slowdown to Hyperband due to a “burn-in” phase.
We also introduce a parameter κ which denotes the proportion of the n i models in each iteration that must be trained to the full r i iterations.
This is similar to setting the criterion based on the n th best model in the previous section.
See Appendix section C for an algorithmic representation of f-Hyperband.
We empirically evaluate f-Hyperband using Cuda-Convnet trained on CIFAR-10 and SVHN datasets.
Figure 6 shows that f-Hyperband evaluates the same number of unique conﬁgurations as Hyperband within half the compute time, while achieving the same ﬁnal accuracy within standard error.
When reinitializing hyperparameter searches, one can use previously-trained set of SRMs to achieve even larger speedups.
Figure 8 in Appendix shows that one can achieve up to a 7x speedup in such cases.
C ONCLUSION
In this paper we introduce a simple, fast, and accurate model for predicting future neural network performance using features derived from network architectures, hyperparameters, and time-series performance data.
We show that the performance of drastically different network architectures can be jointly learned and predicted on both image classiﬁcation and language models.
Using our simple algorithm, we can speedup hyperparameter search techniques with complex acquisition functions, such as a Q -learning agent, by a factor of 3x to 6x and Hyperband—a state-of-the-art hyperparameter search method—by a factor of 2x, without disturbing the search procedure.
We outperform all competing methods for performance prediction in terms of accuracy, train and test time, and speedups obtained on hyperparameter search methods.
We hope that the simplicity and success of our method will allow it to be easily incorporated into current hyperparameter optimization pipelines for deep neural networks.
With the advent of large scale automated architecture search (Baker et al., 2017;
Zoph & Le, 2017), methods such as ours will be vital in exploring even larger and more complex search spaces.
A PPENDIX A D ATASETS AND ARCHITECTURES
Deep Resnets (TinyImageNet): We sample 500 ResNet architectures and train them on the TinyImageNet † dataset (containing 200 classes with 500 training images of 32 × 32 pixels) for 140 epochs.
We vary depths, ﬁlter sizes and number of convolutional ﬁlter block outputs.
Filter sizes are sampled from { 3 , 5 , 7 } and number of ﬁlters is sampled from { 2 , 3 , 4 , ..., 22 } .
Each ResNet block is composed of three convolutional layers followed by batch normalization and summation layers.
We vary the number of blocks from 2 to 18, giving us networks with depths varying between 14 and 110.
Each network is trained for 140 epochs, using Nesterov optimizer.
The learning rate is set to 0.1 and learning rate reduction and momentum are set to 0.1 and 0.9 respectively.
Deep Resnets (CIFAR-10):
We sample 500 39-layer ResNet architectures from a search space similar to Zoph & Le (2017), varying kernel width, kernel height, and number of kernels.
We train these models for 50 epochs on CIFAR-10.
Each architecture consists of 39 layers: 12 conv , a 2x max pool , 9 conv , a 2x2 max pool , 15 conv , and softmax .
Each conv layer is followed by batch normalization and a ReLU nonlinearity.
Each block of 3 conv layers are densely connected via residual connections and also share the same kernel width, kernel height, and number of learnable kernels.
Kernel height and width are independently sampled from { 1 , 3 , 5 , 7 } and number of kernels is sampled from { 6 , 12 , 24 , 36 } .
Finally, we randomly sample residual connections between each block of conv layers.
Each network is trained for 50 epochs using the RMSProp optimizer, with weight decay 10 − 4 , initial learning rate 0.001, and a learning rate reduction to 10 − 5 at epoch 30 on the CIFAR-10 dataset.
MetaQNN CNNs (CIFAR-10 and SVHN): We sample 1,000 model architectures from the search space detailed by Baker et al.
(2017), which allows for varying the numbers and orderings of convolution, pooling, and fully connected layers.
The models are between 1 and 12 layers for the
SVHN experiment and between 1 and 18 layers for the CIFAR-10 experiment.
Each architecture is trained on SVHN and CIFAR-10 datasets for 20 epochs.
Table 3 displays the state space of the
MetaQNN algorithm
[Table]
(PTB): We sample 300 LSTM models and train them on the Penn Treebank dataset forepochs.
Number of hidden layer inputs and lstm cells was varied from 10 to 1400 in steps of 20.
Each network was trained for 60 epochs with batch size of 50 and trained the models using stochastic gradient descent.
Dropout ratio of 0.5 was used to prevent overﬁtting.
Dictionary size of 400 words was used to generate embeddings when vectorizing the data.
Cuda-Convnet (CIFAR-10 and SVHN): We train Cuda-Convnet architecture (Krizhevsky, 2012) with varying values of initial learning rate, learning rate reduction step size, weight decay for convolutional and fully connected layers, and scale and power of local response normalization layers.
We train models with CIFAR-10 for 60 epochs and with SVHN for 12 epochs.
Table 4 show the hyperparameter ranges for the Cuda Convnet experiments.
[Table]
CIFAR-10, Imagenet, SVHN
[Table]
B H YPERPARAMETER SELECTION IN R ANDOM F OREST AND SVM BASED EXPERIMENTS
When training SVM and Random Forest we divided the data into training and validation and used cross validation techniques to select optimal hyperparameters.
The SVM and RF model was then trained on full training data using the best hyperparameters.
For random forests we varied number of trees between 10 and 800, and varied ratio of number of features from 0.1 to 0.5.
For ν -SVR, we perform a random search over 1000 hyperparameter conﬁgurations from the space C ∼ LogUniform( 10 − 5 , 10), ν ∼ Uniform(0, 1), and γ ∼ LogUniform( 10 − 5 , 10) (when using the RBF kernel).
[Figure]
Percent Learning Curve Observed
[Figure]
SVR - Linear
SVR - RBF
BNN
LastSeenValue
LCE
LCE-No Outlier
[Figure]
The major jump in speedup comes at iteration 10, where we have trained more than 100 models to
C F -H YPERBAND
Algorithm 1 of this text replicates Algorithm 1 from Li et al.
(2017), except we initialize two dictionaries: D to store training data and M to store performance prediction models.
D [ r ] will correspond to a dictionary containing all datasets with prediction target epoch r .
D [ r ][ τ ] will correspond to the dataset for predicting y r based on the observed y ( t ) 1 − τ , and M [ r ][ τ ] will hold the corresponding performance prediction model.
We will assume that the performance prediction model will have a train function, and a predict function that will return the prediction and standard deviation of the prediction.
In addition to the standard Hyperband hyperparameters R and η , we include ∆ and δ described in Section 4 and κ .
During each iteration of successive halving, we train n i conﬁgurations to r i epochs; κ denotes the fraction of the top n i models that should be run to the full r i iterations.
This is similar to setting the criterion based on the n th best model in the previous section.
We also detail the run_then_return_validation_loss function in Algorithm 2.
This algorithm runs a set of conﬁgurations, adds training data from observed learning curves, trains the performance prediction models when there is enough training data present, and then uses the models to terminate poor conﬁgurations.
It assumes we have a function max_k input : R – (Max resources allocated to any conﬁguration)
η – (default η = 3 ) ∆ – (Probability threshold for early termination)
δ – (Performance offset for early termination)
d – (# points required to train performance predictors) κ – (Proportion of models to train) initialize :
D = dict()
M = dict() s max = (cid:98) log η
( R ) (cid:99)
B = ( s max + 1) R 1 for s ∈ { s max , . . .
, 0 } do n = (cid:100) BR η s s +1 (cid:101) , r = Rη − s // begin SUCCESSIVEHALVING with ( n , r ) inner loop
T = get_hyperparameter_configuration( n ) for i ∈ { 0 , . . .
, s } do n i = (cid:98)
nη − i (cid:99) , r i = rη i n next =
(cid:98) n i η (cid:99)
if i !
=
s else
L = run_then_return_validation_loss( T, r i , n next , D, M )
T = top_k( T, L, (cid:98) n i η (cid:99) ) end 11 end
Algorithm 2: run return loss input : T – hyperparameter conﬁgurations r – resources to use for training n – # conﬁgurations in next iteration of successive halving
D – dictionary storing training data
M – dictionary storing performance prediction models initialize :
L = [] 1 for t ∈ T do (cid:96) =
[] for i ∈ { 0 , . .
, r − 1 } do (cid:96) i =
run_one_epoch_return_validation_loss( t ) (cid:96) .append
( (cid:96)
if r ][ i
.trained( then ˆ
M r ] i
) if Φ ( max k(
, κn ) − δ ; ˆ y r
) ≥ ∆ then
L .append( ˆ y r ) break end end else if i ==
r − 1 then
L .append( (cid:96) i ) end end if length
( D
) <
d and length then { D [ r
[ i ] .append i ] , (cid:96) [ r ] } ) :
i ∈ r − then
end end 23 end 24 return L