A CCELERATING N EURAL A RCHITECTURE S EARCH USING P ERFORMANCE P REDICTION
A BSTRACT
Methods for neural network hyperparameter optimization and architecture search are computationally expensive due to the need to train a large number of model conﬁgurations.
In this paper, we show that simple regression models can predict the ﬁnal performance of partially trained model conﬁgurations using features based on network architectures, hyperparameters, and time-series validation performance data.
We empirically show that our performance prediction models are much more accurate than prominent Bayesian counterparts, are simpler to implement, and are faster to train.
Our models can predict ﬁnal performance in both visual classiﬁcation and language modeling domains, are effective for predicting performance of drastically varying model architectures, and can even generalize between model classes.
Using these prediction models, we also implement an early stopping method for hyperparameter optimization and architecture search, which obtains a speedup of a factor up to 6x in both hyperparameter optimization and architecture search.
Finally, we empirically show that our early stopping method can be seamlessly incorporated into both reinforcement learning-based architecture selection algorithms and bandit based search methods.
Through extensive experimentation, we empirically show our performance prediction models and early stopping algorithm are state-of-the-art in terms of prediction accuracy and speedup achieved while still identifying the optimal model conﬁgurations.
I NTRODUCTION
Signiﬁcant human expertise and labor is required for designing high-performing neural network architectures and successfully training them for different applications.
Ongoing research in two areas—architecture search and hyperparameter optimization—attempts to reduce the amount of human intervention required for these tasks.
Hyperparameter optimization methods (e g., Hutter et al.
; Snoek et al.
; Li et al.
(2017)) focus primarily on obtaining good optimization hyperparameter conﬁgurations for training human-designed networks, whereas architecture search algorithms (Bergstra et al., 2013; Verbancsics & Harguess, 2013; Baker et al., 2017; Zoph & Le, 2017) aim to design neural network architectures from scratch.
Both sets of algorithms require training a large number of neural network conﬁgurations for identifying the right set of hyperparameters or the right network architecture and are hence computationally expensive.
When sampling many different model conﬁgurations, it is likely that many subpar conﬁgurations will be explored.
Human experts are quite adept at recognizing and terminating suboptimal model conﬁgurations by inspecting their partial learning curves.
In this paper we seek to emulate this behavior and automatically identify and terminate subpar model conﬁgurations in order to speedup both architecture search and hyperparameter optimization methods.
Our method parameterizes learning curve trajectories with simple features derived from model architectures, training hyperparameters, and early time-series measurements from the learning curve.
We use these features to train a set of simple regression models that predict the ﬁnal validation performance of partially trained neural network conﬁgurations using a small training set of fully trained curves, and we empirically validate our [Figure] method in both image classiﬁcation and language modeling domains.
We use these predictions along with uncertainty estimates obtained from small model ensembles to construct a simple early stopping algorithm that can speedup both architecture search and hyperparameter optimization methods.
While there is some prior work on neural network performance prediction using Bayesian methods (Domhan et al., 2015; Klein et al., 2017), our proposed method is signiﬁcantly more accurate, accessible, and efﬁcient.
We hope that our work leads to inclusion of neural network performance prediction and early stopping in the practical neural network training pipeline.
R ELATED W ORK
Neural Network Performance Prediction: There has been limited work on predicting neural network performance during training.
Domhan et al.
(2015) introduce a weighted probabilistic model for learning curves and utilize this model for speeding up hyperparameter search in small convolutional neural networks (CNNs) and fully-connected networks (FCNs).
Building on Domhan et al.
, Klein et al.
(2017) train Bayesian neural networks for predicting unobserved learning curves using a training set of fully and partially observed learning curves.
Both methods rely on expensive Markov chain Monte Carlo (MCMC) sampling procedures and handcrafted learning curve basis functions.
We also note that Swersky et al.
(2014) develop a Gaussian Process kernel for predicting individual learning curves, which they use to automatically stop and restart conﬁgurations.
Architecture Search: We deﬁne architecture search as an algorithmic approach for designing neural network architectures from scratch.
The earliest architecture search approaches were based on genetic algorithms (Schaffer et al., 1992; Stanley & Miikkulainen, 2002; Verbancsics & Harguess, 2013) or
Bayesian optimization (Bergstra et al., 2013; Shahriari et al., 2016).
More recently, reinforcement learning methods have become popular.
Baker et al.
(2017) use Q-learning to design competitive
CNNs for image classiﬁcation.
Zoph & Le (2017) use policy gradients to design state-of-the-art
CNNs and recurrent cell architectures.
Several methods for architecture search (Cortes et al., 2017;
Negrinho & Gordon, 2017; Zoph et al., 2017; Brock et al., 2017; Suganuma et al., 2017) have been proposed this year since the publication of Baker et al.
and Zoph & Le (2017).
Hyperparameter Optimization: We deﬁne hyperparameter optimization as an algorithmic approach for ﬁnding optimal values of architecture independent hyperparameters such as learning rate and batch size, along with a limited search through the network design space.
Bayesian hyperparameter optimization methods include those based on sequential model-based optimization (SMAC)
(Hutter et al., 2011), Gaussian processes (GP) (Snoek et al., 2012), TPE (Bergstra et al., 2013), and neural networks Snoek et al.
However, random search or grid search is most commonly used in practical settings (Bergstra & Bengio, 2012).
Recently, Li et al.
(2017) introduced Hyperband, a multiarmed bandit-based efﬁcient random search technique that outperforms state-of-the-art Bayesian optimization methods.
N EURAL N ETWORK P ERFORMANCE P REDICTION
We ﬁrst describe our model for neural network performance prediction, followed by a description of the datasets used to evaluate our model, and ﬁnally present experimental results.
3. M ODELING L EARNING C URVES
Our goal is to model the validation performance y T of a neural network conﬁguration x ∈ X ⊂ R d at epoch T ∈
Z + using previous performance observations y ( t ) .
For each conﬁguration x trained for T epochs, we record a time-series y ( T ) =
y 1 , y 2 , . .
, y T of validation performances.
We train a population of n conﬁgurations, obtaining a set S =
{ ( x 1 , y 1 ( T )) , ( x 2 , y 2 ( T )) , . . .
, ( x n , y n ( T )) } .
This problem formulation is very similar to Klein et al.
Note that most architecture and hyperparameter search methods naturally collect S .
We propose to use a set of features u x , derived from the neural network conﬁguration x , along with a subset of time-series performances y ( τ ) =
( y t ) t =1 , 2 ,...,τ (where 1 ≤ τ < T ) from S to train a regression model for estimating y T .
We use models without natural support for variable size input, so we train T − 1 regression models, where each successive model uses one more point of the time-series validation data.
As we shall see in subsequent sections, this use of sequential regression models (SRM) is more computationally efﬁcient and more precise than methods that train a single model.
Features: We use features based on time-series (TS) validation performances, architecture parameters (AP), and hyperparameters (HP).
(1) TS: Assume we are training the τ th model in the SRM.
TS features include the validation performances y ( τ ) =
( y t ) t =1 , 2 ,...,τ , the ﬁrst-order differences of validation performances, i.e. y t (cid:48) =
( y t − y t − 1 ) , the second-order differences of validation performances, i.e. y t (cid:48)(cid:48) =
( y t (cid:48) − y (cid:48)
t − 1 ) , and mean and standard deviation of performances.
(2) AP: These include total number of weights and number of layers.
(3) HP: These include all hyperparameters used for training the neural networks, e.g. initial learning rate and learning rate decay (full list in Appendix
Table 4).
3. D ATASETS AND T RAINING P ROCEDURES
Figure 1 shows example learning curves from three of the datasets considered in our experiments.
We provide brief summary of the datasets below.
Please see Appendix Section A for further details on the search space, preprocessing, hyperparameters and training settings of all datasets.
D ATASETS WITH V ARYING A RCHITECTURES :
Deep Resnets (TinyImageNet): We sample 500 ResNet architectures and train them on the TinyImageNet * dataset (containing 200 classes with 500 training images of 32 × 32 pixels) for 140 epochs.
We vary depths, ﬁlter sizes and number of convolutional ﬁlter block outputs.
The network depths vary between 14 and 110.
Deep Resnets (CIFAR-10):
We sample 500 39-layer ResNet architectures from a search space similar to Zoph & Le (2017), varying kernel width, kernel height, and number of kernels.
We train these models for 50 epochs on CIFAR-10.
MetaQNN CNNs (CIFAR-10 and SVHN): We sample 1,000 model architectures from the search space detailed by Baker et al.
(2017), which allows for varying the numbers and orderings of convolution, pooling, and fully connected layers.
The models are between 1 and 12 layers for the
SVHN experiment and between 1 and 18 layers for the CIFAR-10 experiment.
Each architecture is trained on SVHN and CIFAR-10 datasets for 20 epochs.
LSTM (PTB): We sample 300 LSTM models and train them on the Penn Treebank dataset forepochs, evaluating perplexity on the validation set.
We vary number of LSTM cells and hidden layer inputs between 10-1400.
D ATASETS WITH V ARYING H YPERPARAMETERS :
Cuda-Convnet (CIFAR-10 and SVHN): We train Cuda-Convnet architecture (Krizhevsky, 2012) with varying values of initial learning rate, learning rate reduction step size, weight decay for convolutional and fully connected layers, and scale and power of local response normalization layers.
We train models with CIFAR-10 for 60 epochs and with SVHN for 12 epochs. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
3. P REDICTION P ERFORMANCE
Choice of Regression Method: We now describe our results for predicting ﬁnal neural network performance.
For all experiments, we train our SRMs on 100 randomly sampled neural network conﬁgurations.
We obtain the best performing method using random hyperparameter search over 3-fold cross-validation.
We then compute the regression performance over the remainder of the dataset using the coefﬁcient of determination R 2 .
We repeat each experiment 10 times and report the results with standard errors.
We experiment with a few simple regression models, including ordinary least squares (OLS), random forests (RF), Bayesian linear regression (BLR), and ν -support vector machine regression ( ν -SVR).
For both OLS and ν -SVR, we choose between the linear and
RBF kernels based on prediction performance—after hyperparameter search, in case of ν -SVR.
We chose 100 as the number of sampled conﬁgurations somewhat arbitrarily, though we found that prediction performance was quite poor below 50 and improves steadily after.
As seen in Table 1, ν -SVR performs the best on most datasets, though not by a large margin.
For the rest of this paper, we use ν -SVR unless otherwise speciﬁed.
Ablation Study on Feature Sets: In Table 2, we compare the predictive ability of different feature sets, training SVR with time-series (TS) features obtained from 25% of the learning curve, along with features of architecture parameters (AP), and hyperparameters (HP).
TS features explain the largest fraction of the variance in all cases and found that in general AP are more important that
HP.
AP features almost match TS on the ResNet (TinyImageNet) dataset, indicating that choice of architecture has a large inﬂuence on accuracy for ResNets.
Figure 2 shows the true vs. predicted performance for all test points in three datasets, trained with TS, AP, and HP features.
Generalization To Out Of Distribution Conﬁgurations: We also test to see whether our SRMs can accurately predict the performance of out-of-distribution neural networks.
For instance, if we train a ν -SVR model on the Deep Resnet (TinyImagenet) dataset at 25% of the learning curve observed on conﬁgurations below median depth (60 layers), we can achieve R 2 = 0 .
86 on conﬁgurations above median depth.
The corresponding number for prediction below median depth is 0.89.
We repeated this experiment all datasets with different hyperparameter values below/above the median and found that SRMs generally showed consistent performance across such splits.
Full results can be found in
Appendix Section H.
[Table]
3.3. C OMPARISON WITH E XISTING M ETHODS :
We now compare the neural network performance prediction ability of SRMs with three existing learning curve prediction methods: (1) Bayesian Neural Network (BNN) (Klein et al., 2017), (2) the learning curve extrapolation (LCE) method (Domhan et al., 2015), and (3) the last seen value (LastSeenValue) heuristic (Li et al., 2017).
When training the BNN, we not only present it with the subset of fully observed learning curves but also all other partially observed learning curves from the training set.
While we do not present the partially observed curves to the SRMs for training, we felt this was a fair comparison as an SRM uses the entire partially observed learning curve during inference.
Methods (2) and (3) do not incorporate prior learning curves during training.
Figure 3 shows the R 2 obtained by each method for predicting the ﬁnal performance versus the percent of the learning curve used for training the model.
We see that in all neural network conﬁguration spaces and across all datasets, ν -SVR outperform the competing methods.
Bayesian linear regression (BLR) also performs comparably well to ν -SVR and better than the more complex Bayesian methods (BNN and LCE).
The LastSeenValue heuristic only becomes viable when the conﬁgurations are near convergence, and its performance is worse than an SRM for very deep models.
We also ﬁnd that the SRMs outperform the LCE method in all experiments, even after we remove a few extreme prediction outliers produced by LCE.
Finally, while BNN outperforms the LastSeenValue and LCE methods when only a few iterations have been observed, it does worse than our proposed method.
In summary, we show that our simple SRMs outperforms existing Bayesian approaches on predicting neural network performance on modern, very deep models in computer vision and language modeling tasks.
Since most of our experiments perform stepwise learning rate decay; it is conceivable that the performance gap between SRMs and both LCE and BNN results from a lack of sharp jumps in their basis functions.
For completeness, we also experimented with exponential learning rate decay (ELRD), which the basis functions in LCE are designed for.
We trained 630 random nets with ELRD, from the 1000 MetaQNN-CIFAR10 nets.
Predicting from 25% of the learning curve, the R 2 is 0.
for ν -SVR, 0.48 for LCE (with extreme outlier removal, negative without), and 0.31 for BNN.
This comparison illuminates another beneﬁt of our method: we do not require handcrafted basis functions to model new learning curve types.
Moreover, SRMs are faster to train and do inference in than LCE and BNN.
On 1 core of a Intel 6700k CPU, an ν -SVR with 100 training points trains in 0.006 seconds, and each inference takes 0.00006 seconds.
In comparison, the LCE code takes 60 seconds and BNN code takes 0.024 seconds on the same hardware for each inference.
A PPLYING P ERFORMANCE P REDICTION F OR E ARLY S TOPPING
To speed up hyperparameter optimization and architecture search methods, we develop an algorithm to determine whether to continue training a partially trained model conﬁguration using our sequential regression models.
This approach follows the idea proposed by Domhan et al.
and further expanded on by Klein et al.
Concretely, if we would like to sample N total neural network [Figure] conﬁgurations, we begin by sampling and training n (cid:28) N conﬁgurations to create a training set S .
We then train a model f ( x f ) , where in our case f is an SRM with T − 1 individual models, to predict y T .
Now, given the current best performance observed y BEST , we would like to terminate training a new conﬁguration x NEW at iteration τ given its partial learning curve y NEW ( τ ) if f ( x NEW f )
=
ˆ y T ≤ y BEST
so as to not waste computational resources exploring a suboptimal conﬁguration.
However, in the case f has poor out-of-sample generalization, we may mistakenly terminate the optimal conﬁguration.
If we assume that our estimate can be modeled as a Gaussian perturbation of the true value ˆ y T ∼ N ( y T , σ ( x , τ )) , then we can ﬁnd the probability p (ˆ y T ≤ y BEST | σ ( x , τ )) =
Φ ( y BEST ; y T , σ ) , where Φ ( · ; µ, σ ) is the CDF of N ( µ, σ ) .
Note that in general the uncertainty will depend on both the conﬁguration and τ , the number of points observed from the learning curve.
Because frequentist models do not admit a natural estimate of uncertainty, we assume that σ is independent of x yet still dependent on τ and estimate it via Leave One Out Cross Validation (LOOCV).
In addition, we show results using a Bayesian linear regression SRM, which has natural uncertainty estimates, removing the need for LOOCV.
Now that we can estimate the model uncertainty, given a new conﬁguration x NEW and an observed learning curve y NEW ( τ ) , we may set our termination criteria to be p (ˆ y T ≤ y BEST ) ≥ ∆ .
∆ balances the trade-off between increased speedups and risk of prematurely terminating good conﬁgurations.
In many cases, one may want several conﬁgurations that are close to optimal, for the purpose of ensembling.
We offer two modiﬁcations in this case.
First, one may relax the termination criterion to p (ˆ y T ≤ y BEST − δ ) ≥ ∆ , which will allow conﬁgurations within δ of optimal performance to complete training.
One can alternatively set the criterion based on the n th best conﬁguration observed, guaranteeing that with high probability the top n conﬁgurations will be fully trained.
4. E ARLY S TOPPING FOR A RCHITECTURE S EARCH
Baker et al.
(2017) train a Q -learning agent to design convolutional neural networks.
In this method, the agent samples architectures from a large, ﬁnite space by traversing a path from input layer to termination layer.
However, the MetaQNN method uses 100 GPU-days to train 2700 neural architectures and the similar experiment by Zoph & Le (2017) utilized 10,000 GPU-days to train 12,800 models on CIFAR-10.
The amount of computing resources required for these approaches [Figure] makes them prohibitively expensive for large datasets (e.g., Imagenet) and larger search spaces.
The main computational expense of reinforcement learning-based architecture search methods is training the neural network conﬁguration to T epochs (where T is typically a large number at which the network stabilizes to peak accuracy).
We now detail the performance of ν -SVR and BLR SRMs in speeding up architecture search.
First, we take 1,000 random models from the MetaQNN (Baker et al., 2017) search space.
We simulate the MetaQNN algorithm by taking 10 random orderings of each set and running our early stopping algorithm.
We compare against the LCE early stopping algorithm (Domhan et al., 2015) as a baseline, which has a similar probability threshold termination criterion.
Our SRM trains off of the ﬁrstfully observed curves, while the LCE model trains from each individual partial curve and can begin early termination immediately.
Despite this “burn in” time needed by an SRM, it is still able to outperform the LCE model (Figure 4).
In addition, ﬁtting the LCE model to a learning curve takes between 1-3 minutes on a modern CPU due to expensive MCMC sampling, and it is necessary to ﬁt a new LCE model each time a new point on the learning curve is observed.
However, the difference in computing time between the methods is not as important as the difference in prediction accuracies.
[Figure]
also plot the cumulative best,
, and
15 performance to show that the agent continues to ﬁnd better architectures.
We furthermore simulate early stopping for ResNets trained on CIFAR-10.
We found that only the probability threshold ∆ = 0 . 99 resulted in recovering the top model consistently.
However, even with such a conservative threshold, the search was sped up by a factor of 3.4 over the baseline.
While we do not have the computational resources to run the full experiment from Zoph & Le (2017), our method could provide similar gains in large scale architecture searches.
It is not enough, however, to simply simulate the speedup because architecture search algorithms typically use the observed performance in order to update an acquisition function to inform future sampling.
In the reinforcement learning setting, the performance is given to the agent as a reward, so we also empirically verify that substituting ˆ y T for y T does not cause the MetaQNN agent to converge to a subpar policy.
Replicating the MetaQNN experiment on CIFAR- (see Figure 5), we ﬁnd that integrating early stopping with the Q -learning procedure does not disrupt learning and resulted in a speedup of 3.8x with ∆ = 0 . 99 .
After training the top models to 300 epochs, we also
[Figure]
C u m u l a t
i v e
M a x i m u m
V a li d a t i o n A cc u r a
[Figure]
par with original results of Baker et al.
4. E ARLY S TOPPING FOR H YPERPARAMETER O PTIMIZATION
Recently, Li et al.
(2017) introduced Hyperband, a random search technique based on multi-armed bandits that obtains state-of-the-art performance in hyperparameter optimization in a variety of settings.
The Hyperband algorithm trains a population of models with different hyperparameter conﬁgurations and iteratively discards models below a certain percentile in performance among the population until the computational budget is exhausted or satisfactory results are obtained.
Among previous work, Klein et al.
(2017) have improved Hyperband by algorithmically choosing the next models to evaluate.
4.2. F AST H YPERBAND
We present a Fast Hyperband (f-Hyperband) algorithm based on our early stopping scheme.
During each iteration of successive halving, Hyperband trains n i conﬁgurations to r i epochs.
In f-Hyperband, we train an SRM to predict y r i and do early stopping within each iteration of successive halving.
We initialize f-Hyperband in exactly the same way as vanilla Hyperband, except once we have trained 100 models to r i iterations, we begin early stopping for all future successive halving iterations that train to r i iterations.
By doing this, we exhibit no computational overhead over Hyperband other than training the SRMs, which we have shown to be cheap.
We also introduce a parameter κ which denotes the proportion of the n i models in each iteration that must be trained to the full r i iterations.
This is similar to setting the criterion based on the n th best model in the previous section.
See Appendix section C for the full algorithmic representation of f-Hyperband.
We empirically evaluate f-Hyperband using Cuda-Convnet trained on CIFAR-10 and SVHN datasets.
Figure 6 shows that f-Hyperband evaluates the same number of unique conﬁgurations as Hyperband within half the compute time, while achieving the same ﬁnal accuracy within standard error.
Moreover, when we run f-Hyperband for the same amount of time as Hyperband, f-Hyperband with a ν -SVR
SRM outperforms f-Hyperband.
When reinitializing hyperparameter searches, one can use previouslytrained set of SRMs to achieve even larger speedups.
Figure 12 in Appendix shows that one can safely achieve up to a 4x speedup in such cases.
C ONCLUSION
In this paper we introduce a simple, fast, and accurate model for predicting future neural network performance using features derived from network architectures, hyperparameters, and time-series performance data.
We show that the performance of drastically different network architectures can be jointly learned and predicted on both image classiﬁcation and language models.
Using our simple algorithm, we can speedup hyperparameter search techniques with complex acquisition functions, such as a Q -learning agent, by a factor of 3x to 6x and Hyperband—a state-of-the-art hyperparameter search method—by a factor of 2x, without disturbing the search procedure.
We outperform all competing methods for performance prediction in terms of accuracy, train and test time, and speedups obtained on hyperparameter search methods.
We hope that the simplicity and success of our method will allow it to be easily incorporated into current hyperparameter optimization pipelines for deep neural networks.
With the advent of large scale automated architecture search (Baker et al., 2017;
Zoph & Le, 2017), methods such as ours will be vital in exploring even larger and more complex search spaces
A PPENDIX A D ATASETS AND ARCHITECTURES
Deep Resnets (TinyImageNet): We sample 500 ResNet architectures and train them on the TinyImageNet † dataset (containing 200 classes with 500 training images of 32 × 32 pixels) for 140 epochs.
We vary depths, ﬁlter sizes and number of convolutional ﬁlter block outputs.
Filter sizes are sampled from { 3 , 5 , 7 } and number of ﬁlters is sampled from { 2 , 3 , 4 , ..., 22 } .
Each ResNet block is composed of three convolutional layers followed by batch normalization and summation layers.
We vary the number of blocks from 2 to 18, giving us networks with depths varying between 14 and 110.
Each network is trained for 140 epochs, using Nesterov optimizer.
The learning rate is set to 0.1 and learning rate reduction and momentum are set to 0.1 and 0.9 respectively.
Deep Resnets (CIFAR-10):
We sample 500 39-layer ResNet architectures from a search space similar to Zoph & Le (2017), varying kernel width, kernel height, and number of kernels.
We train these models for 50 epochs on CIFAR-10.
Each architecture consists of 39 layers: 12 conv , a 2x max pool , 9 conv , a 2x2 max pool , 15 conv , and softmax .
Each conv layer is followed by batch normalization and a ReLU nonlinearity.
Each block of 3 conv layers are densely connected via residual connections and also share the same kernel width, kernel height, and number of learnable kernels.
Kernel height and width are independently sampled from { 1 , 3 , 5 , 7 } and number of kernels is sampled from { 6 , 12 , 24 , 36 } .
Finally, we randomly sample residual connections between each block of conv layers.
Each network is trained for 50 epochs using the RMSProp optimizer, with weight decay 10 − 4 , initial learning rate 0.001, and a learning rate reduction to 10 − 5 at epoch 30 on the CIFAR-10 dataset.
MetaQNN CNNs (CIFAR-10 and SVHN): We sample 1,000 model architectures from the search space detailed by Baker et al.
(2017), which allows for varying the numbers and orderings of convolution, pooling, and fully connected layers.
The models are between 1 and 12 layers for the
SVHN experiment and between 1 and 18 layers for the CIFAR-10 experiment.
Each architecture is trained on SVHN and CIFAR-10 datasets for 20 epochs.
Table 3 displays the state space of the
MetaQNN algorithm.
[Table]
LSTM (PTB) : We sample 300 LSTM models and train them on the Penn Treebank dataset forepochs.
Number of hidden layer inputs and lstm cells was varied from 10 to 1400 in steps of 20.
Each network was trained for 60 epochs with batch size of 50 and trained the models using stochastic gradient descent.
Dropout ratio of 0.5 was used to prevent overﬁtting.
Dictionary size of 400 words was used to generate embeddings when vectorizing the data.
Cuda-Convnet (CIFAR-10 and SVHN): We train Cuda-Convnet architecture (Krizhevsky, 2012) with varying values of initial learning rate, learning rate reduction step size, weight decay for convolutional and fully connected layers, and scale and power of local response normalization layers.
We train models with CIFAR-10 for 60 epochs and with SVHN for 12 epochs.
Table 4 show the hyperparameter ranges for the Cuda Convnet experiments. [Table]
[Table]
B H YPERPARAMETER SELECTION IN P ERFORMANCE P REDICTION M ODELS
When training performance prediction models we divided the data into training and validation and used 3-fold cross validation to select optimal hyperparameters.
The models were then trained on full training data using the best hyperparameters.
For random forests we varied number of trees between 10 and 800, and varied ratio of number of features from 0.1 to 0.5.
For ν -SVR, we perform a random search over 1000 hyperparameter conﬁgurations from the space C ∼ LogUniform( 10 − 5 , 10), ν ∼
Uniform(0, 1), and γ ∼ LogUniform( 10 − 5 , 10) (when using the RBF kernel).
For Bayesian linear regression, we perform random search over the hyperparameters of the gamma prior distribution over both noise and weights in the range [ 10 − 7 , 10 − 5 ].
C F -H YPERBAND
Algorithm 1 of this text replicates Algorithm 1 from Li et al.
(2017), except we initialize two dictionaries: D to store training data and M to store performance prediction models.
D [ r ] will correspond to a dictionary containing all datasets with prediction target epoch r .
D [ r ][ τ ] will correspond to the dataset for predicting y r based on the observed y ( t ) 1 − τ , and M [ r ][ τ ] will hold the corresponding performance prediction model.
We will assume that the performance prediction model will have a train function, and a predict function that will return the prediction and standard deviation of the prediction.
In addition to the standard Hyperband hyperparameters R and η , we include ∆ and δ described in Section 4 and κ .
During each iteration of successive halving, we train n i conﬁgurations to r i epochs; κ denotes the fraction of the top n i models that should be run to the full r i iterations.
This is similar to setting the criterion based on the n th best model in the previous section.
We also detail the run_then_return_validation_loss function in Algorithm 2.
This algorithm runs a set of conﬁgurations, adds training data from observed learning curves, trains the performance prediction models when there is enough training data present, and then uses the models to terminate poor conﬁgurations.
It assumes we have a function max_k , which returns the k th max value or −∞ if the list has less than k values.
Algorithm input : R – (Max resources allocated to any conﬁguration) η
– (default η = 3 ) ∆ – (Probability threshold for early termination)
δ – (Performance offset for early termination)
d – (# points required to train performance predictors) κ – (Proportion of models to train) initialize :
D = dict()
M = dict() s max = (cid:98) log η
( R ) (cid:99)
B = ( s max + 1) R 1 for s ∈ { s max , . . .
, 0 } do n = (cid:100) BR η s s +1 (cid:101) , r = Rη − s // begin SUCCESSIVEHALVING with ( n , r ) inner loop
T = get_hyperparameter_configuration( n ) for i ∈ { 0 , . . .
, s } do n i = (cid:98)
nη − i (cid:99) , r i = rη i n next =
(cid:98) n i η (cid:99)
if i !
=
s else
L = run_then_return_validation_loss( T, r i , n next , D, M )
T = top_k( T, L, (cid:98) n i η (cid:99) ) end 11 end
Algorithm input : T – hyperparameter conﬁgurations r – resources to use for training n – # conﬁgurations in next iteration of successive halving
D – dictionary storing training data
M – dictionary storing performance prediction models initialize :
L = [] 1 for t ∈ T do (cid:96) =
[] for i ∈ { 0 , . .
, r − 1 } do (cid:96) i =
run_one_epoch_return_validation_loss( t ) (cid:96) .append
( (cid:96) i )
break end end else then
end end then
end end 23 end 24 return L
D F -H YPERBAND WITH SVR A CQUISITION F UNCTION
Similar to Klein et al.
[Figure]
it is possible to use our simple models as an acquisition function for
Hyperband.
As shown in Table [Figure] our models have reasonable performance with only AP and HP features used, meaning we can use these models to rank hyperparameter settings before training the conﬁgurations.
In particular, for each Hyperband iteration, we randomly select 10,000 conﬁgurations, weight them by their predicted performance and sample from this weighted distribution.
In Figure
[Figure]
we compare f-Hyperband with this ν -SVR acquisition function to f-Hyperband with the standard uniform acquisition function and also to vanilla Hyperband.
It seems that the acquisition function could help slightly, but it does not improve performance across all experiments
[Figure]
and where it does improve performance it does so only slightly
[Figure]
t i
[Figure]
E G AUSSIAN E RROR A SSUMPTION
In order to have uncertainty estimates with frequentist models, we estimated the mean and variance of the error from leave-one-out cross validation and assumed that the error was Gaussian.
In Figure 8 we compare example error distributions between training and validation from a ν -SVR SRM.
Visually one can see that the assumption holds very well for both visual classiﬁcation tasks and holds slightly less well for the language modeling task.
In Figure 9, we show the mean log likelihood of held out prediction errors being drawn from a Gaussian parameterized by the mean and variance of the error on the training set is very close to the mean log likelihood of samples drawn from the same distribution, which shows that our Gaussian error assumption is relatively strong.
F A NALYZING I MPORTANCE OF F EATURES IN THE P REDICTION M ODEL
We used a linear ν -SVR model to compare the weights of all features, which are normalized before training.
Figure 10 shows the weight trends for time-series features, and Tables 5, 6, andcompare weights of some architecture features and hyperparameter features to statistics of time-series features for the MetaQNN (CIFAR-10), LSTMs (Penn Treebank), and Deep Resnets (TinyImagenet) experiments, respectively.
We found the following main insights across datasets
: • The time-series (TS) features are on average have higher weights than HP and AP features (which is conﬁrmed by our ablation studies in Table 2 as well).
The original validation accuracies ( y t ) have higher weights on average than the ﬁrst-order differences ( y t (cid:48) ) and second-order differences ( y t (cid:48)(cid:48) ).
• In general, later epochs in y t have higher weights.
In other words, the latest performance of the model available for prediction is much more important for performance prediction than initial
M e t a
Q NN
C i f a r10% Observed 25% Observed 50% Observed 75%
[Figure]
L S T M
P e nn
T r ee b a n k
Prediction Error
D ee p R e s n e t s
T i n y
I m a g e n e t
Prediction Error
Prediction Error Prediction
[Figure]
[Figure]
performance, when the model has just started training.
, the weights for earlier performance metrics are non-zero, indicating there are high order autoregressive functions being learned.
• Early values of ( y t
(cid:48) ) also have high weights, which indicates learning quickly in the beginning of training is a predictor of better ﬁnal performance in our datasets.
• Among AP and HP features, the total number of parameters and depth have non-zero weight for the CNNs [Figure] and they are assigned weights comparable or higher than late epoch accuracies ( y t ).
However, they have much lower weight in the LSTM experiment.
The number of ﬁlters in each convolutional layer also has a high positive weight for CNNs.
In general, architectural features are much more important for CNNs as compared to LSTMs.
Hyperparameters like initial learning rate and step size were generally not as important as the architectural features, which is corroborated by the ablation study in Table 1 in the main text.
[Figure]
[Table] [Figure] [Table]
[Figure]
e c o n d O r d e r D i
ff s
W e i g h t s
[Figure]
[Table] [Table]
[Table]
[Equation] A DDITIONAL [Equation]
Figure 11 shows the performance prediction results for the Cuda Convnet SVHN experiment.
The instability of BLR with low percentage of learning curve observed may be what caused suboptimal performance of BLR f-Hyperband shown in Figure 6.
Figure 12 shows the relative speedup of f-Hyperband over vanilla Hyperband for each consecutive iteration of Hyperband.
However, Figure 13, which shows results for f-Hyperband with pretrained ν -SVR SRMs, shows that more aggressive settings of f-Hyperband result in suboptimal performance.
We see that we can safely get a 3-4x speedup over vanilla Hyperband with pretrained SRMs.
[Figure]
SVR
BNN
BLR
LastSeenValue
LCE
LCE-No Outlier
[Figure]
The major jump in speedup comes at iteration 10, where we have trained more than 100 models to the full R iterations.
[Figure]
C u m u l a t
i v e
M a x i m u m
V a li d a t i o n A cc u r a c y
[Figure]
H SRM R OBUSTNESS TO O UT O F D ISTRIBUTION C ONFIGURATIONS
Table 8 shows that SRMs are relatively robust when training below or above the median of a speciﬁc architecture or training hyperparameter and testing on the other split.
Cuda Convnet (CIFAR-10)
Learning Rate
WeightDecay
[Table] Scale
LRN Power
Test
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]