PASHA: E FFICIENT HPO AND NAS WITH P ROGRESSIVE R ESOURCE A LLOCATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Hyperparameter optimization (HPO) and neural architecture search (NAS) are methods of choice to obtain the best-in-class machine learning models, but in practice they can be costly to run.
When models are trained on large datasets, tuning them with HPO or NAS rapidly becomes prohibitively expensive for practitioners, even when efficient multi-fidelity methods are employed.
We propose an approach to tackle the challenge of tuning machine learning models trained on large datasets with limited computational resources.
Our approach, named PASHA, extends ASHA and is able to dynamically allocate maximum resources for the tuning procedure depending on the need.
The experimental comparison shows that PASHA identifies well-performing hyperparameter configurations and architectures while consuming significantly fewer computational resources than ASHA.
I NTRODUCTION
Hyperparameter optimization (HPO) and neural architecture search (NAS) yield state-of-the-art models, but often are a very costly endeavor, especially when working with large datasets and models.
For example, using the results of (Sharir et al., 2020) we can estimate that evaluating 50 configurations for a 340-million-parameter BERT model (Devlin et al., 2019) on the 15GB Wikipedia and Book corpora would cost around $500,000.
To make HPO and NAS more efficient, researchers explored how we can learn from cheaper evaluations (e.g. on a subset of the data) to later allocate more resources only to promising configurations.
This created a family of methods often described as multifidelity methods.
Two well-known algorithms in this family are Successive Halving (SH) (Jamieson & Talwalkar, 2016; Karnin et al., 2013) and Hyperband (HB) (Li et al., 2018).
Multi-fidelity methods significantly lower the cost of the tuning.
Li et al.
(2018) reported speedups up to 30x compared to standard Bayesian Optimization (BO) and up to 70x compared to random search.
Unfortunately, the cost of current multi-fidelity methods is still too high for many practitioners, also because of the large datasets used for training the model.
As a workaround, they need to design heuristics which can select a set of hyperparameters or an architecture with a cost comparable to training a single configuration, for example, by training the model with multiple configurations for a single epoch and then selecting the best-performing candidate.
On one hand, such heuristics lack robustness and need to be adapted to the specific use-cases in order to provide good results.
On the other hand, they build on an extensive amount of practical experience suggesting that multi-fidelity methods are often not sufficiently aggressive in leveraging early performance measurements and that identifying the best performing set of hyperparameters (or the best architecture) does not require training a model until convergence.
For example, Bornschein et al.
(2020) show that it is possible to find the best hyperparameter – number of channels in ResNet101 architecture (He et al., 2015) for ImageNet (Deng et al., 2009) – using only one tenth of the data. However, it is not known beforehand that one tenth of data is sufficient for the task.
The aim of our work is to design a method that consumes fewer resources than standard multi-fidelity algorithms such as Hyperband (Li et al., 2018) or ASHA (Li et al., 2020) and nonetheless is able to identify configurations that produce models with a similar predictive performance after being fully re-trained from scratch. It is common practice to retrain the model on a combination of training and validation sets to obtain the best performance after optimizing the hyperparameters.
In order to achieve the speedup, we propose a variant of ASHA, called PASHA (Progressive ASHA), that starts with a small amount of initial maximum resources and gradually increases them as needed.
ASHA in contrast has a fixed amount of maximum resources, which is a hyperparameter defined by the user and is difficult to select.
Our empirical evaluation shows that PASHA can save a significant amount of resources while finding similarly well-performing configurations as conventional ASHA.
To summarize, our contributions are as follows: 1) We introduce a new approach called PASHA that dynamically selects the amount of maximum resources to allocate for HPO or NAS (up to a certain budget), 2) Our empirical evaluation shows the approach significantly speeds up HPO and NAS without sacrificing the performance, and 3)
We show the approach can be successfully combined with sample-efficient strategies based on Bayesian Optimization, highlighting the generality of our approach.
Our implementation is based on the Syne Tune library (Salinas et al., 2022).
R ELATED WORK
Machine learning systems used in real-world applications often rely on a large number of hyperparameters, and require testing many combinations of them in order to identify the optimal solution.
This makes data-inefficient techniques such as Grid Search or Random Search (Bergstra & Bengio, 2012) very expensive in most practical scenarios.
Various approaches have been proposed to find good parameters more quickly, and they can be classified into two main families: 1) Bayesian Optimization: evaluates the most promising configurations by modelling their performance.
The methods are sample-efficient but they are often designed for environments with limited amount of parallelism; 2)
Multi-fidelity: sequentially allocates more resources to configurations with better performance and allows high level of parallelism during the tuning.
From these two families, multi-fidelity methods have typically been faster when run at scale and will be the focus of this work.
It is also possible to combine ideas from the two families together, for example as done in BOHB by Falkner et al.
and we will test a similar method in our experiments.
Successive halving (Karnin et al., 2013; Jamieson & Talwalkar, 2016) is conceptually the simplest multi-fidelity method.
Its key idea is to run all configurations using a small amount of resources, which depends on the minimum resources and the minimum early stopping rate, and then successively promote only a fraction of the most promising configurations to be trained using more resources.
Another popular multi-fidelity method, called Hyperband (Li et al., 2018), performs successive halving with different early stopping rates.
ASHA (Li et al., 2020) extends the simple and very efficient idea of successive halving by introducing asynchronous evaluation of different configurations, which leads to further practical speedups thanks to better resource allocation.
Related to the problem of efficiency in HPO, cost-aware HPO explicitly accounts for the cost of the evaluations of different configurations.
Previous work on cost-aware HPO for multi-fidelity algorithms such as CAHB (Ivkin et al., 2021) keeps a tight control on the budget spent during the
HPO process.
This is different from our work, as we reduce the budget spent by terminating the HPO procedure early instead of allocating the compute budget in its entirety.
Moreover, PASHA could be combined with CAHB to leverage the cost-based resources allocation.
Recently, researchers considered dataset subsampling to speedup the search of the best hyperparameters or architectures.
Shim et al.
(2021) have combined coresets with PC-DARTS (Xu et al., 2020) and showed that they can find well-performing architectures using only 10% of the data and 8.8x less search time.
Similarly, Visalpara et al.
(2021) have combined subset selection methods with the Tree-structured Parzen Estimator (TPE) for hyperparameter optimization (Bergstra et al., 2011).
With a 5% subset they obtained between an 8x to 10x speedup compared to standard TPE.
However, in both cases it is difficult to say in advance what subsampling ratio to use.
For example, the 10% ratio in (Shim et al., 2021) incurs no decrease in accuracy, while reducing further to 2% leads to a substantial (2.6%) drop in accuracy.
In practice, it is difficult to find a trade-off between the time required for tuning (proportional to the subset size) and the loss of performance for the final model because these change, sometimes wildly, between datasets.
We approach this issue in a principled way using our PASHA algorithm.
Further, Zhou et al.
(2020) have observed that for a fixed number of iterations, rank consistency is better if we use more training samples and fewer epochs rather than fewer training samples and more epochs.
This observation gives further motivation for using the whole dataset for HPO/NAS and using approaches like PASHA to save computational resources.
P ROBLEM SETUP
The problem of selecting the best configuration of a machine learning algorithm to be trained is formalized in Jamieson & Talwalkar (2016) as a non-stochastic bandit problem.
In this setting the learner (the hyperparameter optimizer) receives N hyperparameter configurations and it has to identify the best performing one with the constraint of not spending more than a fixed amount of resources R (e.g. total number of training epochs) on a specific configuration.
R is considered given, but in practice users do not have a good way for selecting it, which can have undesirable consequences: if the value is too small, the model performance will be sub-optimal, while if the budget is too large, the user will incur a significant cost without any practical return.
This leads users to overestimate R , setting it to a large amount of resources in order to guarantee the convergence of the model.
We maintain the concept of maximum amount of resources in our algorithm but we prefer to interpret R as a “safety net”, a cost not to be surpassed (e.g. in case an error prevents a normal behaviour of the algorithm), instead of the exact amount of resources spent for the optimization.
This setting could be extended with additional assumptions, based on empirical observation, removing some extreme cases and leading to a more practical setup.
In particular, when working with large datasets we observe that the curve of the loss for configurations (called arms in the bandit literature) continuously decreases (in expectation).
Moreover, “crossing points” between the curves are rare (excluding noise), and they are almost always in the very initial part of the training procedure.
Viering & Loog (2021); Mohr & van Rijn (2022) provide an analysis of learning curves and note that in practice most learning curves are well-behaved, with Bornschein et al. (2020); Domhan et al.
(2015) reporting similar findings.
More formally, let us define R as the maximum number of resources needed to train an ML algorithm to convergence.
Given π m ( i ) the ranking of configuration i after using m resources for training, there exists minimum R ∗ much smaller than R such that for all amounts of resources r larger than R ∗ the rankings of configurations trained with r resources remain the same: ∃ R ∗ ≪
R : ∀ i ∈ { configurations } , ∀ r > R ∗ , π R ∗ ( i ) =
π r ( i ) .
The existence of such a quantity, limited to the best performing configuration, is also assumed by Jamieson & Talwalkar (2016), and it is leveraged to quantify the budget required to identify the best performing configuration.
If we knew R ∗ , it would be sufficient to run all configurations with exactly that amount of resources to identify the best one and then just train the model from scratch with all the data using that configuration.
Unfortunately that quantity is unknown and can only be estimated during the optimization procedure.
M ETHOD
Our approach, named PASHA, is an extension of ASHA (Li et al., 2020) inspired by the “doubling trick” (Auer et al., 1995).
PASHA targets improvements for hyperparameter tuning on large datasets by hinging on the assumptions made about the crossing points of the learning curves in Section 3.
The algorithm starts with a small initial amount of resources and progressively increases them if the ranking of the configurations in the top two rungs (rounds of promotion) has not stabilized.
The ability of our approach to stop early automatically is the key benefit.
We illustrate the approach in Figure 1, showing how we stop evaluating configurations for additional rungs if rankings are stable. We describe the details of our proposed approach in Algorithm 1.
Given η , a hyperparameter used both in ASHA and PASHA to control the fraction of configurations to prune, PASHA sets the maximum resources to be used for evaluating a configuration using the reduction factor η and the minimum amount of resources r to be used.
The approach increases the maximum number of resources allocated to promising configurations each time the ranking of configurations in the top two rungs becomes inconsistent. For example, if we can currently train configurations up to rungand the ranking of configurations in rung 1 and rung 2 is not consistent, then we allow training part of the configurations up to rung 3, i.e. one additional rung. The minimum amount of resources r is a hyperparameter to be set by the user. It is significantly easier to set compared to R as r is the minimum amount of resources required to see a meaningful difference in the performance of the models, and it can be easily estimated empirically by running a few small-scale experiments.
1: input minimum resource r , reduction factor η 2: function PASHA () 3: t = 0 , R
= η 2 r , K 0 =
⌊ log η
( R 0 /r )
⌋ 4: while desired do 5: for each free worker do 6: ( θ, k ) = get job () 7: run then return val loss ( θ, rη k ) 8: end for 9: for completed job ( θ , k ) with loss l do 10:
Update configuration θ in rung k with loss l .
11: if k ≥ K t − 2 then 12: π k = configuration ranking ( k ) 13: end if 14: if k = K t − 1 and π k ̸≡ π k − 1 then 15: t = t +16:
R t = η t R17:
K t =
⌊ log η
( R t /r )
⌋ 18: end if 19: end for 20: end while 21: end function 22: function get job () 23: // Check if there is a promotable config.
24: for k = K t − 1 , . .
, 1 , 0 do 25: candidates = top k ( rung k, | rung k | /η ) 26: promotable = { c ∈ candidates : c not promoted } 27: if | promotable | > 0 then 28: return promotable [0] , k +29: end if 30: // If not, grow bottom rung.
Draw random configuration θ .
32: return θ,33: end for 34: end function
We also set a maximum amount of resources R so that PASHA can default to ASHA if needed and avoid increasing the resources indefinitely.
While it is not generally reached, it provides a safety net.
4. S OFT RANKING
In soft ranking, configurations are still sorted by predictive performance but are considered equivalent if the performance difference is smaller than a value ϵ (or equal to it).
Instead of producing a sorted list of configuration, this provides a list of lists where for every position of the ranking there is a list of equivalent configurations.
The concept is explained graphically in Figure 2, and we also provide a formal definition.
For a set of n configurations c 1 , c 2 , · · · , c i , · · · , c n and performance metric f (e.g. accuracy) with f ( c 1 ) ≤ f ( c 2 ) ≤ · · · ≤ f ( c i ) ≤ · · · ≤ f ( c n ) , soft rank at position i is defined as soft rank i = { c j ∈ configurations :
| f ( c i ) − f ( c j ) | ≤ ϵ } .
When deciding on if to increase the resources, we go through the ranked list of configurations in the top rung and check if the current configuration at the given rank was in the list of configurations for that rank in the previous rung.
If there is a configuration which does not satisfy the condition, we increase resources.
4. A UTOMATIC ESTIMATION OF ϵ BY MEASURING NOISE IN RANKINGS
Every operation involving randomization gives slightly different results when repeated, the training process and the measurement of performance on the validation set are no exception.
In an ideal world, we could repeat the process multiple times to compute empirical mean and variance to make a better decision. Unfortunately this is not possible in our case since the repeating portions of the training process will defeat the purpose of our work: speeding up the tuning process. Understanding when the differences between the performance measured for different configurations are “significant” is crucial for ranking them correctly. We devise a method to estimate a threshold below which differences are meaningless.
Our intuition is that configurations with different performance maintain their relative ranking over time. On the other hand, configurations that repeatedly swap their rankings perform similarly well and the performance difference in the current epoch or rung is simply due to noise.
We want to measure this noise and use it to automatically estimate the threshold value ϵ to be used in the soft-ranking described above.
Formally we can define a set of pairs of configurations that perform similarly well by the following:
[Equation]
for resource levels (e.g., epochs) r j > r k > r l , using the same notation as earlier to refer to resources.
Note that resources r j , r k , r l do not need to differ by 1 epoch – there can be e.g. several epochs in between.
We only consider configurations ( c, c ′ ) that made it to the latest rung, so rη K t − 1 ≥ r j > rη K t − 2 .
The value of ϵ can then be calculated as the N -th percentile of distances between the performances of configurations in S :
[Equation]
The exact value of r j depends on the considered pair of configurations ( c, c ′ ) . To uniquely define f r j , we take the maximum resources r j currently available for both configurations in the considered pair ( c, c ′ ) . Let us consider the following example setup: the top rung has 8 epochs and the next one hasepochs, there are three configurations c a , c b , c c that made it to the top rung and were trained for 8, 8 and 6 epochs so far respectively. The set of distances between configurations in S would then be {| f 8 ( c a ) − f 8 ( c b ) | , | f 6 ( c a ) − f 6 ( c c ) | , | f 6 ( c b ) − f 6 ( c c ) |} .
The value of ϵ is recalculated every time we receive new information about the performances of configurations.
Initially the value of ϵ is set to 0, which means that we check for exact ranking if we cannot yet calculate the value of ϵ .
E XPERIMENTS
In this section we empirically quantify the advantage provided by PASHA.
Its goal is not to provide a model with a higher accuracy, but to identify the best configuration in a shorter amount of time so that we can then re-train the model from scratch.
Overall, we target a significantly faster tuning time and on-par predictive performance when comparing with the models identified by state-of-the-art optimizers like ASHA.
Re-training after HPO or NAS is important because HPO and NAS in general require to reserve a significant part of the data (often around 20 or 30%) to be used as a validation set.
Training with fewer data is not desirable because in practice it is observed that training a model on the union of training and validation sets provides better results.
We tested our method on two different sets of experiments.
The first set evaluates the algorithm on
NAS problems and uses NASBench201 (Dong & Yang, 2020), while the second set focuses on HPO and was run on two large-scale tasks from PD1 benchmark (Wang et al., 2021).
5. S ETUP
Our experimental setup consists of two phases: 1) run the hyperparameter optimizer until N =candidate configurations are evaluated; and 2) use the best configuration identified in the first phase to re-train the model from scratch.
For the purpose of these experiments we re-train all the models using only the training set.
This avoids introducing an arbitrary choice on the validation set size and allows us to leverage standard benchmarks such as NASBench201.
In real-world applications the model can be trained on both training and validation sets.
All our results report only the time invested in identifying the best configuration since the re-training time is comparable for all optimizers.
All results are averaged over multiple repetitions, with the details specified for each set of experiments separately.
We use N = 90 -th percentile when calculating the value of ϵ .
We use 4 workers to perform evaluations in parallel and asynchronously.
The choice of R is sensitive for ASHA since it can make the optimizer consume too many resources and penalize the performance of the algorithm.
To have a fair comparison, we make R dataset-dependent adopting the maximum amount of resources in the considered benchmarks.
r is also dataset-dependent and η , the halving factor, is set to 3 unless otherwise specified.
The same values are used for both ASHA and PASHA.
We compare PASHA with ASHA (Li et al., 2020), a state-of-the-art approach for hyperparameter optimization, and other relevant baselines.
In particular, we consider “one-epoch baseline” that trains all configurations for one epoch (the minimum available resources) and then selects the most promising configuration, and “random baseline” that randomly selects the configuration.
For our one-epoch baseline we sample N = 256 configurations, using the same scheduler and seeds as for PASHA and ASHA.
For our random baseline we sample N = 2560 configurations at random and directly look up their performance.
5. NAS EXPERIMENTS
For our NAS experiments we leverage the well-known NASBench201 (Dong & Yang, 2020) benchmark.
The task is to identify the network structure providing the best accuracy on three different datasets (CIFAR-10, CIFAR-100 and ImageNet16-120) independently.
We use r = 1 epoch and
R = 200 epochs.
We repeat the experiments using 5 random seeds for the scheduler and 3 random seeds for NASBench201 (all that are available), resulting in 15 repetitions.
Some configurations in
NASBench201 do not have all seeds available, so we impute them by averaging over the available seeds.
To measure the predictive performance we report the best accuracy on the combined validation and test set provided by the creators of the benchmark.
The results in Table 1 suggest that PASHA consistently leads to strong improvements in runtime, while achieving similar accuracy values as the baseline algorithm ASHA.
The one-epoch baseline has noticeably worse accuracies than ASHA or PASHA, suggesting that PASHA does a good job of deciding when to continue increasing the resources – it does not stop too early.
Random baseline is a lot worse than the one-epoch baseline, so there is value in performing NAS.
We also report the maximum resources used to find how early the ranking becomes stable in PASHA.
Note that the time required to train a model is about 1.3h for CIFAR-10 and CIFAR-100, and about 4.1h for
ImageNet16-120, making the total tuning time of PASHA comparable or faster than the training time.
[Table] We also ran additional experiments testing PASHA with a reduction factor of η = 2 and η = 4 instead of η = 3 , the usage of PASHA as a scheduler in MOBSTER (Klein et al., 2020) and alternative ranking functions.
These experiments provided similar findings as the above and are described next.
5.2. R EDUCTION FACTOR
An important parameter for the performance of multi-fidelity algorithms like ASHA is the reduction factor.
This hyperparameter controls the fraction of pruned candidates at every rung.
The optimal theoretical value is e and it is typically set to 2 or 3.
In Table 2 we report the results of the different algorithms ran with η = 2 and η = 4 on CIFAR-100 (the full set of results is in Appendix A).
The gains are consistent also for η = 2 and η = 4 , with a larger speedup when using η = 2 as that allows PASHA to make more decisions and identify earlier that it can stop the search
[Table]
[Table]
5.2. B AYESIAN O PTIMIZATION
Bayesian Optimization combined with multi-fidelity methods such as Successive Halving can improve the predictive performance of the final model (Klein et al., 2020).
In this set of experiments, we verify PASHA can speedup also these kinds of methods.
Our results are reported in Table [Table] where we can clearly see PASHA obtains a similar accuracy result as ASHA with significant speedup.
[Table]
5.2. A LTERNATIVE RANKING FUNCTIONS
We have considered a variety of alternative ranking functions in addition to the soft ranking function that automatically estimates the value of ϵ by measuring noise in rankings.
These include simple ranking (equivalent to soft ranking with ϵ = 0 . 0 ), soft ranking with fixed values of ϵ or obtained using various heuristics (for example based on the standard deviation of objective values in the previous rung), Rank Biased Overlap (RBO) (Webber et al., 2010), and our own reciprocal rank regret metric (RRR) that considers the objective values of configurations.
Details of the ranking functions and additional results are in Appendix B.
Table 4 shows a selection of the results on CIFAR-100 with full results in the appendix.
We can see there are also other ranking functions that work well and that simple ranking is not sufficiently robust – some benevolence is needed.
However, the ranking function that estimates the value of ϵ by measuring noise in rankings (to which we refer simply as PASHA) remains the easiest to use, is well-motivated and offers both excellent performance and large speedup.
[Table] [Table]
HPO EXPERIMENTS
We further utilize the PD1 HPO benchmark (Wang et al., 2021) to show the usefulness of PASHA in large-scale settings.
In particular, we take WMT15 German-English (Bojar et al., 2015) and ImageNet (Deng et al., 2009) datasets that use xformer (Lefaudeux et al., 2021) and ResNet50 (He et al., 2015) models.
Both of them are datasets with a large amount of training examples, in particular WMT
German-English has about 4.5M examples, while ImageNet has about 1.3M examples.
The task in PD1 is to optimize four hyperparameters that influence the training of the models: base learning rate η ∈ (cid:2) 10 − 5 , 10 . 0 (cid:3) (log scale), momentum 1 − β ∈ (cid:2) 10 − 3 , 1 . 0 (cid:3) (log scale), polynomial learning rate decay schedule power p ∈ [0 . 1 , 2 . 0] (linear scale) and decay steps fraction λ ∈
[0 . 01 , 0 . 99] (linear scale). The minibatch size used for WMT experiments is 64, while the minibatch size for ImageNet experiments is 512.
There are 1414 epochs available for WMT and 251 for ImageNet.
Note that there are also other datasets available in the PD1 benchmark, but these only have a small number of epochs with 1 epoch being the minimum amount of resources.
As a result there would not be enough rungs to see benefits of the early stopping provided by PASHA.
If resources could be defined in terms of fractions of epochs, PASHA could be beneficial there too. Most public benchmarks have resources defined in terms of epochs, but in practice it is possible to define resources also in alternative ways. We use 1-NN as a surrogate model for the PD1 benchmark.
We repeat our experiments using 5 random seeds (same seeds as for NASBench201) and there is only one dataset seed available.
The results in Table 5 show that PASHA leads to large speedups on both WMT and ImageNet datasets.
The speedup is particularly impressive for the significantly larger WMT dataset where it is about 15.5x, highlighting how PASHA can significantly accelerate the HPO search on datasets with millions of training examples (WMT has about 4.5M training examples).
The one-epoch baseline obtains similar accuracy as ASHA and PASHA for WMT, but performs significantly worse on ImageNet dataset.
This result suggests that simple approaches such as the one-epoch baseline are not robust and solutions such as PASHA are needed (which we also saw on NASBench201).
Selecting the hyperparameters randomly leads to significantly worse performance than any of the other approaches.
[Table]
D ISCUSSION
PASHA is an algorithm that is designed to make HPO and NAS more accessible to machine learning practitioners as it significantly lowers the time and cost required to find well-performing configurations.
Its limitation is similar to that of ASHA and other multi-fidelity methods: it may prune configurations early that would perform well if they were trained for longer.
While we used PASHA on benchmarks that define resources in terms of epochs, it is not needed to set the resources on the level of epochs.
In fact, in many large datasets it will be sufficient to train the model only for a small number of epochs.
In these cases it can be useful to specify the rung levels in terms of iterations or data points processed so that we can still evaluate if to increase the resources multiple times.
PASHA could then find the best hyperparameters even without seeing all of the training examples.
In other words, if the model is trained only for a few epochs and resources are defined in terms of epochs, PASHA cannot do much.
However, if the minimum resources are defined as a fraction of an epoch, PASHA could still be useful in those cases.
C ONCLUSIONS
In this work we introduced a new variant of Successive Halving, called PASHA, that progressively increases the resources allocated to promising configurations.
PASHA considers the rankings of configurations at successive rung levels and if the rankings have stabilized, stops the HPO procedure, returning the best configuration found.
The approach is easy to use as it automatizes the choice of its internal parameters in an intuitive yet principled way based on noise in rankings of configurations.
Despite its simplicity, PASHA has led to strong improvements in the tuning time.
For example, in many cases it has reduced the time needed to about one third compared to ASHA without a noticeable impact on the quality of the found configuration.
In the case of the largest dataset with millions of training examples it has reduced the tuning time by a factor of about 15x.
We have also demonstrated it is possible to successfully combine PASHA with more advanced search strategies based on Bayesian Optimization to obtain improvements in accuracy at a fraction of the time.
Further work could investigate the definition of rungs and resource levels, with the aim of understanding how they impact the decisions of the algorithm.
More broadly this applies not only to PASHA but also other successive halving algorithms in general.
R EPRODUCIBILITY S TATEMENT
We include the code for our approach as part of the supplementary material, including details for how to run the experiments.
We use pre-computed benchmarks that make it possible to run the NAS and HPO experiments even without large computational resources. A R EDUCTION FACTOR
Table 6 shows the full set of results for our experiments with different reduction factors.
The behaviour is the same across all cases.
[Table]
B E XPERIMENTS WITH ALTERNATIVE RANKING FUNCTIONS B. D ESCRIPTION
PASHA employs a ranking function whose choice is completely arbitrary.
In our main set of experiments we used soft ranking that automatically estimates the value of ϵ by measuring noise in rankings.
In this set of experiments we would like to evaluate different criteria to define the ranking of the candidates.
We describe the functions considered next.
B.1. D IRECT RANKING
As a baseline, we study if we can use the simple ranking of configurations by predictive performance (e.g., sorting from the ones with the highest accuracy to the ones with the lowest).
If any of the configurations change their order, we consider the ranking unstable and increase the resources.
B.1. S OFT RANKING VARIATIONS
We consider several variations of soft ranking.
The first variation is to fix the value of the ϵ parameter.
We have considered values 0.01, 0.02, 0.025, 0.03, 0.05.
The second set of variations aim to estimate the value of ϵ automatically, using various heuristics.
The heuristics we have evaluated include:
• Standard deviation: calculate the standard deviation of the considered performance measure (e
g. accuracy) of the configurations in the previous rung and set a multiple of it as the value of ϵ – we tried multiples of 1, 2 and 3.
• Mean distance: value of ϵ is set as the mean distance between the score of the configurations in the previous rung.
• Median distance: similar to the mean distance, but using the median distance.
There are various benefits for estimating the value of ϵ by measuring noise in rankings, as presented in our paper:
• There is no need to set the value of ϵ manually.
• Estimation of ϵ has an intuitive motivation that makes sense.
• The value of ϵ can dynamically adapt to the different stages of hyperparameter optimization.
• The approach works well in practice.
B.1. R ANK BIASED OVERLAP (RBO) (W EBBER ET AL ., 2010)
A score that can be broadly interpreted as a weighted correlation between rankings.
We can specify how much we want to prioritize the top of the ranking using parameter p that is between 0.0 and 1.0, with a smaller value giving more priority to the top of the ranking.
The best value is 1.0, while it gives value of 0.0 for rankings that are completely the opposite.
We compute the RBO value and then compare it to the selected threshold t , increasing the resources if the value is less than the threshold.
B.1.
R ECIPROCAL R ANK R EGRET (RRR)
A key insight is that configurations can be very similar to each other and differences in their rankings will not affect the quality of the found solution significantly.
To account for this we look at the objective values of the configurations (e.g. accuracy) and compute the relative regret that we would pay at the current rung if we would have assumed the ranking at the previous rung correct.
We define reciprocal rank regret (RRR) as:
[Equation]
where f represents the ordered scores in the top rung, f ′ represents the reordered scores from the top rung according to the second rung, n is the number of configurations in the top rung and p is the parameter that says how much attention to give to the top of the ranking.
The weights w i sum toand can be selected in different ways to e.g. give more priority to the top of the ranking.
For example, we could use the following weights:
[Equation]
The metric has an intuitive interpretation: it is the average relative regret with priority on top of the ranking.
The best value of RRR is 0.0, while the worst possible value is 1.0.
We also consider a version of RRR which considers the absolute values of the differences in the objectives - Absolute RRR (ARRR).
We have evaluated these additional ranking functions using NASBench201 benchmark.
B. R ESULTS
We report the results in Table 7, 8 and 9.
We see there are also several other variations that achieve strong results across a variety of datasets within NASBench201, most notably soft ranking 2 σ and variations based on RRR.
In these cases we obtain similar performance as ASHA, but at a significantly shorter time.
[Table]
[Table]
[Table]