A UTOMATA : Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep neural networks have seen great success in recent years; however, training adeep model is often challenging as its performance heavily depends on the hyperparameters used.
In addition, finding the optimal hyper-parameter configuration,even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms,can be time-consuming, requiring multiple training runs over the entire datasetfor different possible sets of hyper-parameters.
Our central insight is that usingan informative subset of the dataset for model training runs involved in hyperparameter optimization, allows us to find the optimal hyper-parameter configurationsignificantly faster.
In this work, we propose A UTOMATA , a gradient-based subsetselection framework for hyper-parameter tuning.
We empirically evaluate theeffectiveness of A UTOMATA in hyper-parameter tuning through several experimentson real-world datasets in the text, vision, and tabular domains.
Our experimentsshow that using gradient-based data subsets for hyper-parameter tuning achievessignificantly faster turnaround times and speedups of 3 × -30 × while achievingcomparable performance to the hyper-parameters found using the entire dataset.
Introduction
In recent years, deep learning systems have found great success in a wide range of tasks, such asobject recognition [15], speech recognition [17] and machine translation [1], making people’s liveseasier on a daily basis.
However, in the quest for near-human performance, more complex and deepermachine learning models trained on increasingly large datasets are being used at the expense ofsubstantial computational costs.
Furthermore, deep learning is associated with a significantly largenumber of hyper-parameters such as the learning algorithm, batch size, learning rate and modelconfiguration parameters ( e.g. , depth, number of hidden layers, etc. )
that need to be tuned.
Hence,running extensive hyper-parameter tuning and auto-ml pipelines is becoming increasingly necessaryto achieve state-of-the-art models.
However, tuning the hyper-parameters requires multiple trainingruns over the entire datasets (which are significantly large nowadays), resulting in staggering computecosts, running times, and, more importantly, CO2 emissions.
To give an idea of staggering compute costs, we consider an image classification task on a relativelysimple CIFAR-10 dataset where a single training run using a relatively simple model class of Residual
Networks [16] for 300 epochs on a V100 GPU takes around 4 hours.
If we perform 1000 trainingruns (which is not uncommon today) naively using grid search for hyper-parameter tuning, it willtake 4000 GPU hours.
The resulting CO2 emissions would be between 440 to 1000 kg of CO2emitted 1 , which is equivalent to 1100 to 2500 miles of car travel in the US.
Similarly, the costs of [Figure] training state-of-the-art NLP models and vision models on larger datasets like ImageNet are evenmore staggering [48] 2 .
Naive hyper-parameter tuning methods like grid search [2] often fail to scale up with the dimensionality of the search space and are computationally expensive.
Hence, more efficient and sophisticated Bayesian optimization methods [4, 19, 5, 46, 27] have dominated the field of hyper-parameter optimization in recent years.
Bayesian optimization methods aim to identify good hyper-parameterconfigurations quickly by building a posterior distribution over the search space and by adaptivelyselecting configurations based on the probability distribution.
More recent methods [49, 50, 10, 27]try to speed up configuration evaluations for efficient hyper-parameter search; these approaches speedup the configuration evaluation by adaptively allocating more resources to promising hyper-parameterconfigurations while eliminating poor ones quickly.
Recent works like SHA [20], Hyperband [32], ASHA [34] use aggressive early-stopping strategies tostop not-so-promising configurations quickly while allocating more resources to the promising ones.
Generally, these resources can be the size of the training set, number of gradient descent iterations,training time, etc.
Nickson et al.
[40], Krueger et al.
[29] try to quickly evaluate a configuration’sperformance on a large dataset by evaluating the training runs on small, random subsets and theyempirically show that small data subsets could suffice to estimate a configuration’s quality.
Similarly,past works [40, 29] show that very small data subsets can be effectively used to find the best hyperparameters quickly.
However, all these approaches have naively used random training data subsetsand did not place much focus on selecting informative subsets instead.
Our central insight is thatusing small informative data subsets allows us to find good hyper-parameter configurations moreeffectively than random data subsets.
An earlier work called G RAD M ATCH [23] demonstrated the effectiveness of gradient-based datasubset selection for selecting informative data subsets by using it for efficient model training in asupervised learning setting.
In this work, we empirically investigate the advantage of using gradientbased subset selection algorithms, mainly G RAD M ATCH , for hyper-parameter tuning comparedwith using random subsets and the full dataset for hyper-parameter tuning.
Fundamentally, we use G RAD M ATCH to select data subsets that can be used for efficiently tuning the hyper-parameters.
To this end, we propose A UTOMATA , an efficient hyper-parameter tuning framework that combinesexisting hyper-parameter search and scheduling algorithms with intelligent subset selection.
Further,our experimental results illustrate that very small subsets (1%, 5% subsets) can be used for effectivehyper-parameter tuning because we are primarily concerned with retaining the ordering of hyperparameters rather than the final accuracy, thereby enabling us to achieve better speedups and energyefficiency (and more significant CO2 emissions reduction).
1.1 Related Work
Hyper-parameter Tuning Approaches: A number of algorithms have been proposed for hyperparameter tuning including grid search 3 , Bayesian algorithms [3], random search [43], etc.
Furthermore, a number of scalable toolkits and platforms for hyper-parameter tuning exist like Ray-tune [36], H2O automl [31], etc.
See [45, 54] for a survey of current approaches and also tricks for hyperparameter tuning for deep models.
The biggest challenges of existing hyper-parameter tuningapproaches are a) the large hyper-parameter search space and b) the increased training times oftraining models.
Recently, Li et.
al.
[33] have proposed an efficient approach for parallelizinghyper-parameter tuning using Asynchronous Successive Halving Algorithm (ASHA).
A UTOMATA iscomplementary to such approaches and can be be combined effectively with them.
Data Subset Selection Approaches:
Several recent papers have used submodular functions 4for data subset selection towards various applications like speech recognition [53, 52], machinetranslation [26] and computer vision [21].
Other common approaches for subset selection include theusage of coresets.
Coresets are weighted subsets of the data, which approximate certain desirablecharacteristics of the full data ( e.g. , the loss function) [12].
Coreset algorithms have been usedfor several problems including k -means and k -median clustering [14], SVMs [8] and Bayesianinference [6].
Recent coreset selection-based methods [38, 24, 23, 25] have shown great promise forefficient and robust training of deep models.
C RAIG [38] tries to select a coreset summary of thetraining data that estimate the full training gradient closely.
Whereas GLISTER [24] poses the coresetselection problem as a discrete-continuous bilevel optimization problem that minimizes the validationset loss.
Similarly, RETRIEVE [25] also uses a discrete bilevel coreset selection problem to selectunlabeled data subsets for efficient semi-supervised learning.
Another approach G RAD -M ATCH [23]selects coreset summary that approximately matches the full training loss gradient using orthogonalmatching pursuit.
The contributions of our work can be summarized as follows:
A UTOMATA Framework:
Wepropose A UTOMATA a framework that combines intelligent gradient based subset selection withhyper-parameter search and scheduling algorithms to enable faster hyper-parameter tuning.
Ourwork empirically studies the role of intelligent data subset selection for hyper-parameter tuning.
Inparticular, we seek to answer the following question:
Is it possible to use small informative datasubsets between 1% to 30% for faster configuration evaluations in hyper-parameter tuning, therebyenabling faster tuning times while maintaining comparable accuracy to tuning hyper-parameters onthe full dataset?
Importance of Intelligent Subset Selection: Our empirical results in Figure 1bdemonstrate that G RAD M ATCH [23] adopted by A UTOMATA preserves the original hyperparameterordering better than other baseline subset selection strategies including R ANDOM and C RAIG .
Inaddition, gradient-based subset selection strategies such as G RAD M ATCH and C RAIG preserve theoriginal hyper-parameter ordering better compared to R ANDOM , thereby showcasing the importanceof intelligent subset selection approaches. Effectiveness of A UTOMATA : We empirically demonstrate the effectiveness of A UTOMATA framework used in conjunction with existing hyper-parametersearch algorithms like TPE, Random Search, and hyper-parameter scheduling algorithms like Hyperband, and ASHA through a set of extensive experiments on multiple real-world datasets.
We give asummary of the speedup vs. relative performance achieved by A UTOMATA compared to full datatraining in Figure 1a.
More specifically, A UTOMATA achieves a speedup of 3x - 30x with minimalperformance loss for hyper-parameter tuning.
Further, in Section 3, we show that the gradient-basedsubset selection approach of A UTOMATA outperforms the previously considered random subsetselection for hyper-parameter tuning.
A UTOMATA Framework
In this section, we present A UTOMATA , a hyper-parameter tuning framework, and discuss its differentcomponents shown in Figure 2.
The A UTOMATA framework consists of three components: ahyper-parameter search algorithm that identifies which configuration sets need to be evaluated, agradient-based subset selection algorithm for training and evaluating each configuration efficiently, Comparison of Configurations Choice of Hyper-parameters
R epea t un t il t he r equ i r ed nu m be r o f h
y pe r- pa r a m e t e r s a r e e v a l ua t ed DSS based model training loop using the selected configuration DSS based model training loop using the selected configuration
DSS
[Figure]
training loop using the selected configuration
[Figure]
Promoted Configurations ComponentConfiguration- ConfigurationConfiguration-n and a hyper-parameter scheduling algorithm that provides early stopping by eliminating the poorconfigurations quickly. With A UTOMATA framework, one can use any of the existing hyper-parametersearch and hyper-parameter scheduling algorithms and still achieve significant speedups with minimalperformance degradation due to faster configuration evaluation using gradient-based subset training.
Notation Denote by H , the set of configurations selected by the hyper-parameter search algorithm.
Let D = ( x i , y i ) }
Ni =1 , denote the set of training examples, and V = { ( x j , y j ) } Mj =1 the validationset.
Let θ i denote the classifier model parameters trained using the configuration i ∈ H .
Let S i be thesubset used for training the i th configuration model θ i and w i be its associated weight vector i.e.,each data sample in the subset has an associated weight that is used for computing the weighted loss.
We superscript the changing variables like model parameters θ , subset S with the timestep t to denotetheir specific values at that timestep.
Next, denote by L jT ( θ i ) =
L T ( x j , y j , θ i ) , the training loss ofthe j th data sample in the dataset for i th classifier model, and let L T ( θ i ) =
(cid:80) k ∈D
L kT ( θ i ) be the lossover the entire training set for i th configuration model.
Let, L jT ( S , θ i ) =
(cid:80) k ∈X L T ( x k , y k , θ i ) bethe loss on a subset S ⊆ V of the training examples at timestep j .
Let the validation loss be denotedby L V .
Component-1: Hyper-parameter Search Algorithm - Given a hyper-parameter search space,hyper-parameter search algorithms provide a set of configurations that need to be evaluated.
A naiveway of performing the hyper-parameter search is Grid-Search, which defines the search space as agrid and exhaustively evaluates each grid configuration.
However, Grid-Search is a time-consumingprocess, meaning that thousands to millions of configurations would need to be evaluated if thehyper-parameter space is large.
In order to find optimal hyper-parameter settings quickly, Bayesianoptimization-based hyper-parameter search algorithms have been developed.
To investigate theeffectiveness of A UTOMATA across the spectrum of search algorithms, we used the Random Searchmethod and the Bayesian optimization-based TPE method as representative hyper-parameter searchalgorithms.
We provide more details on Random Search and TPE in Appendix E.
Component-2: Subset based Configuration Evaluation - Earlier, we discussed how a hyperparameter search algorithm presents a set of potential hyper-parameter configurations that need to beevaluated when tuning hyper-parameters.
Every time a configuration needs to be evaluated, prior worktrained the model on the entire dataset until the resource allocated by the hyper-parameter scheduler isexhausted.
Rather than using the entire dataset for training, we propose using subsets of informativedata selected based on gradients instead.
As a result, given any hyper-parameter search algorithm, wecan use the data subset selection to speed up each training epoch by a significant factor (say 10x) , thusimproving the overall turnaround time of the hyper-parameter tuning.
We use G RAD M ATCH [23], agradient based subset selection strategy in A UTOMATA because G RAD M ATCH is able to preservethe original ordering of hyper-parameters better than R ANDOM and other baselines even when usingsmall subset sizes as shown in Figure 1b.
Hence, the critical advantage of A UTOMATA is that wecan achieve speedups while still retaining the hyper-parameter tuning algorithm’s performance in finding the best hyper-parameters.
We provide more details on hyper-parameter ordering retentionexperiments in Section 3.
The fundamental feature of A UTOMATA is that the subset selected by
A UTOMATA changes adaptively over time, based on the classifier model training.
Thus, instead ofselecting a common subset among all configurations, A UTOMATA selects the subset that best suitseach configuration.
We present the gradient-based subset selection process of A UTOMATA below.
Gradient Based Subset Selection ( GSS ): A UTOMATA adopts G RAD M ATCH [23], a gradient basedsubset-selection strategy, to select a subset S and its associated weight vector w such that theweighted subset loss gradient best approximates the entire training loss gradient.
The subset selectionof A UTOMATA for i th configuration at time step t is as follows:
[Equation]
The additional regularization term prevents the assignment of very large weight values to datasamples, thereby reducing the possibility of overfitting on a few data samples.
Killamsetty et al.[23] proved that the optimization problem given in Equation (1) is approximately submodular.
Therefore, the above optimization problem can be solved using greedy algorithms with approximationguarantees [9, 37].
Therefore, like Killamsetty et al.
[23], we also use a greedy algorithm calledorthogonal matching pursuit (OMP) to solve the above optimization problem.
The goal of A UTOMATAis to accelerate the hyper-parameter tuning algorithm while preserving its original performance.
Efficiency is an essential factor that A UTOMATA considers even when selecting subsets.
Due to this,we employ a faster per-batch subset selection introduced in the work [23] in our experiments, whichis described in the following section.
Per-Batch Subset Selection: Instead of selecting a subset of data points, one selects a subset of mini-batches by matching the weighted sum of mini-batch training gradients to the full trainingloss gradients. We visualize the difference between per-sample and per-batch subset selection in
Figure 5 of the Appendix.
Therefore, one will have a subset of selected mini-batches and theassociated mini-batch weights.
One trains the model on the selected mini-batches by performingmini-batch gradient descent using the weighted mini-batch loss.
Let us denote the batch size as B , and the total number of mini-batches as b N = NB , and the training set of mini-batches as D B .
Let us denote the number of mini-batches that needs to be selected as b k = kB .
Let us denotethe subset of mini-batches that needs to be selected as S B i and denote the weights associated withmini-batches as w B i = { w B i 1 , w B i 2 · · · w B ik } for the i th model configuration.
Let us denote themini-batch gradients as ∇ θ L B 1 T ( θ i ) , · · · , ∇ θ L B bN T ( θ i ) be the mini-batch gradients for the i th modelconfiguration.
Let us denote L B T ( θ i ) =
(cid:80) k ∈ [1 ,b N ] L B k T ( θ i ) be the loss over the entire training set.
The subset selection problem of mini-batches at time step t can be written as follows:
[Equation]
In the per-batch version, because the number of samples required for selection is b k is less than k ,the number of greedy iterations required for data subset selection in OMP is reduced, resulting in aspeedup of B × .
A critical trade-off in using larger batch sizes is that in order to get better speedups,we must also sacrifice data subset selection performance.
Therefore, it is recommended to use smallerbatch sizes for subset selection to get a optimal trade-off between speedups and performance.
Asolution to the batch size dependency is to use a smaller batch size only for subset selection andcombine the smaller batches to produce larger batches for training.
In order to demonstrate that theeffectiveness of A UTOMATA is not dependent on the size of the training batch, in our experiments on
Images, we use a fixed batch size (instead of as a hyper-parameter) of B = 20 , and on text datasets,we use the batch size as a hyper-parameter with B ∈ [16 , 32 , 64] .
Apart from per-batch selection,we use model warm-starting to get more informative data subsets.
Further, in our experiments,we use a regularization coefficient of λ = 0 after analyzing the performance of A UTOMATA withdifferent values of λ .
We present the ablation study on λ in Appendix G.3.1.
We give more details onwarm-starting below.
Warm-starting data selection: We warm-start each configuration model by training on the entiretraining dataset for a few epochs similar to [23].
The warm-starting process enables the model to have informative loss gradients used for subset selection.
To be more specific, the classifier model istrained on the entire training data for T w = κTkN epochs, where k is the coreset size, T is the totalnumber of epochs, κ is the fraction of warm start, and N is the size of the training dataset.
We use aκ value of 0 ( i.e. , no warm start) for experiments using Hyperband as scheduling algorithm, and a κvalue of 0 .
35 for experiments using ASHA.
We present the ablation studies for κ in Appendix G.3.2.
Component-3: Hyper-parameter Scheduling Algorithm - Hyper-parameter scheduling algorithmsimprove the overall efficiency of the hyper-parameter tuning by terminating some of the poorconfigurations runs early.
In our experiments, we consider Hyperband [32], and ASHA [34], whichare extensions of the Sequential Halving algorithm (SHA) [20] that uses aggressive early stoppingto terminate poor configuration runs and allocates an increasingly exponential amount of resourcesto the better performing configurations.
SHA starts with n number of initial configurations, eachassigned with a minimum resource amount r .
The SHA algorithm uses a reduction factor η toreduce the number of configurations in each round by selecting the top 1 η th fraction of configurationswhile also increasing the resources allocated to these configurations by η times each round.
Wediscuss Hyperband and ASHA and the issues within SHA that each of them addresses in moredetail in Appendix F. Detailed pseudocode of the A UTOMATA algorithm is provided in Appendix Cdue to space constraints in the main paper.
We use the popular deep learning framework [41] forimplementation of A UTOMATA framework, Ray-tune[36] for hyper-parameter search and schedulingalgorithms, and CORDS [22] for subset selection strategies.
Experiments
In this section, we present the effectiveness and the efficiency of A UTOMATA framework for hyperparameter tuning by evaluating A UTOMATA on datasets spanning text, image, and tabular domains.
Further, to assess A UTOMATA ’s effectiveness across the spectrum of existing hyper-parameter searchand scheduling algorithms, we conduct experiments using combinations of different search andscheduling algorithms.
As discussed earlier, we employ Random Search [43], TPE [4] as representative hyper-parameter search algorithms, and Hyperband [32], ASHA [34] as representativehyper-parameter scheduling algorithms.
It should be noted that the combination of TPE and HyperBand is not the same as the BOHB algorithm [11].
In BOHB [11], a multivariate kernel densityestimator (KDE) was used, whereas the TPE [4] used a hierarchy of one-dimensional KDEs.
However, we believe the takeaways would remain the same even with other approaches.
We repeat eachexperiment five times on the text and tabular datasets, thrice on the image datasets, and report themean accuracy and speedups in the plots.
Below, we provide further details on datasets, baselines,models, and the hyper-parameter search space used for experiments.
Baselines: Our experiments aim to demonstrate the consistency and efficiency of A UTOMATAmore specifically, the effectiveness of A UTOMATA ’s gradient-based subset selection (GSS) for hyperparameter tuning.
As baselines, we replace the GSS subset selection strategy in A UTOMATA withdifferent subset selection strategies, namely R ANDOM (randomly sample a same sized subset as A UTOMATA from the training data), C RAIG [38] (a gradient-based subset selection proposed forefficient supervised learning), and F ULL (using the entire training data for model training duringconfiguration evaluation).
For ease of notation, we refer to baselines by the names of correspondingsubset selection strategies.
Note that by C RAIG baseline, we mean the faster per-batch version of CRAIG [38] for subset selection shown [23] to be more efficient than the original.
In addition, forall methods, we do not use any warm-start for experiments with Hyperband and use a warm start ofκ = 0 .
35 for experiments with ASHA.
We give more details on the reason for using warm-start with ASHA and no warm-start with Hyperband in Appendix G.3.2.
We perform experiments with differentsubset size fractions of 1%, 5%, 10%, and 30%.
In our experiments, we compare our approach’saccuracy and efficiency (time/energy) with Full training, Per Batch CRAIG, and Random selection.
Datasets, Model Architecture, and Experimental Setup:
To demonstrate the effectivenessof A UTOMATA for hyper-parameter tuning, we performed experiments on datasets spanning text,image and tabular domains.
Text datasets include SST2 [47], SST5 [47], glue-SST2 [51], and TREC6 [35, 18].
Image datasets include CIFAR10 [28], CIFAR100 [28], and Street View House
Numbers (SVHN) [39].
Tabular datasets include DNA, SATIMAGE, LETTER, and CONNECT-4from LIBSVM (a library for Support Vector Machines (SVMs)) We give more details on datasetsizes and splits in Appendix G.2. For the Text datasets, we use the LSTM model (from PyTorch)
[Figure]
with trainable GloVe [42] embeddings of 300 dimension as input.
For Image datasets, we use the ResNet18 [16] and ResNet50 [16] models.
For Tabular datasets, we use a multi-layer perceptronwith 2 hidden layers.
Once the best hyper-parameter configuration is found, we perform one morefinal training of the model using the best configuration on the entire dataset and report the achievedtest accuracy.
We use final training for all methods except F ULL since the models trained on smalldata subsets (especially with small subset fractions of 1%, 5%) during tuning do not achieve hightest accuracies.
We also include the final training times while calculating the tuning times for a morefair comparison 5 For text datasets, we train the LSTM model for 20 epochs while choosing subsets(except for F ULL )
every 5 epochs.
The hyper-parameter space includes learning rate, hidden size& number of layers of LSTM, batch size of training.
Some experiments (with TPE as the searchalgorithm) use 27 configurations in the hyper-parameter space, while others use 54.
More detailson hyper-parameter search space for text datasets are given in Appendix G.4.1.
For image datasets,we train the ResNet [16] model for 300 epochs while choosing subsets (except for F ULL ) every 20epochs i.e., R = 20 .
We present an ablation study analyzing the effect of epoch interval ( R ) on theperformance of A UTOMATA in Appendix G.3.3.
Based on the results, we observe that for visionexperiments, using a R value of 20 gives best efficiency vs performance tradeoff.
We use a Stochastic Gradient Descent (SGD) optimizer with momentum set to 0.9 and weight decay factor set to 0.0005.
The hyper-parameter search space consists of a choice between the Momentum method and Nesterov Accelerated Gradient method, choice of learning rate scheduler and their corresponding parameters,and four different group-wise learning rates.
We use 27 configurations in the hyper-parameter spacefor Image datasets.
More details on hyper-parameter search space for image datasets are given in Appendix G.4.2.
For tabular datasets, we train a multi-layer perceptron with 2 hidden layers for 200epochs while choosing subsets every 10 epochs.
The hyper-parameter search space consists of achoice between the SGD optimizer or Adam optimizer, choice of learning rate, choice of learningrate scheduler, the sizes of the two hidden layers and batch size for training.
We use 27 configurationsin the hyper-parameter space for Tabular datasets.
More details on hyper-parameter search space fortabular datasets are provided in Appendix G.4.3.
Hyper-parameter Ordering Retention Experiments: We evaluate the effectiveness of differentsubset selection strategies in preserving original hyper-parameter ordering by comparing the orderingobtained using A UTOMATA ’s gradient-based subset selection (GSS), R ANDOM , C RAIG strategieswith that obtained using full data.
Intuitively, we want to analyze whether the original hyperparameter ordering is preserved even when using small subsets for model training.
To examine this,we experiment on the CIFAR10 [28], Trec6 [35, 18] datasets using a ResNet18 [16] model and LSTMmodel respectively.
Hyper-parameter search for CIFAR10 dataset includes a grid search over 144configurations of four group-wise learning rates of ResNet18 model, optimizer, and training batch size. Hyper-parameter search for TREC6 dataset includes a grid search over 108 configurations of learningrate, optimizer, LSTM hidden size, training batch size, and number of final fully connected layers.
Figure 1b shows the Spearman rank correlation values between the hyper-parameter ordering obtainedusing 1%, 5%, and 10% subsets selected by R ANDOM , C RAIG , and A UTOMATA ’s gradient-basedsubset selection (GSS) and Full data hyper-parameter ordering on CIFAR10 [28] and TREC6 [35, 18]datasets.
Figure 1b demonstrates that GSS is more effective than R ANDOM and C RAIG in preservinghyper-parameter ordering even when using small subsets.
Hyper-parameter Tuning Results: Results comparing the accuracy vs. efficiency tradeoff of different subset selection strategies for hyper-parameter tuning are shown in Figure 3.
Performance is compared for different sizes of subsets of training data: 1%, 5%, 10%, and 30% along with four possiblecombinations of search algorithm (Random or TPE) and scheduling algorithm (ASHA or Hyperband).
Text datasets results: Sub-figures(3a, 3b, 3c, 3d) show the plots of relative test error vs. speed ups,both w.r.t full data tuning for SST5 dataset with different combinations of search and schedulingmethods.
Similarly, in sub-figures(3e, 3f, 3g, 3h) we present the plots of relative test error vs. speedups for TREC6 dataset.
From the results, we observe that A UTOMATA achieves best speed up vs.accuracy tradeoff and consistently gives better performance even with small subset sizes unlike otherbaselines like R ANDOM , C RAIG .
In particular, A UTOMATA achieves a speedup of 9.8 × and 7.35 ×with a performance loss of 2.8% and a performance gain of 0.9% respectively on the SST5 datasetwith TPE and Hyperband.
Additionally, A UTOMATA achieves a speedup of around 3.15 × , 2.68 ×with a performance gain of 3.4%, 4.6% respectively for the TREC6 dataset with TPE and ASHA.
Image datasets results: Sub-figures(3i, 3j, 3k, 3l) show the plots ofrelative test error vs. speed ups, both w.r.t full data tuning for CIFAR10dataset with different combinations of search and scheduling methods.
Similarly, sub-figures (3m, 3n, 3o, 3p) show the plots of relative test errorvs. speed ups on CIFAR100.
The results show that A UTOMATA achievesthe best speed up vs. accuracy tradeoff consistently compared to otherbaselines.
More specifically, A UTOMATA achieves a speedup of around15 × , 8.7 × with a performance loss of 0 .
14% respectively on the CIFAR10 dataset with Random and Hyperband.
Further, A UTOMATAachieves a speedup of around 3.7 × , 2.3 × with a performance gain of1% , 2% for CIFAR100 dataset with TPE and ASHA.
In Figure 4, weshow the effectiveness of the A UTOMATA with large configuration spaceby repeating the experiment on the CIFAR10 dataset with 110 configurations using Random as thesearch algorithm and Hyperband as the scheduler.
Tabular datasets results: Sub-figures(3q, 3r, 3s, 3t) show the plots of relative test error vs. speedups for the CONNECT-4 dataset.
In practice, larger subset sizes should lead to better performance.
However, the plots(3q, 3r, 3s, 3t) do not depict the same picture.
This is due to the inherent classimbalance within the CONNECT-4 dataset.
In this case, smaller subset sizes with gradient basedsubset selection mitigated the effects of class imbalance and enabled better model performance byselecting a similar proportion of samples from each class.
Furthermore, using larger subset sizescaused gradient-based subset selection to select more samples from overrepresentative classes, thusreducing the final model’s accuracy.
Finally, A UTOMATA consistently achieved better speedup vs.accuracy tradeoff compared to other baselines on CONNECT-4 as well.
Despite the fact that C RAIG and GSS have similar optimization problems, the performance of
C RAIG is poor compared to A UTOMATA , since C RAIG suboptimally optimizes an upper bound of thegradient error term.
In addition, C RAIG is slower from a computational complexity standpoint sinceit requires the construction of a similarity kernel.
Owing to space constraints, we provide additionalresults showing the accuracy vs. efficiency tradeoff on additional text, image, and tabular datasetsin the Appendix G.5.
It is important to note that A UTOMATA obtains better speedups when usedfor hyper-parameter tuning on larger datasets and larger models (in terms of parameters).
Apartfrom the speedups achieved by A UTOMATA , we show in Appendix G.6 that it also achieves similarreductions of energy consumption and CO2 emissions, thereby making it more environmentallyfriendly.
A UTOMATA can be used in conjunction with any hyper-parameter search algorithm andscheduler.
However, It should be noted that the speedups obtained using A UTOMATA are heavilydependent upon the scheduler.
Some schedulers, such as ASHA, are highly efficient and caneffectively discard poor hyper-parameter configurations early, thus reducing the advantages causedby using data subsets.
Therefore, the speedup achieved by A UTOMATA is approximately 3x whenusing ASHA as a scheduler and around 10x-15x when using Hyperband.
Nevertheless, we observethat using A UTOMATA still reduces tuning time and thus power consumption.
Conclusion, Limitations, and Broader Impact
We introduce A UTOMATA , an efficient hyper-parameter tuning framework that uses intelligent subsetselection for model training for faster configuration evaluations.
Further, we perform extensiveexperiments showing the effectiveness of A UTOMATA for Hyper-parameter tuning.
In particular, itachieves speedups of around 10 × - 15 × using Hyperband as scheduler and speedups of around 3 ×even with a more efficient ASHA scheduler.
A UTOMATA significantly decreases CO2 emissions andenergy-efficient, reducing the environmental impact of hyper-parameter tuning on society at large.
We hope that the A UTOMATA as framework will encourage community to consider subset selectionapproaches for faster hyper-parameter tuning, helping us move closer to the goal of Green AI [44].
Research like this will help substantially reduce the cost of training large models and AutoML,thereby help democratize machine learning among smaller companies, individuals and academicgroups.
One of the limitations of A UTOMATA is that in scenarios in which no performance loss isdesired, we do not know the minimum subset size that gives the best speed up and, therefore, need touse larger subset sizes such as 10%, 30%.
In the future, we consider adapting the subset size basedon model performance for each configuration to remove the dependency on subset size.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] .
Please refer to Section 4 of the
main paper.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] .
We
discuss broader impacts and societal impacts in Section 4 of the main paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer to
Appendix A
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please refer to Appendix G.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Please refer to Appendix G.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Appendix G.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Please refer to Appendix B
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We are providing the source code in the supplementary material(
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We are using publicly available datasets, the authors pointed us
to the links.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
Our data does not use personally identifiable
information or offensive content.
Check Appendix G.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
, if
applicable?
[N/A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]