Towards Learning Universal Hyperparameter Optimizers with Transformers
Anonymous Author(s)
AfﬁliationAddress
Abstract
Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efﬁciency over objectivefunctions from a similar distribution.
However, existing methods are restrictedto learning from experiments sharing the same set of hyperparameters.
In thispaper, we introduce the O PT F ORMER , the ﬁrst text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy andfunction prediction when trained on vast tuning data from the wild.
Our extensiveexperiments demonstrate that the O PT F ORMER can imitate at least 7 different
HPO algorithms, which can be further improved via its function uncertainty estimates.
The O PT F ORMER also learns a robust prior distribution for hyperparameterresponse functions, and can thereby provide more accurate and better calibratedpredictions than Gaussian Processes.
This work paves the path to future extensionsfor training a Transformer-based model as a general HPO optimizer.
Introduction
The emergence of public machine learning data platforms such as OpenML [1] and hyperparameteroptimization (HPO) services such as Google Vizier [2], Amazon SageMaker [3] and Microsoft
Azure [4] have made large-scale datasets containing hyperparameter evaluations accessible.
There isgrowing interest in leveraging such data to meta-learn hyperparameter optimization algorithms [5–8].
However, dealing with large datasets consisting of experimental trials in the wild can be challenging,due to large variations in HPO problems and their associated text metadata (e g. shown later in Table1).
Thus, most meta and transfer-learning HPO methods [7–16] consider a restrictive setting whereall tasks must share the same set of hyperparameters so that the input data can be represented asﬁxed-sized vectors.
Consequently, such methods only exploit a small portion of the available data tolearn priors.
This drawback is more severe for large datasets which contain signiﬁcant amounts ofuseful information.
To overcome these limitations, we introduce the O PT F ORMER , a general hyperparameter optimizationframework based on Transformers [17].
Transformers have demonstrated excellent performancein many data tasks, ranging from natural language [18], images [19, 20], biological data [21, 22],code [23, 24], and control [25, 26].
Here, we investigate how to use a Transformer as a universalinterface for modelling experimental data and learn HPO algorithms, as given a sufﬁcient amount ofdata, a Transformer can potentially learn a more complex prior distribution than standard Bayesian
Optimization (BO) with Gaussian processes (GPs), especially as the Transformer possesses certaincomputational advantages over GPs for large datasets.
We introduce a serialization scheme to convert a combination of any metadata and an optimizationtrajectory into text, represented as a sequence of tokens, and formulate the HPO task as a sequencemodeling problem.
We adopt a supervised learning approach, by learning to predict parameters and
OptFormer c o n v n e t
o n c
i f a r, m e t r i c s : acc u
r ac y
o p t _ k w .
l r , d o u b l e , [ 1 e - 6 , 1 e - 2 ] ; o p t _ t y p e , ca t e go r
i ca l , [ “ S G D ” , A d a m ” ]
[Figure]
Metadata
[Figure]
hyperparameter response functions from tuning data (See Fig. 1).
In order to further improve optimization performance, we augment the model by utilizing its own function prediction inference (Section 4.3).
Extensive experiments on both public and private datasets demonstrate the
O PT F ORMER ’s promising tuning and generalization abilities.
In summary, our contributions are as follows:
• We formulate, to the best of our knowledge, the ﬁrst meta-learning HPO framework to learn bothpolicy and function priors from data across different search spaces.•
The O PT F ORMER is capable of learning the behaviors of 7 diverse blackbox optimization algorithms relying on a broad class of methods (non-adaptive, evolutionary, and Bayesian).• Furthermore, the O PT F ORMER learns the prior over objective functions and provides both accurateand well calibrated predictions, in many cases signiﬁcantly surpassing GPs in log-predictivelikelihood and expected conﬁdence error (ECE) [27].
• Lastly, O PT F ORMER policies augmented with Thompson Sampling are competitive HPO algorithms.
To the best of our knowledge, this is the ﬁrst time Transformers are augmented withacquisition functions for online adaptation.
Preliminaries
2.1 Meta-learning for hyperparameter optimization
HPO aims to ﬁnd a set of hyperparameters x from search space X to maximize a model performancemetric, y = f
( x ) , often referred to as a response function.
Table 1 shows an example of HPOexperimental data.
Following the HPO nomenclature [2, 28], an experimental study consists ofmetadata ( m ) and a history of trials ( h ).
The metadata contains arbitrary unstructured information,including but not limited to descriptions of the problem, optimization algorithm, names, types andvalue ranges of hyperparameters.
The history after t trials, h t = ( x 1 , y 1 , . . .
, x t , y t ) , contains asequence of trials, each of which consists of a parameter suggestion x and function value y .
The goal of the meta-learning approach for HPO is to learn the shared knowledge among the objectivefunctions f from a dataset of multiple tuning experiments represented as studies and to obtain anoptimal HPO algorithm for new hyperparameter tuning tasks from a similar distribution to those inthe dataset.
An HPO algorithm π maps the metadata and history to a distribution over hyperparameter suggestions,i.e. π ( x t +1 | m, h t ) .
Using the terminology of ofﬂine RL [29], we refer to the algorithm used togenerate the trajectories in a dataset as the behavior policy π b .
We primarily consider search spaces X with a ﬁxed number D of hyperparameters per task, andhence x = ( x (1) , . . .
, x ( D ) ) , with each hyperparameter x ( d ) being of type DOUBLE , INTEGER ,
DISCRETE , or CATEGORICAL (see Appendix A.1 for details).
More complex search spaces can besupported as discussed in Section 7.
2.2 Transformer model
[Table]
The Transformer model is an efﬁcient attention-based neuralnetwork architecture for sequence modeling [17].
We adoptthe T5 Transformer encoder-decoder architecture [30].
Theencoder and decoder each consist of a stack of multi-head selfattention layers which construct pairwise interactions betweenpositions, followed by position-wise feed-forward networks.
The encoder converts a sequence of input token representationsm , to a sequence of continuous embeddings, which is fed tothe decoder to generate a sequence of output tokens h oneelement at a time (see Fig. 1).
Related work
There has been a rich set of works in meta-learning and transferlearning by modifying speciﬁc core components of the BOpipeline, such as the acquisition function or the GP, in order totackle
BO’s myopic behavior, or obtaining more informationfrom similar tasks.
For instance, approaches include learningnew acquisition functions [31], multi-task BO [7–13] and BOfor transfer learning using contextual GPs [14–16].
[32] alsostudies the use of meta-BO for hyperparameter tuning tasksin machine learning.
However, all of these works consider aﬁxed search space.
A more radical meta-learning approach to non-differentiableoptimization trains recurrent neural networks (RNNs) as neural optimizers from scratch.
[33] ﬁrst proposed training an
RNN with gradient descent to optimize blackbox functionsand hyperparameters while [34, 35] train RNNs using reinforcement learning (RL) to solve RL tasks.
Unfortunately,prior works are limited to ﬁxed search spaces and only useonline generated data, constraining the training objectives tobe cheaply computable.
In this work, we wish to overcome the limitations of previousworks by exploiting the Transformer architecture.
Numerous works have demonstrated Transformers’ strong capabilities in ﬂexible symbolic and numerical manipulation.
On the symbolic side,
Transformers have been shown able to manipulate symbolic mathematical expressions [36–38] andgenerate code [23, 24].
Furthermore, on the numerical side, Transformers have also been shown ableto perform linear algebra computations [39], Bayesian Inference [40], and ofﬂine RL [25, 26, 41].
For AutoML speciﬁcally, [42] has demonstrated Transformers’ and analogous graph neural networks’abilities to use dataset descriptions and metadata to generate classiﬁcation and data preprocessingpipelines.
However, to date, there has been little effort in attacking the full problem of hyperparametertuning in the blackbox optimization setting.
In this paper, the challenging task of learning algorithmsfrom blackbox optimization trajectories can be seen as a signiﬁcant extension of both symbolic andnumerical manipulation.
Since the underlying algorithm can be composed of multiple symbolic andmathematical operations with unbounded complexity, the model must infer potentially very complexbehavior over long horizons.
Universal interface and model for hyperparameter optimization
In this section, we provide a universal interface for modeling HPO studies with mixed textual andnumerical information as a sequence of discrete tokens.
We train our O PT F ORMER as a generative
[Table]
model on a given dataset and explain how to use the O PT F ORMER ’s parameter and function predictionabilities to implement an HPO policy
[Table]
4.1 Study tokenization
To generalize over HPO problems of different parameter sizes, types, and metadata, we propose toserialize the study as a one-dimensional textual sequence, also advocated in [26].
Unfortunately, anaive serialization approach, e
[Table]
g. via JSON [43], will produce unnecessarily long sequences
[Table]
To improve scalability, we compress the textual representation of metadata m by removing redundant phrases and punctuation (e.g., "parameter", quotes) and encoding keywords (e.g., "name","algorithm") and enumerating types (e.g. "DOUBLE") into single tokens.
For the historical sequence h , we convert every DOUBLE and INTEGER parameter along with everyfunction value into a single token, by normalizing and discretizing them into integers, with anquantization level of Q = 1000 ; e.g.¯ x = int[ x norm · Q ] , where x
norm =
( x − x min ) / ( x max − x min ) .
[Equation]
The range of x is deﬁned by the search space and the range of y is obtained from observed values inh .
For other types, we use the index in their value set.
The shortened text string is then converted to a sequence of tokens via the SentencePiece tokenizer [44](see Table 2 for an example).
Every trial is represented by text, which is represented as a sequence ofnormalized and quantized tokens, (cid:104) ¯ x (1) t , . .
, ¯ x ( D ) t , (cid:63), ¯ y t , "|" (cid:105) , where the token (cid:63) separates parameterand function values and "|" separates trials.
See Appendix A.2 for further details on tokenization.
4.2 Model and training loss
After tokenization, the converted historical sequence is as follows:
[Equation]
We can now apply a Transformer model to learn the conditional distribution of tokens in ¯ h usingthe chain rule, given the metadata ¯ m , as depicted in Fig.
The joint distribution is presented in
Appendix D.1.
Given a dataset D of hyperparameter optimization studies, we train the O PT F ORMER by maximizingthe weighted log-likelihood for each study ( m, h ) ∼ D :
[Equation]
with w n =
if ¯ h ( n ) ∈ { (cid:63), "|" } and w n = 1 otherwise.
That is, we mask out the separator tokens ( (cid:63) ,"|") and predict parameter ¯ x and function tokens ¯ y only.
Note that ¯ h ( n ) denotes the n -th token, thatis the n -th element of the list in Equation (2), and ¯ h (1: n − 1) denotes all tokens up to the ( n − 1) -thtoken.
Further details and data augmentations are provided in Appendix D.2.
4.3 Inference and decoding
Parameter prediction: To decode the predicted parameter token ¯ x ( d ) t back to its original parameterrange, we truncate the output distribution to the vocabulary range corresponding to valid parametervalues [0 , Q ) and reverse our tokenization procedure in Section 4.1.
For a DOUBLE or INTEGERparameter x , we use a piecewise constant distribution:
[Equation]
For all other parameter types, ¯ x corresponds to the index of the set of feasible values.
Putting thesetogether, we may now sample parameter x t from the model’s prior distribution and thus deﬁne an
HPO policy:
[Equation]
As we use a supervised learning loss, we expect π prior to approximate the behavior policy π b .
Note that traditional BO algorithms require running Bayesian inference and then conducting a globalsearch in the hyperparameter space with an acquisition function.
Thus the runtime complexity ofmaking one hyperparameter suggestion is cubic in t for a typical GP-based BO method that performs
ARD each iteration [45].
In contrast, generating one suggestion by the O PT F ORMER consists ofdecoding D parameter tokens with an input sequence of ( D + 3) t tokens, which are then parsed intothe D parameter values, producing a runtime of O
( D 2 t ) linear in t , with proper caching.
Function prediction: To decode the real-valued function y t from the discrete distribution
P θ (¯ y t | ¯ m, ¯ h t − 1 , ¯ x t ) , we construct the same piecewise constant distribution as in Eq.
(4) with therange [ y min , y max ] used in tokenization.
Note that the limited support of y will not be a concernfor HPO when either the range is known or we set the range large enough compared to observedvalues.
For more general use as a few-shot function prediction model, one could consider adoptingthe Riemann Distribution in [40], which supports an unbounded range.
Augmented HPO policies with function prediction: At best, the learned policy π prior can onlyperform as well as the original policy π b when using behavioral cloning.
However, we can takeadvantage of the model’s simultaneous function prediction ability to improve the policy with modelbased planning or ofﬂine RL techniques.
While a comprehensive study of policy improvementsfor Transformers is out of the scope of this work, we consider here a simple yet effective policyimprovement operator: sampling M = 100 candidate suggestions from π prior and choosing thesuggestion with the highest score deﬁned by an acquisition function u ( · ) as follows:
[Equation]
Common acquisition functions include Upper Conﬁdence Bound (UCB), expected improvement (EI),and Thompson Sampling (TS).
At a high level, this approach to combining imitated policies withfunction prediction is reminiscent of the idea behind the ofﬂine RL approach of BCQ [46].
Data
Training the O PT F ORMER requires HPO studies with optimization trajectories.
The most naturaldataset we possess is a large collection of about 750K proprietary hyperparameter tuning studies, denoted as RealWorldData 1 , each with on average 300 trials, covering a vast class of machine learningmodels with applications from vision, speech, NLP and robotics and representing a distribution of
HPO tasks in practice.
These studies were generated with a mixture of non-adaptive, evolutionary,and BO algorithms.
However, as the dataset does not contain sufﬁcient algorithm information, wehave to treat the corresponding behavior policy as a randomly mixed algorithm π b .
In addition, we create two new datasets based on public benchmarks.
HPO-B is the largest publicbenchmark for HPO containing about 1.9K tuning tasks, most of which use one of 16 shared searchspaces.
In the continuous evaluation setting, it ﬁts an XGBoost model to the trial data of every tuning task as the objective function.
For further control over speciﬁc function dimensions and properties, weuse the blackbox optimization benchmark BBOB [47], consisting of 24 types of synthetic functionswith customizable properties (dimension sizes, rotations, shifts, discretizations, noise types) werandomize over.
For each of the two public benchmarks (HPO-B and BBOB), we apply a ﬁxed set of 7 HPOalgorithms to generate a dataset of optimization trajectories.
In contrast to RealWorldData, we specifythe algorithm name in the metadata m as part of the conditioning input for our model.
The controlledalgorithms used are: (1) Grid Search, (2) Shufﬂed Grid Search, (3) Random Search, (4) Regularized
Evolution [48], (5) Hill-Climbing, (6) Eagle Strategy [49], and (7) Vizier’s GP-UCB [2].
Appendix Bcontains detailed explanations of the algorithms.
[Table]
Experiments
We train a single Transformer model with 250M parameters on the union of the three datasetsdescribed above, RealWorldData, HPO-B, and BBOB (hyperparameter details in Appendix D.2).
Each dataset contains a corresponding “test” set of functions, either using synthetic functions (BBOB)or ﬁtting a machine learning model to obtain the objective (RealWorldData, HPO-B).
We evaluatemainly on the two natural HPO benchmarks, RealWorldData and HPO-B.
To aggregate results across functions with different output scaling, we deﬁne our performance metricat trial t as the best-so-far normalized function value max i ∈{ 1: t } ( y i − y rand ) /
( y max − y rand ) , wherey rand is the median of function values randomly sampled in the search space to be robust to outliers,and y max is the maximum, if known, or best value found by any algorithm.
For the HPO-B benchmark,we use the recommended bounds provided in [5].
We also consider other metrics when comparingdifferent algorithms in Appendix E.3, including the performance proﬁle and average ranking.
Weﬁnd our results are consistent over different metrics.
Because the O PT F ORMER is trained to predict the conditional distributions of parameter and functionvalues, we would like to answer the following questions when evaluating on unseen test problems:
Can the O PT F ORMER learn to imitate multiple HPO algorithms with one model?
(Section 6.1)
Can the O PT F ORMER learn a good prior over hyperparameter response functions?
(Section 6.2)3.
Is the O PT F ORMER a competitive approach for HPO?
(Section 6.3)6.1 Imitating HPO policies
We ﬁrst evaluate how well the O PT F ORMER can learn the conditional distribution of parametersuggestions given by the behavior policies in the dataset, and how well it can imitate multiplealgorithms.
As the algorithm’s name is contained in the metadata m , we can modify the behaviour ofthe policy π prior ( x t +1 | m, h t ) simply by altering this variable.
Fig.
2a compares two different policiesto the O PT F ORMER , when it is conditioned on the corresponding policy name.
We observe a goodmatch between the imitated algorithms and the O PT F ORMER (additional algorithms are shown in
Appendix E.1).
In Fig.
2b we run target policies on the BBOB dataset’s test functions and compare the optimizationtrajectories of the algorithms and the O PT F ORMER .
In Fig.
2c we compare the average and standarddeviation of the best normalized function values at trial 100.
Our model imitates most algorithmsvery accurately in both the mean and variance except for the most complicated algorithm, Vizier, [Figure] Regularized Evolution [Figure]
where π prior is slightly worse in the LUNACEK benchmark.
We expanded on this in Appendix E.1.
Because Vizier is the best performing HPO algorithm among all considered, the O PT F ORMER willimitate Vizier faithfully, although not perfectly, in the following experiments.
6.2 Learning priors for hyperparameter response functions
In this section, we assess the O PT F ORMER ’s ability to learn the conditional distribution of thefunction value as a few-shot function regressor.
Speciﬁcally, for every function in each test dataset,we repeatedly sample up to 200 random trials ( x 1 , y 1 , . . .
x t , y t ) , t ≤ 200 , and predict the conditionaldistribution p ( y t | x 1 , y 1 , . . .
, x t ) .
We compare with a GP model with output warping — detailsprovided in Appendix B.
We report the log-predictive likelihood log p ( y t | x t , . . . )
in Table 4.
[Table]
[Figure]
[Table]
[Figure] GP OptFormer As uncertainty estimation is important for HPO, we also evaluate how well the function predictivedistribution is calibrated.
When a predictive distribution p θ ( y | . . . ) matches the true distribution,the estimated CDF F ( y )
= (cid:82) y
−∞ p θ ( y (cid:48) | . . . ) dy (cid:48) will be uniformly distributed.
In Fig.
3, we plot thecumulative histogram of F ( y ) on RealWorldData test set and check the deviation from the diagonalline to assess goodness-of-ﬁt as proposed by Rosenblatt [50].
The O PT F ORMER has a smallerdeviation than GP almost across the entire range.
We also compare calibration performance using theexpected calibration error (ECE) [27].
Readers are referred to [27] for a detailed explanation of ECE.
We observe from Table 4 that the O PT F ORMER achieves better predictive likelihood and ECE thanthe GP on both datasets.
6.3 Augmenting a prior policy with function prediction
We evaluate the O PT F ORMER as a hyperparameter optimization algorithm on two benchmarks,
RealWorldData and HPO-B.
We compare our prior policy, the O PT F ORMER , and an augmentedpolicy with Thompson Sampling, O PT F ORMER (TS), against standard HPO baselines, including
Random Search, our implementation of GP-UCB, and the well-tuned Vizier service.
For HPO-B,we also include the GP (not to be confused with our GP-UCB) and DGP (GP with deep kernel)baseline results provided by the original paper [5].
Additionally, we include three recent transferlearning methods based on multi-task GP models: ABLR [12, 51], FSBO [7], and HyperBO [52, 53](implementation details in Appendix B).
Please note that all of these transfer learning methods requirelearning GPs on multiple tasks sharing the same search space.
Therefore, none of them apply to the
RealWorldData benchmark where every study has its own search space.
[Figure]
We show the trajectory of the best normalized function value averaged over all functions from eachbenchmark in Fig.
While the prior policy returned by the O PT F ORMER does not perform as wellas Vizier, it is comparable or slightly better than our GP-UCB baseline and ABLR.
The most signiﬁcant improvement is achieved when we augment our prior policy with Thompson
Sampling.
The resulting O PT F ORMER (TS) matches the performance of Vizier on RealWorldDatawith a diverse set of test functions, and performs signiﬁcantly better than all of the baselines on
HPO-B. This illustrates that the O PT F ORMER is able to learn the distribution of functions in themeta-training split and transfers to the meta-testing split.
We suspect the reason that the O PT F ORMERdoes not outperform Vizier on RealWorldData is due to the test functions being generated by a GPsurrogate, thus clearly resulting in an advantage for GP-based methods.
Furthermore, the temporaltrain/test splits in RealWorldData lead to a more difﬁcult distribution shift than the transfer setting in
HPO-B.
It is worth noting that to run 100 trials for about half of the test functions, the required history tokensequence is longer than the 1024-token length used in training, with the maximum length about twicethe training horizon.
The superior performance of O PT F ORMER (TS) thus demonstrates its goodgeneralization performance beyond the optimization horizon it is trained for.
6.4 Ablations
We provide further ablations on three important components for our policy:
Training dataset.
To understand the impact of the training datasets on the O PT F ORMER , we trainthree variants on individual datasets (O PT F ORMER -"R","H","B" respectively for RealWorldData,
HPO-B, BBOB) and study their transfer learning performances on HPO-B. Fig.
5a veriﬁes thattraining with in-domain data ("H") gives better performance than training over the more diverseacross-domain HPO RealWorldData data ("R"), which is better than training over the synthetic BBOBdata ("B").
Nonetheless, training on RealWorldData is enough to give comparable performanceto the best transfer learning baseline at the end of 100 trials.
While training on all of the datasets (O PT F ORMER ) only gives a slight advantage over O PT F ORMER -H, this suggests that more data doesnot hurt the model’s performance but rather may improve it, even if the extra data is out-of-domain.
Meta-data m .
While we have demonstrated how the O PT F ORMER ’s behavior can be controlled bythe algorithm name in metadata m , here we study whether the O PT F ORMER learns to depend on othermeta information.
At inference time, we provide minimum information in m (O PT F ORMER -min) byexcluding all textual information and parameter value ranges.
We only keep necessary informationsuch as parameter types and algorithm names.
Fig.
5b shows that the prior policy of O PT F ORMERmin performs comparably with the O PT F ORMER , partly due to the use of data augmentation (see
Appendix D.2).
The augmented policy O PT F ORMER -min (TS) (dashed orange) improves upon theprior policy but is signiﬁcantly worse than the full model, suggesting that the missing metadataimpacts the model’s predictions on function values.
Prior policy.
Section 6.3 demonstrated the beneﬁt of adding an acquisition function to the priorpolicy.
A natural question is whether a good prior policy is needed at all.
In Fig.
5c, we replacethe prior policy in O PT F ORMER (TS) with random search (Random Search (TS), dashed blue line).
While adding Thompson Sampling still improves this random search policy’s performance, the bestmethod requires both a good prior policy and the acquisition function.
[Figure]
Limitations and future extensions
We list a few limitations of this work and discuss some potential extensions.
(1) We did not considerparameters that do not always apply or are subject to dynamic constraints depending on otherparameter values.
Such parameters are common in AutoML [54] and NAS applications [55].
Ourwork can be extended to support these applications, by providing the conditional speciﬁcations astext in metadata m .
(2) We also considered only sequential optimization with a batch size of 1.
Tosupport parallel suggestions, one could apply random masking to input function value observations tosimulate scenarios with parallel pending trials [33].
(3) While we trained the Transformer to clonethe behavior policy ofﬂine, there is an extensive literature on ofﬂine RL [29] that could be appliedhere [25, 46, 56–59].
One could also consider meta-training acquisition functions as in [31] withinthe same model and online ﬁne-tuning as in [7, 41] (4) We considered a single objective function,though multiple objectives can be easily included by outputting multiple function tokens in a trial.
Conclusion
We presented ﬁrst step to learning a universal transformer model for hyperparameter optimizationfrom large scale datasets of tuning experiments with vastly different search spaces, and experimentdescriptions.
By training on a diverse set of synthetic or real-world tuning trajectories, we demonstrated the capacity of a single Transformer model to imitate 7 fundamentally different HPO policies,learn to make well calibrated few-shot function predictions, and provide promising optimizationperformance on unseen test functions comparable with the existing, long-tried GP-based baselines.
Many extensions are readily conceivable for future exploration. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the pa-
per’s contributions and scope?
[Yes] Abstract and introduction point out the speciﬁc
contributions in bullet points, which we have covered throughout or methodology and
experiments.
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] The main
potential negative societal impacts of our work come from the inherent associated risks
of using large Transformer models, such as bias.
However, currently this does not apply
to our work, as our data only contains numerical optimization data, and we are not
using language models as of yet.
Such a risk may grow however, if in future works,
we
e.g. use pretrained language models for warm-starting in our O PT F ORMER framework.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] The most relevant potential ethics concern is our use of RealWorldData.
However, as we mention in Appendix C, we anonymized the data to prevent users from
being identiﬁed in the model.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] Not applicable;
no theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] Not applicable; no
theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
the link to the primary T5x codebase in Appendix D.2, along with relevant hyperpa-
rameters.
But we do not include the training data.
The datasets generated on public
benchmarks, BBOB and HPO-B, can be reproduced by running publicly available HPO
algorithms.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] These are speciﬁed in Appendix A, Appendix C, and Appendix D.2.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In experiments, we plotted error bars using 1 standarddeviation over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have cited [47] for
our use of BBOB dataset, [5] for the HPO-B dataset, and [30] for training T5 models.
(b) Did you mention the license of the assets?
[Yes] For data, we mentioned the use of
Apache 2.0 License when using the HPO-B dataset.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We cited and linked the T5x codebase (see Appendix D.2) throughout this paper.
We
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The BBOB dataset is synthetic, and the HPO-B dataset is public.
For the RealWorldData dataset, users by default are agreeing to the condition that
their
tuning data may be used for research purposes.
We have also ensured anonymity over
the data by removing personally identiﬁable information.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content
Yes
] As discussed in Appendix C, we anonymized
user
details from studies in the RealWorldData dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Not applicable.
[Table]
[Table]
, if applicable
[Table]
[N/A
[Table]
[Table]
the estimated
[Table]
paid to
[Table]
and the total amount
spent on participant compensation?
[N/A] Not applicable
[Table]