Efficient Non-Parametric Optimizer Search for Diverse Tasks
Anonymous Author(s)
AffiliationAddress email
Abstract
Efficient and automated design of optimizers plays a crucial role in full-stack
AutoML systems.
However, prior methods in optimizer search are often limited bytheir scalability, generability, or sample efficiency.
With the goal of democratizingresearch and application of optimizer search, we present the first efficient, scalableand generalizable framework that can directly search on the tasks of interest.
Wefirst observe that optimizer updates are fundamentally mathematical expressionsapplied to the gradient.
Inspired by the innate tree structure of the underlying mathexpressions, we re-arrange the space of optimizers into a super-tree, where eachpath encodes an optimizer.
This way, optimizer search can be naturally formulatedas a path-finding problem, allowing a variety of well-established tree traversalmethods to be used as the search algorithm.
We adopt an adaptation of the Monte
Carlo method to tree search, equipped with rejection sampling and equivalentform detection that leverage the characteristics of optimizer update rules to furtherboost the sample efficiency.
We provide a diverse set of tasks to benchmark ouralgorithm and demonstrate that, with only 128 evaluations, the proposed frameworkcan discover optimizers that surpass both human-designed counterparts and prioroptimizer search methods.
Introductions
Motivated by a vision of democratizing machine learning, the central objective for automatedmachine learning (AutoML), such as automated architecture [22, 25, 28, 39, 53, 59, 62] / optimizer[10, 12, 15, 17, 57, 64] / loss [35] / augmentation search [36, 38], lies in reducing the need for expertdesign on a diverse set of tasks.
To achieve this goal, it is critical for AutoML systems to exhibit ahigh level of efficiency, so that they can be directly applied to a variety of tasks without consuminga humongous amount of computing resources.
A widely successful example of such an effort is
DARTS [39] in Neural Architecture Search (NAS), which reduces the search cost from thousands of
GPU days of early RL-based algorithms to a single digit, enabling direct application of NAS systemsto a wide range of tasks [31, 32, 37, 43, 48].
Inspired by the success of efficient NAS methods, we turn our attention to another important butmuch less studied area of AutoML - Automated optimizer search, where an efficient, scalableand generalizable framework is still absent.
Optimizer search aims to automatically designa suitable update function that takes gradients as inputs and produces update directions for theoptimizee’s parameters.
Pioneering work in this area, coined Learning to Optimize (L2O) , adoptsa data-driven approach by replacing human-designed update rules with a learnable parametricfunction [10, 12, 17, 57].
However, parametric optimizers are fundamentally not scalable to largemodels or datasets, as inferring its parameters typically requires expensive meta-learning steps suchas backpropagating through gradient descent [10, 15, 64].
Moreover, the learned optimizer oftengeneralizes poorly to even minor variants of its training task (Figure 3) [10, 64].
Poor scalability and generability prevent L2O from being served as a general-purpose optimizer search framework thatcan be directly applied to tasks of interest .
The aforementioned limitations of parametric optimizers bring our attention to another line of methodthat searches over the discrete space of non-parametric update functions 1 , which generally exhibitthe same level of scalability and generality as human-designed optimizers [15, 49].
NOS-RL [15]extends early RL-based NAS framework [22] to optimizer search, proposing to learn a sequentialcontroller to produce optimizer update rules according to a predefined pattern.
However, NOS-RL issample inefficient, requiring over 10k evaluations to find good candidates.
More recently, AutoMLZero [49, 65] proposes to search over the vast space of computer codes for the entire ML
pipeline(including the optimizer).
The excessive generality of its search space makes it even more costly torun than RL-based method.
The search cost of existing non-parametric optimizer search frameworksmakes them computationally prohibitive not only for practitioners to apply but also for researchers toanalyze.
With the goal of democratizing research and practical applications of automated optimizer design,we introduce the first efficient, scalable, and generalizable optimizer search framework that canbe directly applied to a wide range of tasks.
We observe that non-parametric update rules areessentially mathematical expressions, with an innate tree structure where nodes are elementary mathoperators and edges represent their I/Os.
Consequently, generating an update rule can be viewed asprogressively appending nodes to the expression tree until it is complete.
Inspired by this observation,we re-imagine the optimizer search space as a super-tree of mathematical expressions.
Each leaf nodeon the super-tree contains an optimizer, and the path towards it represents the generation processof that optimizer’s underlying expression.
With the tree-structured search space, optimizer searchcan be naturally formulated as a path-finding problem, allowing a wide range of well-establishedtree-traversal methods to be used as the search algorithm.
We show that a simple adaptation of
Monte Carlo Sampling [29, 51], equipped with our proposed rejection sampling and equivalent-formdetection, can already produce remarkable results on our search space within a fraction of budgetscompared with NOS-RL ( ∼ 1%).
We extensively evaluate the proposed framework on a diverse set of learning tasks: digit classification with MNISTNET [10], image classification with ConvNet [15], graph learning with(Cluster-)GAT [21, 27], norm-bounded adversarial attack on robustly trained models [20, 44, 45],and BERT fine-tuning on NLP datasets [33, 54].
These tasks cover both constraint and unconstrainedoptimizations and span over a large variety of models and datasets.
Despite the simplicity, the proposed framework is able to discover update rules that surpass human-designed optimizers and prioroptimizer search methods, with a budget of only 128 evaluations.
We hope the proposed frameworkcould lower the barrier of entry to practical non-parametric optimizer search, thereby providing anentry point for researchers and practitioners from ML community and beyond to study and utilizeautomated optimizer search systems.
Efficient, scalable and generalizable framework for optimizer search2.1 Optimizer design space
Notations and problem formulation Deep learning tasks are frequently expressed as optimizing aloss function L ( · ) defined over parameter domain θ ∈ Θ .
The minimizer of L can thus be obtainedby θ ∗ = arg min θ ∈ Θ L ( θ ) .
For differentiable functions, a standard optimizer typically takes theform of iterative gradient descent: θ t +1 =
θ t − γ ∗ ϕ ( ∇ θ L ( θ t )) , where t is the current iteration, γis the learning rate and ϕ denote the update function.
Existing optimizers primarily differ in theirdesign of update function ϕ ; For example, vanilla gradient descent uses identity mapping ϕ
( x ) = xas the update function, whereas Adam adopts a momentum-based dynamic learning rate schema:ϕ
[Equation]
L ( θ t )) 2 ) , where m ( · ) denotes the momentum function withan internal state.
The goal of optimizer search is to automatically find a suitable update function ϕ over some hypothesisspace Φ .
The hypothesis spaces used in prior work can be divided into two categories: non-parametricand parametric spaces.
Most human-designed optimizers belong to the first category, where the update function ϕ is not trainable.
Learnable optimizers, such as L2LGD2 [10] and SymbolicL2O [64], fallinto the second category.
Our work mainly focuses on non-parametric optimizer search, with the goalof providing an efficient, scalable and generalizable optimizer search framework that can be directlyapply to various tasks.
Optimizer update rules as expression trees The first step toward such a framework is to understand the structure of non-parametric optimizers.
We realize that, fundamentally, optimizers aremathematical expressions consisting of elementary operators ( + , − , sign () , inputs , e.t.c.).
Mathexpressions have an inherent tree structure that preserves its order of execution, where nodes areoperators and edges represent their I/Os.
For instance, Diagram 1 shows the expression tree of
Adam [9]:
[Equation]
[Figure]
where m 1 and m 2 denote the first and second order momentum (which can also be broken down intotheir own expression trees).
Therefore, the generation of an update rule, as a mathematical expression, can also be conducted viatop-down node selection: Take Adam as an example, we first select division ( / ) as the root node.
Forits left child, we pick m 1 , which is a leaf node and thus ends the branch.
For the right child, we select 103 √ , and subsequently pick m 2 to follow it.
At this point, there is no empty branches left, and weobtain the complete update rule for Adam.
[Figure]
A tree-structured search space Inspired bythe completion process of update rules, we rearrange all expressions into a super-tree , whereeach leaf node contains an update rule and eachpath represents its completion process.
Thesuper-tree can be generated in a top-down manner:
Starting from the root node with an emptyupdate rule, we generate each of its child nodesby inserting a different operator into the updaterule, and repeat this process for the generatednodes.
Consequently, an optimizer can be sampled by traversing the super-tree until a leaf nodeis reached.
Since the super-tree can grow infinitely deep, it is often desirable to restrict the tree to a predefined depth N , where only the paths thatcan be completed within depth N are included.
Figure 1 provides an instantiation of our super-tree,where the paths leading to Adam and SGD optimizers are displayed as an example.
The benefit of arranging the optimizer space into a tree is two folds.
Firstly, the tree-based searchspace is tight:
Proposition 1 Define the length of an update rule as the number of operators it includes, then theabove tree-based search space is tight : a super-tree with a maximum depth of N covers all updaterules of length no greater than N .
In a tight search space, all optimizers can be represented at the right level of complexity, allowingthem to be visited by the search algorithm without exploring unnecessarily deep into the super-tree.
Although tightness is a fairly obvious result for our space, it is not the case for the previous searchspace defined in NOS-RL, as we will explain later.
Secondly, with our super-tree, optimizer searchcan be naturally formulated as a path-finding problem, allowing a variety of well-established treetraversal methods to be deployed as search algorithms.
Contents To concretize the content of the search space, we allow three types of operators in theoptimizer update rule:
• a set of p 1 unary operators (e.g. log ( | · | ) , exp ( · ) , (cid:112) | · | , sign ( · ) , drop ( · ) )
• a set of p 2 binary operators (e.g. + , − , × , / , pow ( · , · ) )
• a set of L leaf values (input operators) containing gradients (e.g. g , m 1 ), decays (e.g.
cosine _ decay ), and constants (e
g. 1 , 2 )
This categorization of mathematical operators is not new, as it is also adopted in symbolic mathsolver [34] and NOS-RL [15].
Comparison with NOS-RL’s search space Although both NOS-RL [15] and our framework useelementary math operators as building blocks for optimizers, they have little in common in terms ofthe arrangement of the search spaces.
Optimizers in NOS-RL’s search space are formed by a chain ofpredefined motifs:
b ( u ( I ) , u ( I )) , where b , u , I denote binary, unary and input operators.
Due to thefixed structure of such motifs, NOS-RL’s search space is not tight: there exist many optimizers thattake extra longer sequences to express, potentially lowering their chance of being discovered by thesearch algorithm.
For instance, Diagram 2 shows an optimizer of length 5 , but it takes (10 − 1) nodes(two chained motifs) to represent it under NOS-RL’s arrangement; Moreover, NOS-RL’s search spacealso requires extra bypass operators (e.g. u ( x ) =
x and b ( x, y )
= x ) to cover even human-designoptimizers such as Adam and PGD, further increasing the complexity.
In contrast, our representationof optimizers is directly inspired by the innate structure of its underlying mathematical expressions,resulting in a tight tree-based search space.
In our search space, optimizer search can be naturallyformulated as a form of top-down path-finding problem.
In the next sections, we will detail ourchoice of algorithms for traversing the super-tree, as well as several techniques that leverage thecharacteristics of optimizer update rules to boost the sample efficiency.
2.2 Monte Carlo Sampling for tree traversal
We adopt a simple adaptation of Monte Carlo Sampling to tree traversal [29, 51, 66] (MCT) as thesearch algorithm.
The idea is to assign scores to the nodes in the super-tree (Figure 1), and use thesescores to guide the tree traversal.
We define the score of a node v as a Monte Carlo estimation overunrolling steps from v : If v is an internal node, we randomly generate a set of unrolled paths from vto the corresponding leaf nodes, and take the average score of the resulting optimizers as the score forv ; If v is a leaf node, we set its score to 0 as it cannot be expanded.
The search can thus be conductedas follows: 1).
Starting from the root node v (0) at level 0 , we generate all child nodes { v (1) } of v (0)by inserting each operator from the candidate pool to the update rule in v (0) ; 2).
From there, weselect the child node v ∗ (1) with the highest MC score to expand, and move on to the next level; 3).
The process is repeated until a predefined maximum search level is reached.
Algorithm ?? in the Appendix provides a detailed summary of the complete search process.
Directly applying the MCT algorithm to optimizer search would not perform well under limitedsearch budgets, due to two unique characteristics of optimizer update rules that challenge the sampleefficiency of the Monte Carlo estimates.
Firstly, the majority of mathematical expressions, whendeployed as optimizer update rules, perform poorly or even would not converge.
This is usually notthe case for other AutoML tasks such as neural architecture search, as most networks in the searchspace perform reasonably well.
The large body of poor-performing optimizers not only consumesprecious search budget, but also causes the MC estimation to be unstable.
Secondly, there existsmany mathematical redundancies in the expression space, for example: sign ( sign ( sign ( x )))
canbe reduced to sign ( x ) , and m 1 + √ m 2 √ mis equivalent to m 1 √ m 2 + 1 .
Identifying and eliminating theseredundancies would not only save budget, but also prevent the sampling distribution from biasingtoward mathematically simple and shallow update rules.
To address these issues and further boostthe sample efficiency, we propose two sets of techniques - rejection sampling and equivalent-formdetection.
When combined with these techniques, the simple MCT algorithm becomes particularlyeffective for the optimizer search task.
We will discuss them in detail in the following sections.
2.3 Rejection sampling
Eliminating poor optimizers with a train-free taskagnostic test Inspired by the characteristics of optimizer update rules, we develop a train-free taskagnostic test to eliminate poor optimizers withoutevaluating them.
We propose a necessary conditionfor a valid optimizer: it must produce an acute anglewith steepest descent direction (i.e. gradients).
Wecheck the validity of optimizers against this conditionand only evaluate those that pass the test.
For thetest to be task-agnostic, we feed the optimizer with abatch of random Gaussian vectors in place of actualgradients.
Formally, the descent test can be writtenas:
[Equation]
where λ d is a predefined threshold.
Although our descent test is by-no-mean comprehensive, itcan effectively rule out a large chunk of poor optimizers with negligible false-negative rates, asdemonstrated in Figure 2.
Reducing the variance of MC estimates via score thresholding After applying the descent test,there still remains a non-negligible portion of poor optimizers.
When sampled during the unrollingstep, these optimizers would drastically lower the Monte Carlo score of the stem node, causing the
MC estimation to exhibit high variance and thus become unreliable.
This adverse effect is especiallysevere under the efficient setting when the sample size is small.
Therefore, we propose to simplyreject candidates with scores lower than a predefined threshold, thereby removing them from the MCscores of the corresponding stem nodes.
2.4 Detecting and handling redundancies in mathematically equivalent forms
On-the-fly constraint tree-traversal for redundant path pruning One benefit of formulatingthe search problem as top-down path-finding is that we can easily apply constraints on-the-fly toeliminate undesirable branches - those that lead to mathematically redundant expressions in our case.
We identify three main categories of such constraints:
• child operator that nullifies its parent’s operator.
e g. − ( − x ) = x , ln ( e x ) =
x• child operator that is redundant under its parent.
e.g. clip ( clip ( x ))
=
clip ( x )• sequence of operators that reduces to a constant in the search space.
e.g. (cid:112) | sign ( x ) | = {+
+}
The complete sets of constraints we used can be found in the Appendix.
Enforcing these constraintsduring the traversal can effectively trim down the search tree, allowing the algorithm to explorebranches that lead to more diverse and complex expressions.
Hashing mathematically equivalent expressions Besides enforcing constraints during the traversal, it is also important to detect mathematically equivalent optimizers to avoid duplicated evaluations.
One can always apply off-the-shelf symbolic solvers to identify the equivalence of two expressions, ϕand ϕ ′ , by checking if ( ϕ − ϕ ′ ) can be reduced to 0 .
However, it could become extremely slow as thepool of evaluated optimizers { ϕ i } N 1 gets larger and larger, since we need to solve N pair of equationsevery time a new update rule is sampled.
Instead, we apply hashing to efficiently query the evaluatedcandidate pool for mathematically equivalent optimizers.
Concretely, we assign each optimizer ahash code, obtained by feeding a fixed probing vector as input to the optimizer and recording itsoutput.
When a newly sampled optimizer arrives, we only need to compare its code with the hashtable to check the existence of its equivalent form.
Empirically, it is much faster to run the proposedhashing-based checker than symbolic solvers.
Discussions and relationship to prior work
Automated optimizer design Optimization plays a crucial role in training deep learning models.
Generally, there does not exist one optimizer that aces all scenarios, as different tasks (dataset, architecture, loss, parameterization, e.t.c.) might favor different optimization methods [2, 10].
Thedemand for task-specific optimizers stimulates research interest in developing automated systemsfor optimizer design [10, 12, 15, 17, 40, 49, 57, 63, 64].
Early work adopts a data-driven method bymodeling the optimizer update with a parametric function [10, 12, 57].
L2LGD2 [10] deploys an
LSTM model as the update function that takes historical gradients as input and produces the updatedirection.
However, parametric optimizer search methods are fundamentally limited by its scalability,as inferring its parameters requires expensive meta-learning steps such as back-propagation throughoptimization [10, 57, 64].
Although SymbolicL2O [64] improves the scalability of the learned
LSTM optimizer by distilling it into a lightly-parameterized symbolic optimizer, it still requires apretrained LSTM model to begin with.
Instead of learning a parametric optimizer, NOS-RL [15]directly searches over a discrete space of non-parametric update functions comprised of mathematicaloperators.
It extends early RL-based NAS method [22] to optimizer search, by training a sequentialcontroller to produce the optimizer update rule according to a predefined pattern.
However, similar toits NAS counterpart, NOS-RL is also computationally expensive, requiring over 10k evaluations tofind good candidates.
Symbolic optimization and differential program synthesis
Symbolic optimization (SO) [1, 13,51, 58, 60, 61] aims at optimizing an objective over a symbolic hypothesis space of functions (or morebroadly, programs).
One line of work attempts to recover the unknown equation from its generateddata, with great potential in automating scientific discoveries [52, 61].
Another line of methodsaims at finding a more interpretable and generalizable symbolic model to replace the black-boxneural networks [51, 58, 60]; Applications that witnessed some success include learning symbolicpolicy networks for RL [60] and sequential classification models [51, 58].
The latter is often studiedunder the concept of Program Synthesis [23], where a model is extended to include programmaticrules such as if-else clause, indexing, e.t.c.
SO is closely connected to AutoML at a high level, asboth fields frame their problems as discrete optimization.
Indeed, many existing optimizer searchmethods can find their counterparts in symbolic optimization.
Our method is also inspired by therich body of literature in deep symbolic mathematics and program synthesis, which also explorestree-based expression spaces for differential equations and programs [29, 34, 51, 58, 61].
However,due to significant differences in taskonomy, SO and AutoML methods are often developed separately,converging into different branches of techniques.
Symbolic optimization often studies tasks wherecandidates are cheap to evaluate but finding the global optimal is desired [1, 52, 61]; As a result,sample efficiency is often not the primary concern.
Much to the opposite, in AutoML tasks, candidateevaluations are extremely expensive; Therefore, it is more beneficial to identify a good-enoughcandidate within a limited amount of budget.
Empirical evaluations on a diverse set of tasks
We extensively evaluate the proposed framework on a suite of tasks, covering a variety of modelsand datasets.
On standard benchmark tasks for optimizer search, our method is able to discoveroptimizers that outperform its human-designed and automatically searched counterparts.
In addition,we also show that the proposed framework enables automated optimizer design for many otherpopular learning tasks, such as adversarial attack, GNN training, and BERT finetuning.
Due to thespace limits, we will include detailed descriptions, search settings, and discovered optimizers foreach task in the Appendix.
4.1 General setting
MCT algorithm
Across all experiments, we limit the maximum level of MCT traversal to 4, andset the number of Monte Carlo samples to 32 (a multiple of 8 for parallelism on 8-GPU servers)for each level.
This amounts to a fixed total budget of 128 evaluations.
The maximum depth forthe super-tree is set to 10, which already covers many top-performing optimizers for various tasks.
We use a similar set of elementary operations as NOS-RL to build the optimizers, with only minoradjustments for some tasks (see Appendix for more details).
Optimizer evaluation We follow the default settings and hyperparameters for each task, and onlyswap out the optimizer; This potentially puts our algorithm at a disadvantage, as the hyperparametersare usually tuned around the default optimizers.
Before optimizer evaluation, we perform grid search on a small proxy task (fewer steps) to find a proper learning rate.
During the grid search, we alsoaggressively terminate optimizers if their performance falls under a certain threshold.
Since earlystopped optimizers consume fewer resources than a full evaluation, we do not count them into thebudget (number of evaluations). 4.2 Hand-written digit classification
[Figure]
We first compare our method with the LSTM-based optimizer (L2LGD2) on hand-written digitclassification.
Following L2LGD2 [10], the goal is to minimize the cumulative training loss of asingle-hidden-layer MLP with Sigmoid activation (MNISTNET) on the MNIST dataset; The searchis conducted on MNISTNET for 100 steps with a batch size of 128, and the discovered optimizersare subsequently transferred to three variants of MNISTNET with different activations (MNISTNETReLU), number of hidden layers (MNISTNET-2Layer), and dimensions (MNISTNET-Big).
Underthis setting, our method finishes in 0.92h on RTX 2080ti, much faster than L2LGD2 (2.62h).
As shown in Figure 3, our discovered optimizer achieves the lowest training loss under both directsearch and transfer settings.
Notable, the LSTM-based parametric update function indeed convergesfaster when the number of steps is close to the search phase (black-dotted vertical line on Figure3).
However, it extrapolates poorly to longer trajectories.
As the training proceeds, all other nonparametric optimizers eventually catch-up, achieving much lower training loss.
Moreover, LSTMbased optimizer also generalizes poorly to other model variants (most noticeably MNISTNET-ReLU),revealing its tendency to overfit the search task.
4.3 Image classification with ConvNet
We proceed to evaluate our method on the CIFAR-10 [8] classification task proposed in NOS-RL[15].
The model of choice is a 3-layer ConvNet.
Each layer of this network contains a 32-filter 3x3convolution with ReLU activation and batch normalization.
Following NOS-RL’s setting, for everyoptimizer, the best learning rate is searched over a grid of { 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 } with1 epoch of training, and the discovered learning rate is subsequently used to train the model for alonger period of time (5 epochs).
Since NOS-RL’s implementation is not open-sourced, we reproduceand compare with the two families of discovered optimizers described in NOS-RL paper: AddSignand PowSign.
The results are summarized in Table 1.
For NOS-RL, we display the performance of the top 4variants of PowSign and AddSign, which are obtained after training the controller for over 10kevaluations (Figure 4 in the NOS-RL paper [15]).
With only a fraction ( ∼ 1%) of the search budget,our framework is able to discover optimizers that reach a test accuracy of 77.02%, topping both
PowSign and AddSign optimizers and also human-designed ones by a sizable margin.
The sheerreduction in search cost and the improvement in search performance evince the efficiency andeffectiveness of the proposed framework for discovering better optimizers.
4.4 Adversarial attack
Next, we apply our framework to discover optimizers for constraint optimization.
We select adversarial attack, which aims at finding norm-bounded perturbations in the input space that alter themodel’s predictions.
The de facto optimizer used in adversarial attack is Projected Gradient Descent
[Table]
(PGD) [20].
We consider the most popular l ∞ -norm setting.
For l ∞ -norm bounded attack, PGDtakes the form of: x = Proj B ∞ ϵ ( x o ) ( x + γsign ( ∇ x L ( x )))) , where B ∞ ϵ ( x o ) represents a ϵ ballaround the original image x o w.r.t.
l ∞ -norm.
The models of choice come from the AutoAttacklibrary [45], which holds a leaderboard of top defense methods.
Following their settings, we setϵ = 8 / 255 , and run each optimizer once for 100 steps on every image from the test split [45].
[Table]
On this task, we mainly search for the updaterule inside the projection operator (e.g. sign ()for PGD).
The search is conducted on the pretrained Carmon2019 model [26], and the proposed optimizer is subsequently evaluated onother top defense methods for WideResNet [14](WRN-<depth>-<width>) and ResNet [11] (RN-<depth>).
As shown in Table 2, our discovered optimizer consistently outperforms PGDby a sizable margin.
Surprisingly, we found thatthe algorithm tends to pick log ( | · | ) rather thansign ( · ) as the first operator, resulting in manylog-based optimizers that surpass sign-based PGD.
In addition to PGD, we also compare our log-based optimizer with the best handcrafted and tunedoptimizer for adversarial attack: Adaptive PGD (APGD) [45]; The design of APGD is packed withdomain expertise: it combines a well-tuned momentum update rule with a conditional learning ratedecay based on a handcrafted schedule and sophisticated decay conditions (see Appendix for details).
However, the performance of our automatically discovered optimizer rivals APGD across variousdefense methods, despite of having a much simpler form (see Appendix for details).
This resultdemonstrates the potential of applying our framework to reduce the need of human expertise indesigning optimizers for diverse tasks.
4.5 Node classification on graphs
[Table]
We next test our framework for optimizing graphneural networks to classify nodes on graphs.
Themodel of interest is Graph Attention Network(GAT)
[21], one of the most widely used architectures in graph learning tasks.
We compare our method against Adam [9] - the standardoptimizer for optimizing GATs - on five commonly used graph datasets: OGBN-Product [47],
Cora [4], Citeseer [3], PubMed [6], and PPI [18].
Among them, OGBN-Product is the largest inscale, consisting of 2,449,029 nodes.
Since standard GATs cannot scale to this dataset, we instead adopt an adaptation of cluster-GCN [27]to GAT as the testbed, termed Cluster-GAT.
Cluster-GAT trains standard GAT on smallerpartitions of the original graph, thereby allow- ing the model to be applied to large-scale graphs.
We refer the reader to the Appendix for detaileddescriptions of all GAT implementations and experimental setups.
The results are summarized in Table 3.
On all datasets, our search algorithm is able to discoveroptimizers that outperform Adam.
An interesting observation is that the top-performing optimizersdiscovered for this task almost always contain sign ( · ) operators, revealing the potential of adoptingsign-based optimizers to improve the training of graph neural networks.
4.6 BERT fine-tuning on NLP datasets
[Table]
We also evaluate the proposed framework on BERTfinetuning task on GLUE benchmark [24].
For thistask, we follow all configurations of the HuggingFace [54] implementations: we finetune a pretrained
BERT (base cased) model for 3 epochs on Cola [42],
STS-B [16] and RTE [7] dataset, and 5 epochs on
MRPC [5] and WNLI [24] dataset.
The batch sizeis set to 32.
We compare our discovered optimizerswith the default AdamW [19].
As shown in Table 4,our automatically discovered optimizers outperform
AdamW on all datasets.
Ablation study
In this section, we ablate the proposed framework using the MNISTNET task.
All experiments arerepeated over 4 random seeds to account for randomness in the search phase.
[Table]
Random search baseline We study the effectiveness of
[Table]
our MCT algorithm alone by comparing it with random [Table] sampling.
Concretely, instead of traversing the tree basedon MC scores, we randomly generate all optimizers fromthe root.
Everything else in our framework remains unchanged, including our rejection sampling and equivalentform detection techniques.
This is equivalent to Random Search on our search space.
As shown in
Table 5, MCT algorithm outperforms Random Search baseline by a sizable margin, showing that
the
Monte Carlo node scoring schema can indeed guide the traversal towards promising branches of the [Table]
tree.
[Table]
Score thresholding As discussed in prior sections, score thresholding is important to the performance of the MCT algorithm.
To verify this, we ablate this technique by disabling it in our frameworkwhile keeping everything else the same.
Without score thresholding, the cumulative training loss ofthe proposed optimizers raises from 59.25 ± 2.50 to 60.25 ± 2.87, similar to that of random search.
Conclusion and limitations
Despite the recent advancement of practical AutoML systems in automatizing the design of architectures, data augmentation policies, and hyperparameters, progress in automated discovery of optimizersis still inadequate due to the limitations of prior methods in terms of 1).
efficiency, 2).
generability,and 3).
scalability.
In this paper, we introduce the first optimizer search framework that meets allthese criteria, allowing it to be directly applied to the tasks of interest.
The proposed frameworkdemonstrates promising results across a variety of tasks, from image classification, adversarial attack,to graph learning and BERT finetuning.
Our method by-no-mean solves the optimizer search problem,as there is plenty of room for improvement on the algorithm and search space; Rather, our goal is toopen up a new possibility for future development in non-parametric optimizer search methods.
Wehope the proposed framework could democratize research and applications of automated optimizersearch, and stimulate interest among researchers and practitioners.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
We arenot aware of any negative societal impacts of this work.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Due to space limits, some of these training details are included inthe Appendix(c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] The error bars for all optimizers are shown in the main
text.
The only exception is MNISTNET task, where we removed error bar for a cleanerplot.
We will display the error bar for this task in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [N/A] A.1 Pseudocode for our search algorithm Algorithm 1 and 2 summarize the complete search process.
rule
B Set of operators used for constructing the search space
Inspired by NOS-RL, we adopt the following set of mathematical operators in our experiments:
• Unary operators: − ( · ) , exp ( · ) , log ( | · | ) , (cid:112) | · | , clip 0 .
003 ( · ) , drop 0 .
1 ( · ) , sign ( · )• Binary operators: + , − , × , / , pow ( · , · )
[Table]
• Input Operators: g , g 2 , g 3 , m 1 , m 2 , m 3 , sign ( g ) , sign ( m 1 ) , Adam , RMSprop , 1 , 2 , ld ,cd , rd
Here, m 1 , m 2 , m 3 denote the first, second and third order momentum respectively, and ld , cd , rddenote linear decay, cosine decay, and restart decay [15]:
[Equation]
where t and T are the current and maximum step.
Following NOS-RL, we use n = 0 .
5 for cosinedecay and n = 20 for restart decay.
We set the bound for clip operator to 0 .
003 , and the dropoutratio to 0 .
1 for drop operator.
Note that one can always include more options of these values byadding new operator variants to the space (e.g. drop 0 .
3 () with dropout ratio set to 0 . 3 ).
For all inputoperators, we use their default PyTorch implementations and hyper-parameters.
The only exceptionis the learning rates for Adam and RMSprop.
We found that under the default learning rate, the normof Adam and RMSprop is sometimes quite small compared with other operands such as sign ( g ) ,making them potentially less effective as a submodule of some optimizers.
Therefore, we raise theirdefault learning rate by 3 × in our experiment.
Note that we make one minor adjustment to the set forthe ConvNet task: g 3 , m 3 , Adam , and RMSprop are removed as they rarely show up on the topsoptimizers.
Our set of operators is a subset of the full operator set presented in Section 4.1 of the NOS-RL paper.
However, note that NOS-RL also uses much smaller subsets rather than the full set to conduct thesearch.
We refer the readers to "Further discussions on NOS-RL baseline" in Appendix C.3 for moredetails.
C More details on experimental settings
C.1 General settings for our search algorithm
Search configurations For all experiment, we allow 4 levels of traversal and set the number of
Monte Carlo samples for each level to 32.
This amounts to a total budget of 128 evaluations.
Themaximum depth for the super-tree is set to 10.
The evaluation of Monte Carlo samples for each levelof traversal are completely independent, and therefore can be easily parallelized onto multiple GPUs.
As mentioned in the main text, we also apply score thresholding during the Monte Carlo estimation.
We use a universal threshold of 10 for losses, 20% for accuracy and correlations.
After the searchphase, top 5 optimizers are usually proposed for further evaluations.
Early stopping We also early stop poor optimizers to speedup the search process.
We use thefollowing standard procedure for deciding whether to terminate the training of an optimizer: If thesearch signal is training loss, we track if the moving average of the training loss keeps increasing forcertain amount of consecutive steps.
If the search signal is accuracy or correlations, we check if theaccuracy fails to reach the score threshold after 10% of training.
Constraints We use the following constraints during tree traversal: 1).
log ( exp ( · )) and − ( − ( · ))are prohibited.
sign ( · ) must not be followed by sign ( · ) , sign ( m 1 ) , sign ( g ) , clip 0 .
003 ( · ) , 1 , 2 ,ld , cd , and rd . 3).
(cid:112) | · | must not be followed by sign and 1 . 4).
clip 0 .
003 ( · ) must not be followedby clip 0 .
003 ( · ) , 1 , 2 , ld , cd , and rd .
C.2 Hand-digit classification with MNISTNET
Task setting
In this task, the goal is to minimize the cumulative training loss of a simple MLP(MNISTNET).
All experimental setups (including model variants) and the LSTM-based optimizer
[Table]
baseline are borrowed from the open-sourced PyTorch implementation of L2LGD2 2 The default
MNISTNET has one 20-dimensional hidden layers with Sigmoid activation.
In addition, we alsoconsider three other variants of MNISTNET: 1).
MNISTNET-2Layer, which doubles the number oflayers in MNISTNET; 2).
MNISTNET-Big, which doubles the hidden layer dimension of MNISTNET; 3).
MNISTNET-ReLU, which replaces the Sigmoid activation in MNISTNET with ReLU.
Allmodels are trained for 1000 steps with a batch size of 128 on the MNIST dataset.
We use a fixed50/50 split of training and testing set for MNIST.
Optimizer evaluation For each optimizer, the best learning rate is obtained from{ 0 . 0006 , 0 . 001 , 0 . 003 , 0 . 006 , 0 . 01 , 0 . 03 , 0 . 06 , 0 . 1 , 0 . 3 , 1 . 0 } .
During the grid search, we train thenetwork for 100 steps.
After that, the network is retrained for 1000 steps with the best learning rate.
We record the cumulative training loss and test accuracy of each optimizer for comparison.
Note thatfollowing the L2LGD2 implementation, the grid search for the LSTM-based optimizer is applied attraining time rather than test time.
Search setting Again, we follow the search settings implemented in the L2LGD2 codebase for ourexperiment.
The search is conducted on the default MNISTNET by training it for 100 steps on thetraining split.
Standard early stopping procedure is enabled for this task.
Empirically, we found thatroughly 7.2% optimizers are terminated.
Discovered optimizers
We represent some of the discovered optimizer down below.
Note that theforms of these optimizers are already simplified using the Sympy library
[Equation]
[Equation]
Interestingly, the pattern m 1 + RMSprop shows up quite frequently in the discovered optimizers forthis task.
More experimental results In addition to the convergence figures in the main text, we also presentthe results in tabular form.
Table 6 summarizes both the cumulative training loss and test accuracy ofthe optimizers on four MNISTNET models.
All models are trained for 1000 steps.
Our discoveredoptimizer achieves the best cumulative training loss and test accuracy for all cases.
C.3 Image classification with ConvNet
Task setting The goal of this task is to train a ConvNet on CIFAR-10 dataset.
Following NOS-RL,the ConvNet has two 32-filter 3x3 convolution layers, each followed by ReLU activation and batchnormalization [15] (Correction: We made a typo in the main text on the number of layers - it shouldbe 2 instead of 3).
We use a fixed held-out validation set of 5000 images for grid search.
Note that theheld-out validation set is used throughout the search phase, and it will be added back to the trainingset during final evaluation of the proposed optimizers.
Optimizer evaluation The grid search is performed over { 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 } for 1epoch of training, and the best learning rate is selected based on the accuracy on held-out validation learning-to-learn-by-gradient-descent-by-gradient- set.
After that, the optimizers are trained for a longer period of time (5 epochs).
The batch size is setto 100.
Search setting For this task, we disable early stopping (i.e. all optimizers will be counted intothe budget.), to establish fair comparisons with NOS-RL’s search budget; The reason is as follows:
Although NOS-RL also aggressively early stops poor optimizers, the authors added them back whenplotting Figure 4 in their paper;
And since our estimation of NOS-RL’s search budget comes from
Figure 4, it would be rational to also disable it in our experiment.
Each evaluation takes about 3minutes to finish.
And the duration for the entire search phase is around 7 hours on a single RTX2080ti.
Discovered optimizers Some of the discovered optimizers are shown below.
Note that the forms ofthese optimizers are already simplified using Sympy library.
conv1 : cd ∗ drop 0 .
1 ( g ) /ld
conv2 : cd ∗ sign ( m 1 ) ∗
[Equation]
[Equation]
/conv3 : drop 1 ( cd ∗ m 1 ) conv4 : m 1 ∗ ( rd + | g | ) ∗ exp ( cd )
Further discussions on NOS-RL baseline NOS-RL applies Reinforcement Learning to traina LSTM controller to generate optimizer update rules according to a predefined pattern.
Due tothe training difficulty, NOS-RL adopts a multi-config-multi-run search strategy: It conducts thesearch multiple times with different subset of operators (unknown) and different optimizer length(5,10,15,20).
Out of all search runs with different configurations, two families of best optimizersare reported in the paper.
This leads to several challenges that prevent us from obtaining exactcomparisons with NOS-RL: 1).
It is difficult to know or measure the exact search cost of NOS-RLdue to its multi-config-multi-run strategy.
The paper only mentions that a single search run can finishin one-day with heavily parallelism on Google’s infrastructures.
Therefore, the best we can do isto make an estimation based on Figure 4 in their paper, where it shows that the controller begins toconverge at least after 10k evaluations. 2).
The particular subsets of operators used during each searchrun is also unknown; The paper only mentions that the search spaces generated by these subsetstypically contains 10 6 to 10 11 update rules.
As a result, we have to pick our own operator set to runthe search on.
We conjecture that the main purpose of NOS-RL paper is to offer the discovered optimizer forpractitioners to use, rather than providing a baseline to stimulate further developments of nonparametric optimizer search methods.
This can be evidenced by its prohibitive search cost, and alsoby the fact that the source code is not released.
The nature of NOS-RL, combined with aforementionedchallenges, necessitate an open-sourced resource-friendly non-parametric optimizer search frameworkfor the community, which we hope to provide in this work.
C.4 Adversarial Attack
Task setting Adversarial attack aims at finding a norm-bounded perturbation to the input spacethat misleads the model predictions.
In this case, the parameter to be optimized is the data itself.
Weuse the AutoAttack library 3 to implement our experiments for this task.
The library contains a set ofdefense methods, as well as an implementation of the APGD optimizer that we used as the baseline.
The attack is conducted on the default test split of CIFAR-10 dataset, which contains 10000 images.
Our metric of choice is attack success rate.
Concretely, if the perturbed image successfully misleadthe model’s prediction into a wrong class, then the attack is successful for that image.
The successrate is thus the percentage of images that the optimizer successfully attacked.
Optimizer evaluation
The search is conducted on the Carmon2019 method.
We use{ 0 . 001 , 0 . 003 , 0 . 006 , 0 . 01 , 0 . 03 , 0 . 06 , 0 . 1 , 0 . 3 , 1 } to search for the best learning rate.
The grid searchis conducted on only 1000 test images.
After that, the optimizers will be evaluated by training for100 steps
Search setting During the search phase, all optimizers are evaluated with only 20 steps, as it isusually enough to identify top optimizers.
To further reduce the search cost, we use only 400 imagesfor grid search and 4800 (400 * 12) images for evaluation.
The search takes around one GPU day tofinish on a single RTX 2080ti.
Discovered optimizers We present some of the discovered optimizers for adversarial attack downbelow.
[Equation]
As mentioned in the main text, we found that many top optimizers are log-based.
More specifically,these optimizers often have the form of log ( | decay + sign ( · ) | ) .
The discovered log-based optimizersare also highly effective when transferred to other defense models, as showing in Table 2. Discussion on Adaptive Projected Gradient Descent (APGD) optimizer APGD is currently thestrongest manually design and tuned optimizer for adversarial attack. It consists of two parts: 1). amomentum update rule and 2).
a dynamic learning rate decay schema.
The momentum update ruletakes the following form:
[Equation]
where k is the current step and µ is the momentum ratio.
The µ is tuned to be 0.25, much lower thanthe default momentum ratio for standard optimizers such as SGD and PGD.
The dynamic learningrate decay schema halves the learning rate if a set of two conditions are satisfied at some predefinedsteps { w j } 100 j =1 :w j − 1 (cid:88) i =
w j −1 L ( x ( i +1) )
<
L ( x ( i ) )
< ρ ∗ ( w j − w j − 1 ) (3) γ ( w j ) ≡ γ ( w j − 1 ) and L ( w j ) min ≡ L ( w j − 1 ) min (4) where ρ is a threshold term and L denotes the loss function.
The steps ( w j ) to check for theseconditions are set to { 23 , 42 , 58 , 71 , 81 , 88 , 94 , 100 } .
As we can see, APGD has a quite complicatedform, and its design also packs a lot of human expertise.
On the other hand, our automaticallydiscovered optimizers are much simpler, while also rivaling the performance of APGD.
C.5 Node classification on graphs
Task Setting We consider node classification task on graphs.
The model of choice is Graph
Attention Network (GAT).
There exists many PyTorch implementations of GAT and its variants, eachcovers only some datasets.
As a result, we have to use more than one codebase for this experiment.
For training cluster-GAT on OGBN-Products dataset, we use the official implementation from OGBNlibrary 4 .
For training vanilla GAT on Cora dataset, we use pyGAT 5 .
For training vanilla GAT on
Citeseer, PubMed, and PPI dataset, we use the implementations from DGL library 6 .
We follow theinstructions provided in their README.md files to run all of our experiments.
The only except is for
Cora dataset, where we disable early stopping in the original implementation.
The reason is that thedefault criteria often terminate training prematurely for our optimizers.
Optimizer evaluation
The grid search is conducted over { 0 . 0003 , 0 . 0006 , 0 . 001 , 0 . 003 , 0 . 006 ,0 . 01 , 0 . 03 , 0 . 06 } , as we found that most of the optimizers’ best learning rates (including Adam)fall into this range.
After the grid search, all optimizers are evaluated under 4 random seeds.
Asmentioned above, all other hyperparameters, including the total number of epochs, batch size, weightdecay, e.t.c., are set to their default values as in the original codebases.
Search setting We deploy standard early stopping schema for this task.
The percentage of earlyterminated optimizers is around 10% to 17% for vanilla GATs.
We found that Cluster-GAT is muchharder to optimize: roughly 25% of the optimizers are early terminated.
The search is conducted oneach dataset separately, and we will discuss the transferability of the discovered optimizers in thenext paragraph.
For Products dataset, we parallelize the search over 8 RTX 2080ti GPUs.
All otherdatasets are ran on a single device.
The search takes about 2 GPU days to finish for Products dataset,1 GPU day for PPI and Cora, and 3 hours for PubMed and Citeseer.
Discovered optimizers We present some of the discovered optimizers on each dataset down below.
Interestingly, we found that sign-based optimizers dominate the graph learning task.products1 : ld ∗ sign ( m 1 ) − Adam products2 : ld ∗ ( sign ( m 1 ) − RMSprop ) cora : sign ( m 1 ) + m
[Table]
citeseer : drop
[Table]
1 (( sign ( [Table] [Table] m
[Table]
[Table] ) pubmed : sign [Table] + sign
[Table] drop [Table] [Table]
Our reproduced accuracy using ogbnleaderboard’s implementation is lower than the displayed number (
[Table] [Table] [Table] More experimental results on the transfersetting Among the optimizers listed above
,we found that "product2" optimizer exhibits thehighest level of transferability.
As shown in
Table 7, it outperforms Adam on all but theciteseer dataset.
Note that for graph learningtask, there exists a non-negligible performancegap between transfer and direct search settings.
We conjecture that it is because different graphdataset might indeed require different optimizers.
The reason is as follows:
Most graph neural networks, including GATs, adopt the message passing framework, where the features ofneighboring nodes are passed to the target nodethrough their edges in the forward pass.
Sincethe connectivity of nodes are defined by the adjacency matrix in the dataset, the computation graph (and thus the learning process) is inherentlyencoded in the dataset itself.
Moreover, some of the datasets and models we considered are inherentlyheterogeneous. For example, PPI is designed for inductive learning, whereas all other dataset arefor transductive setting; and also the Cluster-GAT model we used for OGBN-Products dataset isinherently different from vanilla GATs. C.6 BERT fine-tunning on NLP datasets Task setting We use Hugging Face’s official implementation of BERT finetuning task for ourexperiment 7 .
The goal of this task is to finetune a pretrained BERT (base cased) model on a set of
NLP datasets. Following the instructions on the official repo, we set the number of epochs to 5 for MRPC and WNLI, and 3 for CoLA, STS-B, and RTE dataset. We also observe that finetuning formore epochs generally harms the performance of all optimizers.
The model is trained with a batchsize of 32 on a single GPU.
We refer the reader to Hugging Face’s offical repo (link in the footnote)for more details on this task.
Optimizer evaluation We use { 2 e − 5 , 0 . 0001 , 0 . 0003 , 0 . 0006 , 0 . 001 , 0 . 003 , 0 . 006 , 0 . 01 } for learning rate grid search. Note that this grid is intentionally shifted to cover Hugging Face’s defaultlearning rate for AdamW ( 2 e − 5 ). After the grid search, the optimizers are trained for 4 random seedswith the best learning rate. Search setting Similar to GAT task, our search is conducted on each dataset separately, and wewill discuss the transferability of discovered optimizers later.
We early stop optimizers if theirperformance (accuracy, Matthew’s correlation, or Spearman’s correlation) fall below the defaultthreshold (0.2) during the grid search. Empirically, we found that roughly 13.5% optimizers areterminated. This rate is slightly higher than that of MNISTNET task (7.2%), because we are using thesame threshold as MNISTNET task even though the accuracies (or correlations) on BERT fine-tuningtasks are much lower. For this task, we parallelize the search over 8 RTX A6000 GPUs.
The searchcan be finished in less than 10 hours on all dataset. Discovered optimizers We present some of the discovered optimizers on each dataset down below. Similar to those found on the GAT tasks, many optimizers are sign-based.
Note that the poweroperator in cola2 optimizer might return NaN, which will be mapped to 0 by the sign function. cola1 : drop 0 . 1 ( Adam + g 3 ) cola2 : sign (
[Table]
clip
[Table]
clip [Table] [Table] drop
[Table]
[Table] [Table] + sign [Table]
[Table] )) stsb : sign ( [Table] + 2 /
[Table]
) rte : drop [Table] ( clip
[Table]
) wnli : sign ( m [Table] [Table] m
[Table] [Table] More experimental results on the transfer setting We found that both "cola2" and "rte" optimizer exhibits high level of transferability.
Although theyperform slightly worse than the optimizers directlysearched on the target dataset, it still consistently outperforms AdamW by a sizable margin. The resultsare summarized in Table 8. Note that some of ourreproduced results for AdamW is a bit different thanthe reported numbers on the Hugging Face repository. The reason is that we run each optimizer for 4 seedsand report the average results, whereas the officialrepository only records the number after a single run.
C.7 License
The Hugging Face libary we used is licensed under Apache License 2.0. All other public repositoriesare licensed under MIT License. D Reproducibility & ethics statementsReproducibility We have specified the setup for ours experiments in the main paper and Appendix,including settings for each task and hyperparameters for our method.
The code and the optimizersfound by our methods will also be published on Github upon acceptance to encourage futuredevelopment. Before then, a copy of our code is included in the supplementary material for reference. Ethics We are not aware of any potential ethical concerns regarding our work. E Limitations
Our view of this work is as a starting point of an efficient, scalable, and generalizable framework foroptimizer search. And we expect plenty of room for improvement for future works. For instance, weidentify the following concrete limitations of the method. We use precomputed momentum termsas input to our search space.
This is a practice we borrowed from NOS-RL. Adding commonly used terms ease the job of the search algorithm because it does not have to rediscover them from scratchevery time. However, searching for novel momentum update rules could potentially help to find evenstronger optimizers. In principle, our framework allows it: one can do this by inserting an operatorwith its own internal state.
This would serve as a potential direction for future work. 2). Identifyingproper hyperparameters for an optimizer is essentially for evaluation. In the current work, we usea simple grid search to discover the best learning rate for an optimizer. While it works fine for ourtasks, this could potentially be suboptimal as it might underestimate some optimizers.
Leveragingadvanced fast HPO during the search phase could be another direction to explore. 3). Although ourframework is 100x faster than the comparable method (NOS-RL), it still requires 128 evaluations inthe search phase. These evaluations can be largely parallelized. But potentially, the efficiency can beimproved further with better search algorithms, more train-free tests, knowledge transfer, e.t.c.