Efﬁcient Non-Parametric Optimizer Search for Diverse Tasks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Efﬁcient and automated design of optimizers plays a crucial role in full-stack
AutoML systems.
However, prior methods in optimizer search are often limited bytheir scalability, generability, or sample efﬁciency.
With the goal of democratizingresearch and application of optimizer search, we present the ﬁrst efﬁcient, scalableand generalizable framework that can directly search on the tasks of interest.
Weﬁrst observe that optimizer updates are fundamentally mathematical expressionsapplied to the gradient.
Inspired by the innate tree structure of the underlying mathexpressions, we re-arrange the space of optimizers into a super-tree, where eachpath encodes an optimizer.
This way, optimizer search can be naturally formulatedas a path-ﬁnding problem, allowing a variety of well-established tree traversalmethods to be used as the search algorithm.
We adopt an adaptation of the Monte
Carlo method to tree search, equipped with rejection sampling and equivalentform detection that leverage the characteristics of optimizer update rules to furtherboost the sample efﬁciency.
We provide a diverse set of tasks to benchmark ouralgorithm and demonstrate that, with only 128 evaluations, the proposed frameworkcan discover optimizers that surpass both human-designed counterparts and prioroptimizer search methods.
Introductions
Motivated by a vision of democratizing machine learning, the central objective for automatedmachine learning (AutoML), such as automated architecture [22, 25, 28, 39, 53, 59, 62] / optimizer[10, 12, 15, 17, 57, 64] / loss [35] / augmentation search [36, 38] lies in reducing the need for expertdesign on a diverse set of tasks.
To achieve this goal, it is critical for AutoML systems to exhibit ahigh level of efﬁciency, so that they can be directly applied to a variety of tasks without consuminga humongous amount of computing resources.
A widely successful example of such an effort is
DARTS [39] in Neural Architecture Search (NAS), which reduces the search cost from thousands of
GPU days of early RL-based algorithms to a single digit, enabling direct application of NAS systemsto a wide range of tasks [31, 32, 37, 43, 48].
Inspired by the success of efﬁcient NAS methods, we turn our attention to another important butmuch less studied area of AutoML - Automated optimizer search, where an efﬁcient, scalableand generalizable framework is still absent.
Optimizer search aims to automatically designa suitable update function that takes gradients as inputs and produces update directions for theoptimizee’s parameters.
Pioneering work in this area, coined Learning to Optimize (L2O) , adoptsa data-driven approach by replacing human-designed update rules with a learnable parametricfunction [10, 12, 17, 57].
However, parametric optimizers are fundamentally not scalable to largemodels or datasets, as inferring its parameters typically requires expensive meta-learning steps suchas backpropagating through gradient descent [10, 15, 64].
Moreover, the learned optimizer oftengeneralizes poorly to even minor variants of its training task (Figure 3) [10, 64].
Poor scalability and generability prevent L2O from being served as a general-purpose optimizer search framework thatcan be directly applied to tasks of interest .
The aforementioned limitations of parametric optimizers bring our attention to another line of methodthat searches over the discrete space of non-parametric update functions 1 , which generally exhibitthe same level of scalability and generality as human-designed optimizers [15, 49].
NOS-RL [15]extends early RL-based NAS framework [22] to optimizer search, proposing to learn a sequentialcontroller to produce optimizer update rules according to a predeﬁned pattern.
However, NOS-RL issample inefﬁcient, requiring over 10k evaluations to ﬁnd good candidates.
More recently, AutoMLZero [49, 65] proposes to search over the vast space of computer codes for the entire ML
pipeline(including the optimizer).
The excessive generality of its search space makes it even more costly torun than RL-based method.
The search cost of existing non-parametric optimizer search frameworksmakes them computationally prohibitive not only for practitioners to apply but also for researchers toanalyze.
With the goal of democratizing research and practical applications of automated optimizer design,we introduce the ﬁrst efﬁcient, scalable, and generalizable optimizer search framework that canbe directly applied to a wide range of tasks.
We observe that non-parametric update rules areessentially mathematical expressions, with an innate tree structure where nodes are elementary mathoperators and edges represent their I/Os.
Consequently, generating an update rule can be viewed asprogressively appending nodes to the expression tree until it is complete.
Inspired by this observation,we re-imagine the optimizer search space as a super-tree of mathematical expressions.
Each leaf nodeon the super-tree contains an optimizer, and the path towards it represents the generation processof that optimizer’s underlying expression.
With the tree-structured search space, optimizer searchcan be naturally formulated as a path-ﬁnding problem, allowing a wide range of well-establishedtree-traversal methods to be used as the search algorithm.
We show that a simple adaptation of
Monte Carlo Sampling [29, 51], equipped with our proposed rejection sampling and equivalent-formdetection, can already produce remarkable results on our search space within a fraction of budgetscompared with NOS-RL ( ⇠ 1%).
We extensively evaluate the proposed framework on a diverse set of learning tasks: digit classiﬁcation with MNISTNET [10], image classiﬁcation with ConvNet [15], graph learning with(Cluster-)GAT [21, 27], norm-bounded adversarial attack on robustly trained models [20, 44, 45],and BERT ﬁne-tuning on NLP datasets [33, 54].
These tasks cover both constraint and unconstrainedoptimizations and span over a large variety of models and datasets.
Despite the simplicity, the proposed framework is able to discover update rules that surpass human-designed optimizers and prioroptimizer search methods, with a budget of only 128 evaluations.
We hope the proposed frameworkcould lower the barrier of entry to practical non-parametric optimizer search, thereby providing anentry point for researchers and practitioners from ML community and beyond to study and utilizeautomated optimizer search systems.
Efﬁcient, scalable and generalizable framework for optimizer search2.1 Optimizer design space
Notations and problem formulation Deep learning tasks are frequently expressed as optimizing aloss function L ( · ) deﬁned over parameter domain ✓ 2 ⇥ .
The minimizer of L can thus be obtainedby ✓ ⇤ = arg min ✓ 2 ⇥ L ( ✓ ) .
For differentiable functions, a standard optimizer typically takes theform of iterative gradient descent: ✓ t +1 =
✓ t   ⇤  ( r ✓ L ( ✓ t )) , where t is the current iteration, is the learning rate and  denote the update function.
Existing optimizers primarily differ in theirdesign of update function  ; For example, vanilla gradient descent uses identity mapping  ( x )
= xas the update function, whereas Adam adopts a momentum-based dynamic learning rate schema: ( r ✓ L ( ✓ t ))
= m ( r ✓ L ( ✓ t )) /
p m (( r ✓ L ( ✓ t )) 2 ) , where m ( · ) denotes the momentum function withan internal state.
The goal of optimizer search is to automatically ﬁnd a suitable update function  over some hypothesisspace  .
The hypothesis spaces used in prior work can be divided into two categories: non-parametricand parametric spaces.
Most human-designed optimizers belong to the ﬁrst category, where the update optimizers , which is a function  is not trainable.
Learnable optimizers, such as L2LGD2 [10] and SymbolicL2O [64], fallinto the second category.
Our work mainly focuses on non-parametric optimizer search, with the goalof providing an efﬁcient, scalable and generalizable optimizer search framework that can be directlyapply to various tasks.
Optimizer update rules as expression trees The ﬁrst step toward such a framework is to understand the structure of non-parametric optimizers.
We realize that, fundamentally, optimizers aremathematical expressions consisting of elementary operators ( + ,  , sign () , inputs , e.t.c.).
Mathexpressions have an inherent tree structure that preserves its order of execution, where nodes areoperators and edges represent their I/Os.
For instance, Diagram 1 shows the expression tree of
Adam [9]:
[Equation]
[Figure]
where m 1 and m 2 denote the ﬁrst and second order momentum (which can also be broken down intotheir own expression trees).
Therefore, the generation of an update rule, as a mathematical expression, can also be conducted viatop-down node selection: Take Adam as an example, we ﬁrst select division ( / ) as the root node.
Forits left child, we pick m 1 , which is a leaf node and thus ends the branch.
For the right child, we selectp , and subsequently pick m 2 to follow it.
At this point, there is no empty branches left, and weobtain the complete update rule for Adam.
[Figure]
SGD. A tree-structured search space Inspired bythe completion process of update rules, we rearrange all expressions into a super-tree , whereeach leaf node contains an update rule and eachpath represents its completion process.
Thesuper-tree can be generated in a top-down manner:
Starting from the root node with an emptyupdate rule, we generate each of its child nodesby inserting a different operator into the updaterule, and repeat this process for the generatednodes.
Consequently, an optimizer can be sampled by traversing the super-tree until a leaf nodeis reached.
Since the super-tree can grow inﬁnitely deep, it is often desirable to restrict the tree to a predeﬁned depth N , where only the paths thatcan be completed within depth N are included.
Figure 1 provides an instantiation of our super-tree,where the paths leading to Adam and SGD optimizers are displayed as an example.
The beneﬁt of arranging the optimizer space into a tree is two folds.
Firstly, the tree-based searchspace is tight:
Proposition 1 Deﬁne the length of an update rule as the number of operators it includes, then theabove tree-based search space is tight : a super-tree with a maximum depth of N covers all updaterules of length no greater than N .
In a tight search space, all optimizers can be represented at the right level of complexity, allowingthem to be visited by the search algorithm without exploring unnecessarily deep into the super-tree.
Although tightness is a fairly obvious result for our space, it is not the case for the previous searchspace deﬁned in NOS-RL, as we will explain later.
Secondly, with our super-tree, optimizer searchcan be naturally formulated as a path-ﬁnding problem, allowing a variety of well-established treetraversal methods to be deployed as search algorithms.
Contents and complexity analysis To concretize the content of the search space, we allow threetypes of operators in the optimizer update rule:
• a set of p 1 unary operators (e.g. log ( | · | ) , exp ( · ) , p | · | , sign ( · ) , drop ( · ) )
• a set of p 2 binary operators (e.g. + ,  , ⇥ , / , pow ( · , · ) )
• a set of L leaf values (input operators) containing gradients (e.g. g , m 1 ), decays (e.g.
cosine _ decay ), and constants (e
g. 1 , 2 )
This categorization of mathematical operators is not new, as it is also adopted in symbolic mathsolver [34] and NOS-RL [15].
Comparison with NOS-RL’s search space Although both NOS-RL [15] and our framework useelementary math operators as building blocks for optimizers, they have little in common in terms ofthe arrangement of the search spaces.
Optimizers in NOS-RL’s search space are formed by a chain ofpredeﬁned motifs:
b ( u ( I ) , u ( I )) , where b , u , I denote binary, unary and input operators.
Due to theﬁxed structure of such motifs, NOS-RL’s search space is not tight: there exist many optimizers thattake extra longer sequences to express, potentially lowering their chance of being discovered by thesearch algorithm.
For instance, Diagram 2 shows an optimizer of length 5 , but it takes (10  1) nodes(two chained motifs) to represent it under NOS-RL’s arrangement; Moreover, NOS-RL’s search spacealso requires extra bypass operators (e.g. u ( x ) =
x and b ( x, y )
= x ) to cover even human-designoptimizers such as Adam and PGD, further increasing the complexity.
In contrast, our representationof optimizers is directly inspired by the innate structure of its underlying mathematical expressions,resulting in a tight tree-based search space.
In our search space, optimizer search can be naturallyformulated as a form of top-down path-ﬁnding problem.
In the next sections, we will detail ourchoice of algorithms for traversing the super-tree, as well as several techniques that leverage thecharacteristics of optimizer update rules to boost the sample efﬁciency.
2.2 Monte Carlo Sampling for tree traversal
We adopt a simple adaptation of Monte Carlo Sampling to tree traversal [29, 51, 66] (MCT) as thesearch algorithm.
The idea is to assign scores to the nodes in the super-tree (Figure 1), and use thesescores to guide the tree traversal.
We deﬁne the score of a node v as a Monte Carlo estimation overunrolling steps from v : If v is an internal node, we randomly generate a set of unrolled paths from vto the corresponding leaf nodes, and take the average score of the resulting optimizers as the score forv ; If v is a leaf node, we set its score to 0 as it cannot be expanded.
The search can thus be conductedas follows: 1).
Starting from the root node v (0) at level 0 , we generate all child nodes { v (1) } of v (0)by inserting each operator from the candidate pool to the update rule in v (0) ; 2).
From there, we selectthe child node v ⇤ (1) with the highest MC score to expand, and move on to the next level; 3).
Theprocess is repeated until a predeﬁned maximum search level is reached.
Algorithm 1 in the Appendixprovides a detailed summary of the complete search process.
Directly applying the MCT algorithm to optimizer search would not perform well under limitedsearch budgets, due to two unique characteristics of optimizer update rules that challenge the sampleefﬁciency of the Monte Carlo estimates.
Firstly, the majority of mathematical expressions, whendeployed as optimizer update rules, perform poorly or even would not converge.
This is usually notthe case for other AutoML tasks such as neural architecture search, as most networks in the searchspace perform reasonably well.
The large body of poor-performing optimizers not only consumesprecious search budget, but also causes the MC estimation to be unstable.
Secondly, there existsmany mathematical redundancies in the expression space, for example: sign ( sign ( sign ( x )))
canbe reduced to sign ( x ) , and m 1 + p m 2 p mis equivalent to m 1 p m 2 + 1 .
Identifying and eliminating theseredundancies would not only save budget, but also prevent the sampling distribution from biasingtoward mathematically simple and shallow update rules.
To address these issues and further boostthe sample efﬁciency, we propose two sets of techniques - rejection sampling and equivalent-formdetection.
When combined with these techniques, the simple MCT algorithm becomes particularlyeffective for the optimizer search task.
We will discuss them in detail in the following sections.
2.3 Rejection sampling
Eliminating poor optimizers with a train-free taskagnostic test Inspired by the characteristics of optimizer update rules, we develop a train-free taskagnostic test to eliminate poor optimizers withoutevaluating them.
We propose a necessary conditionfor a valid optimizer: it must produce an acute anglewith steepest descent direction (i.e. gradients).
Wecheck the validity of optimizers against this conditionand only evaluate those that pass the test.
For thetest to be task-agnostic, we feed the optimizer with abatch of random Gaussian vectors in place of actualgradients.
Formally, the descent test can be writtenas:
[Equation]
where  d is a predeﬁned threshold.
Although our descent test is by-no-mean comprehensive, itcan effectively rule out a large chunk of poor optimizers with negligible false-negative rates, asdemonstrated in Figure 2.
Reducing the variance of MC estimates via score thresholding After applying the descent test,there still remains a non-negligible portion of poor optimizers.
When sampled during the unrollingstep, these optimizers would drastically lower the Monte Carlo score of the stem node, causing the
MC estimation to exhibit high variance and thus become unreliable.
This adverse effect is especiallysevere under the efﬁcient setting when the sample size is small.
Therefore, we propose to simplyreject candidates with scores lower than a predeﬁned threshold, thereby removing them from the MCscores of the corresponding stem nodes.
2.4 Detecting and handling redundancies in mathematically equivalent forms
On-the-ﬂy constraint tree-traversal for redundant path pruning One beneﬁt of formulatingthe search problem as top-down path-ﬁnding is that we can easily apply constraints on-the-ﬂy toeliminate undesirable branches - those that lead to mathematically redundant expressions in our case.
We identify three main categories of such constraints:
• child operator that nulliﬁes its parent’s operator.
e.g.  (  x ) = x , ln ( e x ) =
x• child operator that is redundant under its parent.
e.g. clip ( clip ( x ))
=
clip ( x )• sequence of operators that reduces to a constant in the search space.
e.g. p sign ( x ) | =[-
-]
The complete sets of constraints we used can be found in the Appendix.
Enforcing these constraintsduring the traversal can effectively trim down the search tree, allowing the algorithm to explorebranches that lead to more diverse and complex expressions.
Hashing mathematically equivalent expressions Besides enforcing constraints during the traversal, it is also important to detect mathematically equivalent optimizers to avoid duplicated evaluations.
One can always apply off-the-shelf symbolic solvers to identify the equivalence of two expressions, and  0 , by checking if (    0 ) can be reduced to 0 .
However, it could become extremely slow as thepool of evaluated optimizers {  i } N 1 gets larger and larger, since we need to solve N pair of equationsevery time a new update rule is sampled.
Instead, we apply hashing to efﬁciently query the evaluatedcandidate pool for mathematically equivalent optimizers.
Concretely, we assign each optimizer ahash code, obtained by feeding a ﬁxed probing vector as input to the optimizer and recording itsoutput.
When a newly sampled optimizer arrives, we only need to compare its code with the hashtable to check the existence of its equivalent form.
Empirically, it is much faster to run the proposedhashing-based checker than symbolic solvers.
Discussions and relationship to prior work
Automated optimizer design Optimization plays a crucial role in training deep learning models.
Generally, there does not exist one optimizer that aces all scenarios, as different tasks (dataset, architecture, loss, parameterization, e.t.c.) might favor different optimization methods [2, 10].
Thedemand for task-speciﬁc optimizers stimulates research interest in developing automated systemsfor optimizer design [10, 12, 15, 17, 40, 49, 57, 63, 64].
Early work adopts a data-driven method bymodeling the optimizer update with a parametric function [10, 12, 57].
L2LGD2 [10] deploys an
LSTM model as the update function that takes historical gradients as input and produces the updatedirection.
However, parametric optimizer search methods are fundamentally limited by its scalability,as inferring its parameters requires expensive meta-learning steps such as back-propagation throughoptimization [10, 57, 64].
Although SymbolicL2O [64] improves the scalability of the learned
LSTM optimizer by distilling it into a lightly-parameterized symbolic optimizer, it still requires apretrained LSTM model to begin with.
Instead of learning a parametric optimizer, NOS-RL [15]directly searches over a discrete space of non-parametric update functions comprised of mathematicaloperators.
It extends early RL-based NAS method [22] to optimizer search, by training a sequentialcontroller to produce the optimizer update rule according to a predeﬁned pattern.
However, similar toits NAS counterpart, NOS-RL is also computationally expensive, requiring over 10k evaluations toﬁnd good candidates.
Symbolic optimization and differential program synthesis
Symbolic optimization (SO) [1, 13,51, 58, 60, 61] aims at optimizing an objective over a symbolic hypothesis space of functions (or morebroadly, programs).
One line of work attempts to recover the unknown equation from its generateddata, with great potential in automating scientiﬁc discoveries [52, 61].
Another line of methodsaims at ﬁnding a more interpretable and generalizable symbolic model to replace the black-boxneural networks [51, 58, 60]; Applications that witnessed some success include learning symbolicpolicy networks for RL [60] and sequential classiﬁcation models [51, 58].
The latter is often studiedunder the concept of Program Synthesis [23], where a model is extended to include programmaticrules such as if-else clause, indexing, e.t.c.
SO is closely connected to AutoML at a high level, asboth ﬁelds frame their problems as discrete optimization.
Indeed, many existing optimizer searchmethods can ﬁnd their counterparts in symbolic optimization.
Our method is also inspired by therich body of literature in deep symbolic mathematics and program synthesis, which also explorestree-based expression spaces for differential equations and programs [29, 34, 51, 58, 61].
However,due to signiﬁcant differences in taskonomy, SO and AutoML methods are often developed separately,converging into different branches of techniques.
Symbolic optimization often studies tasks wherecandidates are cheap to evaluate but ﬁnding the global optimal is desired [1, 52, 61]; As a result,sample efﬁciency is often not the primary concern.
Much to the opposite, in AutoML tasks, candidateevaluations are extremely expensive; Therefore, it is more beneﬁcial to identify a good-enoughcandidate within a limited amount of budget.
Empirical evaluations on a diverse set of tasks
We extensively evaluate the proposed framework on a suite of tasks, covering a variety of modelsand datasets.
On standard benchmark tasks for optimizer search, our method is able to discoveroptimizers that outperform its human-designed and automatically searched counterparts.
In addition,we also show that the proposed framework enables automated optimizer design for many otherpopular learning tasks, such as adversarial attack, GNN training, and BERT ﬁnetuning.
Due to thespace limits, we will include detailed descriptions, search settings, and discovered optimizers foreach task in the Appendix.
4.1 General setting
MCT algorithm
Across all experiments, we limit the maximum level of MCT traversal to 4, andset the number of Monte Carlo samples to 32 (a multiple of 8 for parallelism on 8-GPU servers)for each level.
This amounts to a ﬁxed total budget of 128 evaluations.
The maximum depth forthe super-tree is set to 10, which already covers many top-performing optimizers for various tasks.
We use a similar set of elementary operations as NOS-RL to build the optimizers, with only minoradjustments for some tasks (see Appendix for more details).
Optimizer evaluation We follow the default settings and hyperparameters for each task, and onlyswap out the optimizer; This potentially puts our algorithm at a disadvantage, as the hyperparametersare usually tuned around the default optimizers.
Before optimizer evaluation, we perform grid search on a small proxy task (fewer steps) to ﬁnd a proper learning rate.
During the grid search, we alsoaggressively terminate optimizers if their performance falls under a certain threshold.
Since earlystopped optimizers consume fewer resources than a full evaluation, we do not count them into thebudget (number of evaluations).
[Figure]
We ﬁrst compare our method with the LSTM-based optimizer (L2LGD2) on hand-written digitclassiﬁcation.
Following L2LGD2 [10], the goal is to minimize the cumulative training loss of asingle-hidden-layer MLP with Sigmoid activation (MNISTNET) on the MNIST dataset; The searchis conducted on MNISTNET for 100 steps with a batch size of 128, and the discovered optimizersare subsequently transferred to three variants of MNISTNET with different activations (MNISTNETReLU), number of hidden layers (MNISTNET-2Layer), and dimensions (MNISTNET-Big).
Underthis setting, our method ﬁnishes in 0.92h on RTX 2080ti, much faster than L2LGD2 (2.62h).
As shown in Figure 3, our discovered optimizer achieves the lowest training loss under both directsearch and transfer settings.
Notable, the LSTM-based parametric update function indeed convergesfaster when the number of steps is close to the search phase (black-dotted vertical line on Figure3).
However, it extrapolates poorly to longer trajectories.
As the training proceeds, all other nonparametric optimizers eventually catch-up, achieving much lower training loss.
Moreover, LSTMbased optimizer also generalizes poorly to other model variants (most noticeably MNISTNET-ReLU),revealing its tendency to overﬁt the search task.
4.3 Image classiﬁcation with ConvNet
We proceed to evaluate our method on the CIFAR-10 [8] classiﬁcation task proposed in NOS-RL[15].
The model of choice is a 3-layer ConvNet.
Each layer of this network contains a 32-ﬁlter 3x3convolution with ReLU activation and batch normalization.
Following NOS-RL’s setting, for everyoptimizer, the best learning rate is searched over a grid of { 1 e  5 , 1 e  4 , 1 e  3 , 1 e  2 , 1 e  1 , 1 } with1 epoch of training, and the discovered learning rate is subsequently used to train the model for alonger period of time (5 epochs).
Since NOS-RL’s implementation is not open-sourced, we reproduceand compare with the two families of discovered optimizers described in NOS-RL paper: AddSignand PowSign.
The results are summarized in Table 1.
For NOS-RL, we display the performance of the top 4variants of PowSign and AddSign, which are obtained after training the controller for over 10kevaluations (Figure 4 in the NOS-RL paper [15]).
With only a fraction ( ⇠ 1%) of the search budget,our framework is able to discover optimizers that reach a test accuracy of 77.02%, topping both
PowSign and AddSign optimizers and also human-designed ones by a sizable margin.
The sheerreduction in search cost and the improvement in search performance evince the efﬁciency andeffectiveness of the proposed framework for discovering better optimizers.
4.4 Adversarial attack
Next, we apply our framework to discover optimizers for constraint optimization.
We select adversarial attack, which aims at ﬁnding norm-bounded perturbations in the input space that alter themodel’s predictions.
The de facto optimizer used in adversarial attack is Projected Gradient Descent
[Table]
(PGD) [20].
We consider the most popular l 1 -norm setting.
For l 1 -norm bounded attack, PGDtakes the form of: x = Proj B 1 ✏ ( x o ) ( x +  sign ( r x L ( x )))) , where B 1 ✏ ( x o ) represents a ✏ ballaround the original image x o w.r.t.
l 1 -norm.
The models of choice come from the AutoAttacklibrary [45], which holds a leaderboard of top defense methods.
Following their settings, we set✏ = 8 / 255 , and run each optimizer once for 100 steps on every image from the test split [45].
[Table]
On this task, we mainly search for the updaterule inside the projection operator (e.g. sign ()for PGD).
The search is conducted on the pretrained Carmon2019 model [26], and the proposed optimizer is subsequently evaluated onother top defense methods for WideResNet [14](WRN-<depth>-<width>) and ResNet [11] (RN-<depth>).
As shown in Table 2, our discovered optimizer consistently outperforms PGDby a sizable margin.
Surprisingly, we found thatthe algorithm tends to pick log ( | · | ) rather than 338 sign ( · ) as the ﬁrst operator, resulting in manylog-based optimizers that surpass sign-based PGD.
In addition to PGD, we also compare our log-based optimizer with the best handcrafted and tunedoptimizer for adversarial attack: Adaptive PGD (APGD) [45]; The design of APGD is packed withdomain expertise: it combines a well-tuned momentum update rule with a conditional learning ratedecay based on a handcrafted schedule and sophisticated decay conditions (see Appendix for details).
However, the performance of our automatically discovered optimizer rivals APGD across variousdefense methods, despite of having a much simpler form (see Appendix for details).
This resultdemonstrates the potential of applying our framework to reduce the need of human expertise indesigning optimizers for diverse tasks.
4.5 Node classiﬁcation on graphs
[Table]
We next test our framework for optimizing graphneural networks to classify nodes on graphs.
Themodel of interest is Graph Attention Network(GAT)
[21], one of the most widely used architectures in graph learning tasks.
We compare our method against Adam [9] - the standardoptimizer for optimizing GATs - on ﬁve commonly used graph datasets: OGBN-Product [47],
Cora [4], Citeseer [3], PubMed [6], and PPI [18].
Among them, OGBN-Product is the largest inscale, consisting of 2,449,029 nodes.
Since standard GATs cannot scale to this dataset, we instead adopt an adaptation of cluster-GCN [27]to GAT as the testbed, termed Cluster-GAT.
Cluster-GAT trains standard GAT on smallerpartitions of the original graph, thereby allow- ing the model to be applied to large-scale graphs.
We refer the reader to the Appendix for detaileddescriptions of all GAT implementations and experimental setups.
The results are summarized in Table 3.
On all datasets, our search algorithm is able to discoveroptimizers that outperform Adam.
An interesting observation is that the top-performing optimizersdiscovered for this task almost always contain sign ( · ) operators, revealing the potential of adoptingsign-based optimizers to improve the training of graph neural networks.
4.6 BERT ﬁne-tuning on NLP datasets
[Table]
We also evaluate the proposed framework on BERTﬁnetuning task on GLUE benchmark [24].
For thistask, we follow all conﬁgurations of the HuggingFace [54] implementations: we ﬁnetune a pretrained
BERT (base cased) model for 3 epochs on Cola [42],
STS-B [16] and RTE [7] dataset, and 5 epochs on
MRPC [5] and WNLI [24] dataset.
The batch sizeis set to 32.
We compare our discovered optimizerswith the default AdamW [19].
As shown in Table 4,our automatically discovered optimizers outperform
AdamW on all datasets.
Ablation study
In this section, we ablate the proposed framework using the MNISTNET task.
All experiments arerepeated over 4 random seeds to account for randomness in the search phase.
[Table]
Random search baseline We study the effectiveness of
[Table]
our MCT algorithm alone by comparing it with randomsampling.
Concretely, instead of traversing the tree basedon MC scores, we randomly generate all optimizers fromthe root.
Everything else in our framework remains unchanged, including our rejection sampling and equivalentform detection techniques.
This is equivalent to Random Search on our search space.
As shown in
Table 5, MCT algorithm outperforms Random Search baseline by a sizable margin, showing that
the
Monte Carlo node scoring schema can indeed guide the traversal towards promising branches of thetree.
Score thresholding As discussed in prior sections, score thresholding is important to the perfor-
[Table]
mance of the MCT algorithm.
To verify this, we ablate this technique by disabling it in our frameworkwhile keeping everything else the same.
Without score thresholding, the cumulative training loss ofthe proposed optimizers raises from 59.25 ± 2.50 to 60.25 ± 2.87, similar to that of random search.
Conclusion and limitations
Despite the recent advancement of practical AutoML systems in automatizing the design of architectures, data augmentation policies, and hyperparameters, progress in automated discovery of optimizersis still inadequate due to the limitations of prior methods in terms of 1).
efﬁciency, 2).
generability,and 3).
scalability.
In this paper, we introduce the ﬁrst optimizer search framework that meets allthese criteria, allowing it to be directly applied to the tasks of interest.
The proposed frameworkdemonstrates promising results across a variety of tasks, from image classiﬁcation, adversarial attack,to graph learning and BERT ﬁnetuning.
Our method by-no-mean solves the optimizer search problem,as there is plenty of room for improvement on the algorithm and search space; Rather, our goal is toopen up a new possibility for future development in non-parametric optimizer search methods.
Wehope the proposed framework could democratize research and applications of automated optimizersearch, and stimulate interest among researchers and practitioners. Reproducibility
Reproducibility We have speciﬁed the setup for ours experiments in the main paper and Appendix,including settings for each task and hyperparameters for our method.
The code and the optimizersfound by our methods will also be published on Github upon acceptance to encourage futuredevelopment.
Before then, a copy of our code is included in the supplementary material for reference.
Ethics We are not aware of any potential ethical concerns regarding our work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We are
not aware of any negative societal impacts of this work. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] Due to space limits, some of these training details are included in the Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] The error bars for all optimizers are shown in the main
text.
The only exception is MNISTNET task, where we removed error bar for a cleaner plot.
We will display the error bar for this task in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]