Efﬁcient Non-Parametric Optimizer Search for Diverse Tasks
Abstract
Efﬁcient and automated design of optimizers plays a crucial role in full-stack
AutoML systems.
However, prior methods in optimizer search are often limited by their scalability, generalization, or sample efﬁciency.
With the goal of democratizing research and application of optimizer search, we present the ﬁrst efﬁcient, scalable and generalizable framework that can directly search on the tasks of interest.
We ﬁrst observe that optimizer updates are fundamentally mathematical expressions applied to the gradient.
Inspired by the innate tree structure of the underlying math expressions, we re-arrange the space of optimizers into a super-tree, where each path encodes an optimizer.
This way, optimizer search can be naturally formulated as a path-ﬁnding problem, allowing a variety of well-established tree traversal methods to be used as the search algorithm.
We adopt an adaptation of the Monte Carlo method to tree search, equipped with rejection sampling and equivalent-form detection that leverage the characteristics of optimizer update rules to further boost the sample efﬁciency.
We provide a diverse set of tasks to benchmark our algorithm and demonstrate that, with only 128 evaluations, the proposed framework can discover optimizers that surpass both human-designed counterparts and prior optimizer search methods.
Our code is publicly available at https://github.com/ruocwang/enos .
Introductions
Motivated by a vision of democratizing machine learning, the central objective for automated machine learning (AutoML), such as automated architecture [22, 26, 29, 40, 55, 60, 62, 65] / optimizer [10, 12, 15, 17, 59, 68] / loss [36] / augmentation search [37, 39], lies in reducing the need for expert design on a diverse set of tasks.
To achieve this goal, it is critical for AutoML systems to exhibit a high level of efﬁciency, so that they can be directly applied to a variety of tasks without consuming a humongous amount of computing resources.
A widely successful example of such an effort is
DARTS [40] in Neural Architecture Search (NAS), which reduces the search cost from thousands of
GPU days of early RL-based algorithms to a single digit, enabling direct application of NAS systems to a wide range of tasks [32, 33, 38, 44, 50].
Inspired by the success of efﬁcient NAS methods, we turn our attention to another important but much less studied area of AutoML - Automated optimizer search, where an efﬁcient, scalable and generalizable framework is still absent.
Optimizer search aims to automatically design a suitable update function that takes gradients as inputs and produces update directions for the optimizee’s parameters.
Pioneering work in this area, coined Learning to Optimize (L2O) , adopts a data-driven approach by replacing human-designed update rules with a learnable parametric function [10, 12, 17, 59].
However, parametric optimizers are fundamentally not scalable to large models or datasets, as inferring its parameters typically requires expensive meta-learning steps such as backpropagating through gradient descent [10, 15, 68].
Moreover, the learned optimizer often generalizes poorly to even minor variants of its training task (Figure 3) [10, 68].
Poor scalability and generalization prevent L2O from being served as a general-purpose optimizer search framework that can be directly applied to tasks of interest .
The aforementioned limitations of parametric optimizers bring our attention to another line of method that searches over the discrete space of non-parametric update functions 1 , which generally exhibit the same level of scalability and generality as human-designed optimizers [15, 51].
NOS-RL [15] extends early RL-based NAS framework [22] to optimizer search, proposing to learn a sequential controller to produce optimizer update rules according to a predeﬁned pattern.
However, NOS-RL is sample inefﬁcient, requiring over 10k evaluations to ﬁnd good candidates.
More recently, AutoMLZero [51, 69] proposes to search over the vast space of computer codes for the entire ML pipeline (including the optimizer).
The excessive generality of its search space makes it even more costly to run than RL-based method.
The search cost of existing non-parametric optimizer search frameworks makes them computationally prohibitive not only for practitioners to apply but also for researchers to analyze.
With the goal of democratizing research and practical applications of automated optimizer design, we introduce the ﬁrst efﬁcient, scalable, and generalizable optimizer search framework that can be directly applied to a wide range of tasks.
We observe that non-parametric update rules are essentially mathematical expressions, with an innate tree structure where nodes are elementary math operators and edges represent their I/Os.
Consequently, generating an update rule can be viewed as progressively appending nodes to the expression tree until it is complete.
Inspired by this observation, we re-imagine the optimizer search space as a super-tree of mathematical expressions.
Each leaf node on the super-tree contains an optimizer, and the path towards it represents the generation process of that optimizer’s underlying expression.
With the tree-structured search space, optimizer search can be naturally formulated as a path-ﬁnding problem, allowing a wide range of well-established tree-traversal methods to be used as the search algorithm.
We show that a simple adaptation of
Monte Carlo Sampling [30, 53], equipped with our proposed rejection sampling and equivalent-form detection, can already produce remarkable results on our search space within a fraction of budgets compared with NOS-RL ( ⇠ 1%).
We extensively evaluate the proposed framework on a diverse set of learning tasks: digit classiﬁcation with MNISTNET [10], image classiﬁcation with ConvNet [15], graph learning with (Cluster-)GAT [21, 28], norm-bounded adversarial attack on robustly trained models [20, 45, 46], and BERT ﬁne-tuning on NLP datasets [34, 56].
These tasks cover both constraint and unconstrained optimizations and span over a large variety of models and datasets.
Despite the simplicity, the proposed framework is able to discover update rules that surpass human-designed optimizers and prior optimizer search methods, with a budget of only 128 evaluations.
We hope the proposed framework could lower the barrier of entry to practical non-parametric optimizer search, thereby providing an entry point for researchers and practitioners from ML community and beyond to study and utilize automated optimizer search systems.
Efﬁcient, scalable and generalizable framework for optimizer search 2.1 Optimizer design space
Notations and problem formulation Deep learning tasks are frequently expressed as optimizing a loss function L ( · ) deﬁned over parameter domain ✓ 2 ⇥ .
The minimizer of L can thus be obtained by ✓ ⇤ = arg min ✓ 2 ⇥ L ( ✓ ) .
For differentiable functions, a standard optimizer typically takes the form of iterative gradient descent: ✓ t +1 =
✓ t   ⇤  ( r ✓ L ( ✓ t )) , where t is the current iteration,  is the learning rate and  denote the update function.
Existing optimizers primarily differ in their design of update function  ; For example, vanilla gradient descent uses identity mapping  ( x ) =
x as the update function, whereas Adam adopts a momentum-based dynamic learning rate schema:  ( r ✓ L ( ✓ t ))
= m ( r ✓ L ( ✓ t )) /
p m (( r ✓ L ( ✓ t )) 2 ) , where m ( · ) denotes the momentum function with an internal state.
The goal of optimizer search is to automatically ﬁnd a suitable update function  over some hypothesis space  .
The hypothesis spaces used in prior work can be divided into two categories: non-parametric and parametric spaces.
Most human-designed optimizers belong to the ﬁrst category, where the update function  is not trainable.
Learnable optimizers, such as L2LGD2 [10] and SymbolicL2O [68], fall into the second category.
Our work mainly focuses on non-parametric optimizer search, with the goal of providing an efﬁcient, scalable and generalizable optimizer search framework that can be directly apply to various tasks.
Optimizer update rules as expression trees The ﬁrst step toward such a framework is to understand the structure of non-parametric optimizers.
We realize that, fundamentally, optimizers are mathematical expressions consisting of elementary operators ( + ,  , sign () , inputs , e.t.c.).
Math expressions have an inherent tree structure that preserves its order of execution, where nodes are operators and edges represent their I/Os.
For instance, Diagram 1 shows the expression tree of
Adam [9]:
[Equation]
[Figure]
where m 1 and m 2 denote the ﬁrst and second order momentum (which can also be broken down into their own expression trees).
Therefore, the generation of an update rule, as a mathematical expression, can also be conducted via top-down node selection: Take Adam as an example, we ﬁrst select division ( / ) as the root node.
For its left child, we pick m 1 , which is a leaf node and thus ends the branch.
For the right child, we select p , and subsequently pick m 2 to follow it.
At this point, there is no empty branches left, and we obtain the complete update rule for Adam.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] A tree-structured search space Inspired by the completion process of update rules, we rearrange all expressions into a super-tree , where each leaf node contains an update rule and each path represents its completion process.
The super-tree can be generated in a top-down manner:
Starting from the root node with an empty update rule, we generate each of its child nodes by inserting a different operator into the update rule, and repeat this process for the generated nodes.
Consequently, an optimizer can be sampled by traversing the super-tree until a leaf node is reached.
Since the super-tree can grow inﬁnitely deep, it is often desirable to restrict the tree to a predeﬁned depth N , where only the paths that can be completed within depth N are included.
Figure 1 provides an instantiation of our super-tree, where the paths leading to Adam and SGD optimizers are displayed as an example.
The beneﬁt of arranging the optimizer space into a tree is two folds.
Firstly, the tree-based search space is tight:
Proposition 1 Deﬁne the length of an update rule as the number of operators it includes, then the above tree-based search space is tight : a super-tree with a maximum depth of N covers all update rules of length no greater than N .
In a tight search space, all optimizers can be represented at the right level of complexity, allowing them to be visited by the search algorithm without exploring unnecessarily deep into the super-tree.
Although tightness is a fairly obvious result for our space, it is not the case for the previous search space deﬁned in NOS-RL, as we will explain later.
Secondly, with our super-tree, optimizer search can be naturally formulated as a path-ﬁnding problem, allowing a variety of well-established tree traversal methods to be deployed as search algorithms.
Contents To concretize the content of the search space, we allow three types of operators in the optimizer update rule:
• a set of p 1 unary operators (e.g. log ( | · | ) , exp ( · ) , p | · | , sign ( · ) , drop ( · ) ) • a set of p 2 binary operators (e.g. + ,  , ⇥ , / , pow ( · , · ) ) • a set of L leaf values (input operators) containing gradient-based terms (e g. g , m 1 ), decays (e.g. cosine _ decay ), and constants (e.g. 1 , 2 )
This categorization of mathematical operators is not new, as it is also adopted in symbolic math solver [35] and NOS-RL [15].
Comparison with NOS-RL’s search space Although both NOS-RL [15] and our framework use elementary math operators as building blocks for optimizers, they have little in common in terms of the arrangement of the search spaces.
Optimizers in NOS-RL’s search space are formed by a chain of predeﬁned motifs:
b ( u ( I ) , u ( I )) , where b , u , I denote binary, unary and input operators.
Due to the ﬁxed structure of such motifs, NOS-RL’s search space is not tight: there exist many optimizers that take extra longer sequences to express, potentially lowering their chance of being discovered by the search algorithm.
For instance, Diagram 2 shows an optimizer of length 5 , but it takes (10  1) nodes (two chained motifs) to represent it under NOS-RL’s arrangement; Moreover, NOS-RL’s search space also requires extra bypass operators (e.g. u ( x ) =
x and b ( x, y )
= x ) to cover even human-design optimizers such as Adam and PGD, further increasing the complexity.
In contrast, our representation of optimizers is directly inspired by the innate structure of its underlying mathematical expressions, resulting in a tight tree-based search space.
In our search space, optimizer search can be naturally formulated as a form of top-down path-ﬁnding problem.
In the next sections, we will detail our choice of algorithms for traversing the super-tree, as well as several techniques that leverage the characteristics of optimizer update rules to boost the sample efﬁciency.
2.2 Monte Carlo Sampling for tree traversal
We adopt a simple adaptation of Monte Carlo Sampling to tree traversal [30, 53, 70] (MCT) as the search algorithm.
The idea is to assign scores to the nodes in the super-tree (Figure 1), and use these scores to guide the tree traversal.
We deﬁne the score of a node v as a Monte Carlo estimation over unrolling steps from v : If v is an internal node, we randomly generate a set of unrolled paths from v to the corresponding leaf nodes, and take the average score of the resulting optimizers as the score for v ; If v is a leaf node, we set its score to 0 as it cannot be expanded.
The search can thus be conducted as follows: 1).
Starting from the root node v (0) at level 0 , we generate all child nodes { v (1) } of v (0) by inserting each operator from the candidate pool to the update rule in v (0) ; 2).
From there, we select the child node v ⇤ (1) with the highest MC score to expand, and move on to the next level; 3).
The process is repeated until a predeﬁned maximum search level is reached.
Algorithm 1 in the Appendix provides a detailed summary of the complete search process.
Directly applying the MCT algorithm to optimizer search would not perform well under limited search budgets, due to two unique characteristics of optimizer update rules that challenge the sample efﬁciency of the Monte Carlo estimates.
Firstly, the majority of mathematical expressions, when deployed as optimizer update rules, perform poorly or even would not converge.
This is usually not the case for other AutoML tasks such as neural architecture search, as most networks in the search space perform reasonably well.
The large body of poor-performing optimizers not only consumes precious search budget, but also causes the MC estimation to be unstable.
Secondly, there exists many mathematical redundancies in the expression space, for example: sign ( sign ( sign ( x ))) can be reduced to sign ( x ) , and m [Equation] is equivalent to m 1 p m {+
+} + 1 .
Identifying and eliminating these redundancies would not only save budget, but also prevent the sampling distribution from biasing toward mathematically simple and shallow update rules.
To address these issues and further boost the sample efﬁciency, we propose two sets of techniques - rejection sampling and equivalent-form detection.
When combined with these techniques, the simple MCT algorithm becomes particularly effective for the optimizer search task.
We will discuss them in detail in the following sections.
2.3 Rejection sampling
Eliminating poor optimizers with a train-free taskagnostic test Inspired by the characteristics of optimizer update rules, we develop a train-free taskagnostic test to eliminate poor optimizers without evaluating them.
We propose a necessary condition for a valid optimizer: it must produce an acute angle with steepest descent direction (i.e. gradients).
We check the validity of optimizers against this condition and only evaluate those that pass the test.
For the test to be task-agnostic, we feed the optimizer with a batch of random Gaussian vectors in place of actual gradients.
Formally, the descent test can be written as:
[Equation]
where  d is a predeﬁned threshold.
Although our descent test is by-no-mean comprehensive, it can effectively rule out a large chunk of poor optimizers with negligible false-negative rates, as demonstrated in Figure 2.
Reducing the variance of MC estimates via score thresholding After applying the descent test, there still remains a non-negligible portion of poor optimizers.
When sampled during the unrolling step, these optimizers would drastically lower the Monte Carlo score of the stem node, causing the
MC estimation to exhibit high variance and thus become unreliable.
This adverse effect is especially severe under the efﬁcient setting when the sample size is small.
Therefore, we propose to simply reject candidates with scores lower than a predeﬁned threshold, thereby removing them from the MC scores of the corresponding stem nodes.
2.4 Detecting and handling redundancies in mathematically equivalent forms
On-the-ﬂy constraint tree-traversal for redundant path pruning One beneﬁt of formulating the search problem as top-down path-ﬁnding is that we can easily apply constraints on-the-ﬂy to eliminate undesirable branches - those that lead to mathematically redundant expressions in our case.
We identify three main categories of such constraints:
• child operator that nulliﬁes its parent’s operator.
e.g.  (  x ) = x , ln ( e x ) = x • child operator that is redundant under its parent.
e.g. clip ( clip ( x ))
=
clip ( x ) • sequence of operators that reduces to a constant in the search space.
e.g. p | sign ( x ) | =
The complete sets of constraints we used can be found in the Appendix.
Enforcing these constraints during the traversal can effectively trim down the search tree, allowing the algorithm to explore branches that lead to more diverse and complex expressions.
Hashing mathematically equivalent expressions Besides enforcing constraints during the traversal, it is also important to detect mathematically equivalent optimizers to avoid duplicated evaluations.
One can always apply off-the-shelf symbolic solvers to identify the equivalence of two expressions,  and  0 , by checking if (    0 ) can be reduced to 0 .
However, it could become extremely slow as the pool of evaluated optimizers {  i } N 1 gets larger and larger, since we need to solve N pair of equations every time a new update rule is sampled.
Instead, we apply hashing to efﬁciently query the evaluated candidate pool for mathematically equivalent optimizers.
Concretely, we assign each optimizer a hash code, obtained by feeding a ﬁxed probing vector as input to the optimizer and recording its output.
The probing vector is pre-sampled from Gaussian distribution.
When a newly sampled optimizer arrives, we only need to compare its code with the hash table to check the existence of its equivalent form.
Empirically, it is much faster to run the proposed hashing-based checker than symbolic solvers.
Discussions and relationship to prior work
Automated optimizer design Optimization plays a crucial role in training deep learning models.
Generally, there does not exist one optimizer that aces all scenarios, as different tasks (dataset, architecture, loss, parameterization, e.t.c.) might favor different optimization methods [2, 10].
The demand for task-speciﬁc optimizers stimulates research interest in developing automated systems for optimizer design [10, 12, 15, 17, 41, 51, 59, 66, 68].
Early work adopts a data-driven method by modeling the optimizer update with a parametric function [10, 12, 59].
L2LGD2 [10] deploys an
LSTM model as the update function that takes historical gradients as input and produces the update direction.
However, parametric optimizer search methods are fundamentally limited by its scalability, as inferring its parameters requires expensive meta-learning steps such as back-propagation through optimization [10, 59, 68].
Although SymbolicL2O [68] improves the scalability of the learned
LSTM optimizer by distilling it into a lightly-parameterized symbolic optimizer, it still requires a pretrained LSTM model to begin with.
Instead of learning a parametric optimizer, NOS-RL [15] directly searches over a discrete space of non-parametric update functions comprised of mathematical operators.
It extends early RL-based NAS method [22] to optimizer search, by training a sequential controller to produce the optimizer update rule according to a predeﬁned pattern.
However, similar to its NAS counterpart, NOS-RL is also computationally expensive, requiring over 10k evaluations to ﬁnd good candidates.
Symbolic optimization and differential program synthesis
Symbolic optimization (SO) [1, 13, 53, 61, 63, 64] aims at optimizing an objective over a symbolic hypothesis space of functions (or more broadly, programs).
One line of work attempts to recover the unknown equation from its generated data, with great potential in automating scientiﬁc discoveries [54, 64].
Another line of methods aims at ﬁnding a more interpretable and generalizable symbolic model to replace the black-box neural networks [53, 61, 63]; Applications that witnessed some success include learning symbolic policy networks for RL [63] and sequential classiﬁcation models [53, 61].
The latter is often studied under the concept of Program Synthesis [24], where a model is extended to include programmatic rules such as if-else clause, indexing, e.t.c.
SO is closely connected to AutoML at a high level, as both ﬁelds frame their problems as discrete optimization.
Indeed, many existing optimizer search methods can ﬁnd their counterparts in symbolic optimization.
Our method is also inspired by the rich body of literature in deep symbolic mathematics and program synthesis, which also explores tree-based expression spaces for differential equations and programs [30, 35, 53, 61, 64].
However, due to signiﬁcant differences in taskonomy, SO and AutoML methods are often developed separately, converging into different branches of techniques.
Symbolic optimization often studies tasks where candidates are cheap to evaluate but ﬁnding the global optimal is desired [1, 54, 64]; As a result, sample efﬁciency is often not the primary concern.
Much to the opposite, in AutoML tasks, candidate evaluations are extremely expensive; Therefore, it is more beneﬁcial to identify a good-enough candidate within a limited amount of budget.
Empirical evaluations on a diverse set of tasks
We extensively evaluate the proposed framework on a suite of tasks, covering a variety of models and datasets.
On standard benchmark tasks for optimizer search, our method is able to discover optimizers that outperform its human-designed and automatically searched counterparts.
In addition, we also show that the proposed framework enables automated optimizer design for many other popular learning tasks, such as adversarial attack, GNN training, and BERT ﬁnetuning.
Due to the space limits, we will include detailed descriptions, search settings, and discovered optimizers for each task in the Appendix.
4.1 General setting
MCT algorithm
Across all experiments, we limit the maximum level of MCT traversal to 4, and set the number of Monte Carlo samples to 32 (a multiple of 8 for parallelism on 8-GPU servers) for each level.
This amounts to a ﬁxed total budget of 128 evaluations.
The maximum depth for the super-tree is set to 10, which already covers many top-performing optimizers for various tasks.
We use a similar set of elementary operations as NOS-RL to build the optimizers, with only minor adjustments for some tasks (see Appendix for more details).
Optimizer evaluation We follow the default settings and hyperparameters for each task, and only swap out the optimizer; This potentially puts our algorithm at a disadvantage, as the hyperparameters are usually tuned around the default optimizers.
Before optimizer evaluation, we perform grid search on a small proxy task (fewer steps) to ﬁnd a proper learning rate.
During the grid search, we also aggressively terminate optimizers if their performance falls under a certain threshold.
Since early stopped optimizers consume fewer resources than a full evaluation, we do not count them into the budget (number of evaluations).
4.2 Hand-written digit classiﬁcation
We ﬁrst compare our method with the LSTM-based optimizer (L2LGD2) on hand-written digit classiﬁcation.
Following L2LGD2 [10], the goal is to minimize the cumulative training loss of a single-hidden-layer MLP with Sigmoid activation (MNISTNET) on the MNIST dataset; The search is conducted on MNISTNET for 100 steps with a batch size of 128, and the discovered optimizers are subsequently transferred to three variants of MNISTNET with different activations (MNISTNETReLU), number of hidden layers (MNISTNET-2Layer), and dimensions (MNISTNET-Big).
Under this setting, our method ﬁnishes in 0.92h on RTX 2080ti, much faster than L2LGD2 (2.62h).
As shown in Figure 3, our discovered optimizer achieves the lowest training loss under both direct search and transfer settings.
Notable, the LSTM-based parametric update function indeed converges faster when the number of steps is close to the search phase (black-dotted vertical line on Figure 3).
However, it extrapolates poorly to longer trajectories.
As the training proceeds, all other nonparametric optimizers eventually catch-up, achieving much lower training loss.
Moreover, LSTMbased optimizer also generalizes poorly to other model variants (most noticeably MNISTNET-ReLU), revealing its tendency to overﬁt the search task.
4.3 Image classiﬁcation with ConvNet
We proceed to evaluate our method on the CIFAR-10 [8] classiﬁcation task proposed in NOS-RL [15].
The model of choice is a 2-layer ConvNet.
Each layer of this network contains a 32-ﬁlter 3x convolution with ReLU activation and batch normalization.
Following NOS-RL’s setting, for every optimizer, the best learning rate is searched over a grid of { 1 e  5 , 1 e  4 , 1 e  3 , 1 e  2 , 1 e  1 , 1 } with 1 epoch of training, and the discovered learning rate is subsequently used to train the model for a longer period of time (5 epochs).
Since NOS-RL’s implementation is not open-sourced, we reproduce and compare with the two families of discovered optimizers described in NOS-RL paper: AddSign and PowSign.
The results are summarized in Table 1.
For NOS-RL, we display the performance of the topvariants of PowSign and AddSign, which are obtained after training the controller for over 10k evaluations (Figure 4 in the NOS-RL paper [15]).
With only a fraction ( ⇠ 1%) of the search budget, our framework is able to discover optimizers that reach a test accuracy of 77.02%, topping both
PowSign and AddSign optimizers and also human-designed ones by a sizable margin.
The sheer reduction in search cost and the improvement in search performance evince the efﬁciency and effectiveness of the proposed framework for discovering better optimizers.
4.4 Adversarial attack
Next, we apply our framework to discover optimizers for constraint optimization.
We select adversarial attack, which aims at ﬁnding norm-bounded perturbations in the input space that alter the model’s predictions.
The de facto optimizer used in adversarial attack is Projected Gradient Descent
[Table]
(PGD) [20].
We consider the most popular l 1 -norm setting.
For l 1 -norm bounded attack, PGD takes the form of: x = Proj B 1 ✏ ( x o ) ( x +  sign ( r x L ( x )))) , where B 1 ✏ ( x o ) represents a ✏ ball around the original image x o w.r.t.
l 1 -norm.
The models of choice come from the AutoAttack library [46]
[Table]
which holds a leaderboard of top defense methods.
Following their settings, we set ✏ = 8 /
[Table]
and run each optimizer once for 100 steps on every image from the test split [46]
[Table] [Table]
On this task, we mainly search for the update rule inside the projection operator (e.g. sign () for PGD).
The search is conducted on the pretrained Carmon2019 model [27], and the proposed optimizer is subsequently evaluated on other top defense methods for WideResNet [14] (WRN-<depth>-<width>) and ResNet [11] (RN- <depth>).
As shown in Table 2, our discovered optimizer consistently outperforms PGD by a sizable margin.
Surprisingly, we found that the algorithm tends to pick log ( | · | ) rather than sign ( · ) as the ﬁrst operator, resulting in many log-based optimizers that surpass sign-based PGD.
In addition to PGD, we also compare our log-based optimizer with the best handcrafted and tuned optimizer for adversarial attack: Adaptive PGD (APGD) [46]; The design of APGD is packed with domain expertise: it combines a well-tuned momentum update rule with a conditional learning rate decay based on a handcrafted schedule and sophisticated decay conditions (see Appendix for details).
However, the performance of our automatically discovered optimizer rivals APGD across various defense methods, despite of having a much simpler form (see Appendix for details).
This result demonstrates the potential of applying our framework to reduce the need of human expertise in designing optimizers for diverse tasks.
4.5 Node classiﬁcation on graphs
[Table]
We next test our framework for optimizing graph neural networks to classify nodes on graphs.
The model of interest is Graph Attention Network (GAT) [21], one of the most widely used architectures in graph learning tasks.
We compare our method against Adam [9] - the standard optimizer for optimizing GATs - on ﬁve commonly used graph datasets: OGBN-Product [49],
Cora [4], Citeseer [3], PubMed [6]
[Table]
and PPI [18].
Among them
[Table]
OGBN-Product is the largest in scale, consisting of 2,449,029 nodes.
Since standard GATs cannot scale to this dataset, we instead adopt an adaptation of cluster-GCN [28] to GAT as the testbed, termed Cluster-GAT.
Cluster-GAT trains standard GAT on smaller partitions of the original graph, thereby allow- ing the model to be applied to large-scale graphs.
We refer the reader to the Appendix for detailed descriptions of all GAT implementations and experimental setups.
The results are summarized in Table 3.
On all datasets, our search algorithm is able to discover optimizers that outperform Adam.
An interesting observation is that the top-performing optimizers discovered for this task almost always contain sign ( · ) operators, revealing the potential of adopting sign-based optimizers to improve the training of graph neural networks.
4.6 BERT ﬁne-tuning on NLP datasets
[Table]
We also evaluate the proposed framework on BERT ﬁnetuning task on GLUE benchmark [25].
For this task, we follow all conﬁgurations of the HuggingFace [56] implementations: we ﬁnetune a pretrained
BERT (base cased) model for 3 epochs on Cola [43],
STS-B [16] and RTE [7] dataset
[Table]
and 5 epochs on
MRPC [5] and WNLI [25] dataset.
The batch size is set to 32.
We compare our discovered optimizers with the default AdamW [19].
As shown in Table 4, our automatically discovered optimizers outperform
AdamW on all datasets.
Ablation study
In this section, we ablate the proposed framework using the MNISTNET task.
All experiments are repeated over 4 random seeds to account for randomness in the search phase.
[Table]
Random search baseline We study the effectiveness of our MCT algorithm alone by comparing it with random sampling.
Concretely, instead of traversing the tree based on MC scores, we randomly generate all optimizers from the root.
Everything else in our framework remains unchanged, including our rejection sampling and equivalentform detection techniques.
This is equivalent to Random Search on our search space.
As shown in
Table 5, MCT algorithm outperforms Random Search baseline by a sizable margin, showing that
the
Monte Carlo node scoring schema can indeed guide the traversal towards promising branches of the tree.
Score thresholding As discussed in prior sections, score thresholding is important to the performance of the MCT algorithm.
To verify this, we ablate this technique by disabling it in our framework while keeping everything else the same.
Without score thresholding, the cumulative training loss of the proposed optimizers raises from 59.25 ± 2.50 to 60.25 ± 2.87, similar to that of random search.
Conclusion
Despite the recent advancement of practical AutoML systems in automatizing the design of architectures, data augmentation policies, and hyperparameters, progress in automated discovery of optimizers is still inadequate due to the limitations of prior methods in terms of 1).
efﬁciency, 2).
generalization, and 3).
scalability.
In this paper, we introduce the ﬁrst optimizer search framework that meets all these criteria, allowing it to be directly applied to the tasks of interest.
The proposed framework demonstrates promising results across a variety of tasks, from image classiﬁcation, adversarial attack, to graph learning and BERT ﬁnetuning.
Our method by-no-mean solves the optimizer search problem, as there is plenty of room for improvement on the algorithm and search space; Rather, our goal is to open up a new possibility for future development in non-parametric optimizer search methods.
We hope the proposed framework could democratize research and applications of automated optimizer search, and stimulate interest among researchers and practitioners.
Limitations
Our view of this work is as a starting point of an efﬁcient, scalable, and generalizable framework for optimizer search.
And we expect plenty of room for improvement for future works.
For instance, we identify the following concrete limitations of the method:
We use pre-computed momentum terms as input to our search space.
This is a practice borrowed from NOS-RL.
Adding commonly used terms ease the job of the search algorithm because it does not have to rediscover them from scratch every time.
However, searching for novel momentum update rules could potentially help to ﬁnd even stronger optimizers.
In principle, our framework allows it: one can do this by inserting an operator with its own internal state.
This would serve as a direction for future explorations.
Identifying proper hyperparameters for an optimizer is essentially for evaluation.
In the current work, we use a simple grid search to discover the best learning rate for an optimizer.
While it works ﬁne for our tasks, this could potentially be suboptimal as it might underestimate some optimizers.
Leveraging advanced fast HPO during the search phase could be another direction to explore.
Although our framework is 100x faster than the comparable method (NOS-RL), it still requires 128 evaluations in the search phase.
These evaluations can be largely parallelized.
But potentially, the efﬁciency can be improved further with better search algorithms, more train-free tests, knowledge transfer, e.t c.
Reproducibility & ethics statements
Reproducibility We have speciﬁed the setup for ours experiments in the main paper and Appendix, including settings for each task and hyperparameters for our method.
The code and the optimizers found by our methods will also be published on Github upon acceptance to encourage future development.
Before then, a copy of our code is included in the supplementary material for reference.
Ethics We are not aware of any potential ethical concerns regarding our work.
Acknowledgments and Disclosure of Funding
This work is supported in part by NSF under IIS-2008173, IIS-2048280, an Okawa research grant and a Google research scholar award.