Paper: Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution (ICLR-2024)

Reviewer comment (Reproducibility):
In terms of proposing a new set of algorithm procedures (including new search space, search algorithm, integrity check) to find better optimizers, despite the authors doing a generally good job of documenting their procedures for reproducibility, the actual procedures are very specialized to the ConvNet model architecture (with multiple rounds of different types of heuristics to filter the search candidates) and might still be difficult to replicate and also generalize to a new task, architecture combination.

Author response:
It is common in AutoML tasks applied to neural networks to use a proxy function as training and evaluating thousands of large scale models is infeasible. Small ConvNets have been historically used as such a proxy function. Bello et al (2017) used a ConvNet when searching for their optimizers. [1] used a convnet when evolving activation functions. [2]  used a convnet when evolving loss functions. It is typical to take the best handful from the convnets and then train them large scale, as was performed by Bello et al (2017). The only difference between what we did and Bello et al (2017), was use an elimination protocol that selected the best handful by progressively upscaling the ConvNet. This was performed as our implementation of the optimizer graph was extremely memory intensive, as we had to keep track of atleast 4x the size of the model in memory ($\hat v, \hat s, \hat \lambda$ and momentum) (not counting internal state operands), which did not fit into memory when using EfficientNetV2Small given our computational resources. All final optimizers had to be hard-coded into memory efficient manners to allow for training upon EffNetV2Small. Therefore, we had to devise a way to select the most capable from the discovered optimizers other than hard-coding 150 optimizers.

Reviewer comment (Experiments):
In terms of the actual optimizers found and any insights obtained from analyzing them, it’s not clear how much better the discovered optimizers are compared to existing hand-designed optimizers like Adam. All of the results in Table 6 are evaluated using a single model architecture EffNetV2Small after performing some final optimizer candidate selection directly over this model architecture. Therefore, it’s unclear to what degree these optimizers can generalize to other models. Moreover, focusing on EffNetV2Small over the non-cifar tasks (tasks not used in the optimizer selection), there always exist hand-designed optimizers that are close to, if not better than, the discovered ones. Finally, in terms of interpreting the analytical form of the discovered optimizers, the paragraph at the end of page 8 and beginning of page 9 are more surface-level descriptions of experimental results rather than a distilled summary and insights, making it difficult to parse the key message the authors wish to convey.

Author response:
Please see our official comment (**Updated Unfinished Results**) as we have filled in the missing experiments. We would like to note that when training from scratch on CIFAR-10, CIFAR-100, and TinyImageNet, Opt6 greatly outperformed Adam. When fine-tuning, Opt3 greatly outperformed Adam. In addition, A1 and A5 outperformed Adam across all datasets (except CIFAR-10).  

Please see our official comment (Convergence, Sensitivity, and Theoretical Bounds for Found Deep Learning Optimizers) on the subject. There is not enough room in 9 pages to detail every insight of the final discovered optimizers. Please see Appendix G for in-depth discussion on the topic.  

This is semi-incorrect. Although it is true that the final optimizers were selected based upon their performance on EffNetV2Small, Table 6 does contain results on a different architecture: A ResNet9 (6M parameters) for TinyImageNet. In addition, Appendix H gives a supplementary experiment on training an LSTM (24M parameters) on the PTB dataset.

Reviewer comment (Experiments):
As one of the claimed contributions of the paper is an enlarged optimizer search with more operands, operators, and decay functions, it is expected that the authors should perform an ablation study to understand the benefit of the introduced enlarged search space. The bare-minimum ablation baseline is to train on the datasets in Table 6 using several of the optimizers discovered by Bello et al. (2017) and compare whether the performances of the optimizers discovered in this paper are better than those discovered in a smaller search space.

Author response:
Please see our official comment (**Updated Unfinished Results**) on the subject of unfinished results as we have included the best two discovered optimizers from Bello et al (2017) in our work.  

Unfortunately, we did not have the computational resources to run ablation studies on different combinations of the search space.

Reviewer comment (Experiments):
Part of the results in Table 6 are unfinished as the authors claim they will be finished before the rebuttal period.

Author response:
Please see our official comment (**Updated Unfinished Results**) on the subject of unfinished results.

Reviewer comment (Evaluation):
In Bello et al. (2017), in addition to tables comparing optimizers’ performance, learning curves of optimizers’ generalization performance progression are also shown. For some of the cases, it helps the reader see that Bello et al.'s discovered optimizers can converge faster than hand-designed methods for cases where the final performances might be similar. It is suggested that the authors also consider showing such graphs to provide additional information about the discovered optimizers.

Author response:
Unfortunately, we did not keep the training plots, but only their best test results. As a result, we do not have the computational resources to run every optimizer on each dataset again to provide this material by the end of the deadline.

Reviewer comment (Writing):
On page 4, the authors use the term “non-distributive function.” It is questioned whether the authors actually mean to say “non-homogeneous function.”

Author response:
We refer to a "non-distributed function" as any function such that $f(c*x) \ne c * f(x) $, an example would be  

$ \tan( c * x ) \ne c*\tan(x) \forall c \ge 0$.

Reviewer comment (Experiments):
Besides a large number of experiments, I think the main experiments are centered on or in variations of specific datasets like TinyImageNet and CIFAR-10.

Author response:
> 4. "The paper focuses on EfficientNet and ResNet, what about the performance in different architectures such as transformers or RNNs?"

Unfortunately, we did not have enough computational resources to test each optimizer across different deep learning domains. However, our submission does include results for training an LSTM on the PTB dataset, as recorded in Table 9.

Reviewer comment (Experiments):
One relevant weakness of the paper is that a relevant quantity of experiments were not completed at the time of the paper's writing, as denoted by asterisks in Table 6. The lack of results leaves a gap in the comprehensive evaluation of the results and makes it hard to evaluate the global performance of the proposed approach.

Author response:
Hello Reviewer,

Thank you for taking the time to detail your thoughts, concerns, and grading. We appreciate your criticism in making our work more professional, robust, and clear. Please see our official comment (**Updated Unfinished Results**) on the subject of unfinished results.

Reviewer comment (Experiments):
The authors focused experiments on image classification and language modeling tasks but argued that the results could be extended to other tasks.

Author response:
> 4. "The paper focuses on EfficientNet and ResNet, what about the performance in different architectures such as transformers or RNNs?"

Unfortunately, we did not have enough computational resources to test each optimizer across different deep learning domains. However, our submission does include results for training an LSTM on the PTB dataset, as recorded in Table 9.

Reviewer comment (Evaluation):
I was waiting for a more in-depth analysis of the results, considering the whys, the sensitivity analysis, tendencies, etc. The paper shows the phenomena exist but does not explain them in detail. I know this is common in papers in this field, but we need to improve that. It has an interesting empirical contribution but lacks theoretical support for why certain optimizers perform better than others.

Author response:
> 1. "I was waiting for a more in-depth analysis of the results, considering the whys, the sensitivity analysis, tendencies, etc." 

Please see our official comment (**Convergence, Sensitivity, and Theoretical Bounds for Found Deep Learning Optimizers**)  on the subject.

Reviewer comment (Evaluation):
The fine-tuning results appear to be inconsistent with the from-scratch training results.

Author response:
> 2. "The fine-tuning results appear to be inconsistent with the from-scratch training results."

Yes, this observation does occur as the performance of the optimizers flip when training from scratch and fine-tuning. This also occurs for the baseline deep learning optimizers, such as SGD and QHM. In addition, top performing optimizers in each scenario do not flop in the other. For example, Op6, the best performer when training from scratch, performs comparably to Adam on Flowers and Caltech; and, Opt3, arguably the best performer when fine-tuning, outperformed Adam on CIFAR100 and Tiny. Therefore, we do not see this as a weakness, but an exposition of the strengths and weaknesses for different optimizers.

Reviewer comment (Experiments):
The paper identifies families of optimizers, but it doesn't explore the unique characteristics of each family and how they contribute to the model's performance.

Author response:
> 3. "The paper identifies families of optimizers, but it doesn't explore the unique characteristics of each family and how they contribute to the model's performance."

Please see our official comment (**Convergence, Sensitivity, and Theoretical Bounds for Found Deep Learning Optimizers**)  on the subject. In addition, Appendix F discusses the effects of the final decay functions, learning rate schedules, and weight update equations.