Paper: No learning rates needed: Introducing SaLSa - Stable Armijo Line Search Adaptation (ICLR-2024)

Reviewer comment (Evaluation):
I am a little bit suspicious that the step size drop in Figure 3 is caused by numerical issues. In fact, the sudden drop may happen across several iterations caused by mini-batch noises when the gradient norm becomes small. The authors may want to comment more on this.

Author response:
We are confident that some of the sudden drop stems from numerical issues since the gradient norm drops to true zero, which is not mathematically possible for a non-zero gradient, which we did not observe. We leave a more thorough analysis of this phenomenon for future work.

Reviewer comment (Experiments):
In a more recent work [3], the (stochastic) Armijo condition has been relaxed to allow the found step size not guarantee descent. Such non-monotone condition seems to fix the learning rate drop problem. The authors may want to consider adding this work to their baselines.

Author response:
This is definitely an interesting work. We will consider adding this work as another comparison point in future versions of this paper and added it to the related work.

Reviewer comment (Experiments):
**Fine-tuning tasks are too simple, not convincing.** Most NLP experiments are based on small fine-tuning datasets and small models. These tasks are simple for optimization and insensitive to LR choices. I am hoping to see more challenging tasks such as pretraining tasks for larger models on larger datasets, which rely much heavier on a good lr choice than the fine-tuning tasks shown in the script.

Author response:
A: To address the suitability of SalSa for pretraining tasks we present full pre-training on Image tasks in which our algorithm outperforms the existing baselines. Furthermore, we argue that fine tuning tasks are the most common use case for a practitioner, since pre-training is often infeasible due to resource limitations. Further, it can empirically be observed that these tasks are also sensitive to the learning rate, especially larger sized fine tuning tasks on datasets like MNLI.

Reviewer comment (Evaluation):
**Unsatisfactory performance.** The performance on imagenet tasks is unsatisfactory. Unclear performance on pretraining tasks on larger datasets and larger models.

Author response:
A: The original ResNet paper on ImageNet trained for 12 epochs instead of our 5, used additional data augmentation and performed 10 crop testing to obtain their final score. We omit additional data augmentation and 10 crop testing, since we are not interested in peak performance but rather the comparative performance between different optimizers. To be more comparable to the original ResNet paper we have increased our training time to 12 epochs, which resulted in the same relative results of the training schemes.

Reviewer comment (Experiments):
**More running time.** Backtracking methods require additional forward and backward passing of neural nets, which could be the major computational bottleneck when the model size grows. Though the authors claim they only require 3% additional running time over standard training, there is no real evidence to support this claim. I would suggest the authors report the wall-clock running time and compare it to that of standard training. Further, as the model size grows, the additional forward & backward passing would require much more running time than the claimed 3%.

Author response:
A: To provide further evidence for our estimated compute we added a wall-clock running time table in the appendix, see Figure 13. There we can see that especially for larger datasets (for example ImageNet, MNLI) 3% is a valid estimate. For smaller datasets (QNLI, SST2) initialization and other overhead results in a slightly higher percentage of extra compute.  
In the paper we explain that our line search takes on average 30% extra compute per step, since it needs on average about 1 extra forward pass, no extra backward pass is ever needed. We only have to perform our line search on average every 10 steps.

Reviewer comment (Writing):
In Eq.2: Is v a vector or a scaler?

Author response:
A: The “v” in Equation 3 is a vector.

Reviewer comment (Presentation):
Wrong template? The current script is using ICLR 2023 template, instead of ICLR 2024.

Author response:
A: Thanks for pointing this out, we changed it to the ICLR 2024 template.

Reviewer comment (Experiments):
While the presented results are very compelling, it would be even stronger to show experiments on Transformer models in a pretraining setting as well.

Author response:
Yes this would be great, we leave this to future work.

Reviewer comment (Presentation):
Fig 5: loss label missing at top of (a), accuracy label missing at top of (d).

Author response:
We incorporated all the mentioned suggestions for readability, with the exception of dashed/dotted lines. Thanks for pointing this out.

Reviewer comment (Presentation):
Fig 1: fix occluded label text for y axis.

Author response:
We incorporated all the mentioned suggestions for readability, with the exception of dashed/dotted lines. Thanks for pointing this out.

Reviewer comment (Experiments):
I'm a bit concerned by the quality of the baselines. The accuracies achieved on ImageNet with vanilla SGD seem abnormally low; more care should be used in making sure the baselines ("For the image tasks we compare to a flat learning rate" seems like an unfair comparison to the baseline since line search is able to adapt learning rates. It would be better to compare to the commonly used waterfall schedule or linear decay schedule).

Author response:
The original ResNet paper on ImageNet trained for 12 epochs instead of 5, used additional data augmentation and performed 10 crop testing to obtain their final score. We omit additional data augmentation and 10 crop testing, since we are not interested in peak performance but rather the comparative performance between different optimizers. For better comparison to the original ResNet paper we have increase training time to 12 epochs, resulting in better results - yet the relative performance of the optimizers did not change. 

With regards to more complex learning rate schedules: We showed a more complex learning rate schedule as a stronger baseline, specifically the cosine decay schedule for the NLP experiments. For the image datasets we used a flat baseline to allow comparability to the experiments from previous SLS papers [1,2].

Reviewer comment (Writing):
In general, need to be more careful in using \citep or \citet.

Author response:
Thanks for pointing this out we fixed it according to the ICLR 2024 guidelines

Reviewer comment (Experiments):
It would be helpful to show that the problem depicted in Figures 1 and 3 is alleviated by using SaLSa.

Author response:
We added SaLSa step sizes to Figures 1 and 3 to show how SaLSa does fix the problem.

Reviewer comment (Evaluation):
It's a bit strange to report the average or log average final loss as an aggregate metric, given how different the models and datasets are. Perhaps it would be more informative to report the relative ranks of each method.

Author response:
We added the relative ranks in Tables 4 and 5 in the appendix and added the average rank at the bottom of Table 1 and 2

Reviewer comment (Evaluation):
There are cases where SLS outperforms SaLSa -- it would be helpful to investigate further to understand why.

Author response:
This is definitely interesting, but we leave it for future work.