Paper: Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICLR-2024)

Reviewer comment (Experiments):
Large-scale Adam experiments in Section 6.1 show Prodigy performing similarly to hand-tuned Adam, but this is unconvincing because Adam's global step-size has limited impact on performance. These experiments should include "un-tuned Adam" with default step-size, which might also perform similarly to Prodigy.

Author response:
The reviewer gave us the lowest possible score with the highest possible confidence, and made a few very strong claims in their review such as “The paper makes no new theoretical contributions”, “there is no meaningful improvement”, “it's unclear to me how interesting the global step-size of Adam even is”, etc. Moreover, as the reviewer pointed out themself, all of the claims in the review apply to the D-Adaptation paper, which received the outstanding paper award from ICML, but the reviewer would probably have given the same score to that paper too. Since the reviewer has a strong opinion on the prior literature that might be affecting this review more than the actual content of our work, we are worried that there is little chance for a fruitful discussion here. We have flagged the review to the ACs as a potential conflict of interest, and we refrain from commenting further.

Reviewer comment (Writing):
The paper is also marred by symbolic confusion and occasional grammatical errors, e.g., line 4 in Algorithm 2 mentions $\lambda_k = d_k^2$, but the fifth line of Sec. 2 mentions $\lambda_k = d_k$.

Author response:
Thanks for bringing the notation and grammar issues to our attention, we will change the notation to make it clearer, especially for  $\lambda_k$, and we'll do an extra pass to fix the typos.

Reviewer comment (Experiments):
There is an inherent question underlying weakness 1. It is mentioned below Theorem 1 regarding the careful setting of $\lambda_k$, "While it is not guaranteed to be better in theory, it is usually quite important to do so in practice." However, Sec 6 shows no great practical benefit from the apparent careful setting of $\lambda_k$. There is apparently some slight benefit in test losses for ViT for Resnet-50, but it is not clear how much tuning of $\lambda_k$ was done to achieve this benefit. The main promise of D-adaptation is that one does not need to perform much tuning of such optimization parameters.

Author response:
In all of our experiments, we simply set $\lambda_k=\beta_2^{-k/2}$, which is derived in Appendix B as the right way to get exponential moving averages of the gradient magnitudes. Therefore, Prodigy did not need any tuning. You can also see that Algorithm 4, which we used for the experiments, doesn't require $\lambda_k$ as input. The main reason we introduced $\lambda_k$ in the analysis was to 1) justify EMA and 2) make sure that $d_k$ is still estimated the right way. Furthermore, we didn't tune $\beta_2$ and used the value 0.999 for all methods.   
As the reviewer pointed out, the main benefit comes on vision tasks when training ViT on ImageNet and VGG/ResNet on CIFAR10, giving 1-2% improvement in test accuracy. We want to emphasize that this came at no extra cost and we didn't do any special tuning to achieve the improvement.

Reviewer comment (Theory):
Theorem 1 and 2 show that Prodigy improved on the learning rate of D-adaptation by a factor of $\sqrt{\log(D/d_0)}$. It is not clear if this improvement should be expected to be reflected in empirical problems since in practice we see convergence rates that are much faster than $1/\sqrt{t}$. If the rate does not reflect what happens empirically, getting better rates might be important for its novelty among other convergence rates. However, the related work section states that this new rate matches the one from online learning, which is a more adversarial setting. In the stochastic case, which more closely matches deep learning optimization, the work of Carmon and Hinder (2022) has a $\log \log (D)$ factor in the convergence rate for the deterministic version of the algorithm, which is better than Prodigy's convergence rates. The authors claim they improve on the DoG convergence rates by removing a $\sqrt{\log D/d_0}$ factor, but the DoG rates are for the stochastic setting with locally Lipschitz gradients, which is more general than the deterministic setting where the Prodigy rates hold.

Author response:
Thank you for your detailed review! We found your feedback to be very insightful and we we hope we can clarify some of the confusing parts:
1. Our main contribution is modifying D-Adaptation to adapt the learning rate faster. D-Adaptations at its core is a method that builds directly on top of Adam by estimating the learning rate, with the theory based on Adagrad. While the reviewer is right that the theoretical rate is slower than what we see in practice, D-Adaptation's main contribution is the theory-based estimate of the learning rate, and we are improving this specific aspect of the theory. The fact that D-Adaptation works well in practice suggests that this aspect of the theory is useful, so our $\sqrt{\log(D/d_0)}$ improvement might translate to a better practical performance. As we then validated in the paper using an extensive suite of benchmarks, there is indeed an improvement in some cases (ImageNet, CIFAR-10) and in other cases it works the same way as D-Adaptation and hand-tuned Adam.
2. Indeed, online learning and the stochastic setting of DoG are more general, which we will clarify in the updated version of the paper. We again highlight that our main point is improving upon D-Adaptation, which is of interest due to its practical performance.

Reviewer comment (Theory):
It is not clear what the relevance of the improved theoretical convergence rates is. While it is interesting that the improvement in empirical performance might be connected to the slightly tighter convergence rate, the paper does not discuss if they are actually connected. On the purely theoretical side, the rates match (or slightly improve in the case of DoG) other convergence rates, but only in a more restricted setting.

Author response:
We appreciate your candid feedback, and thank you for the advice!  
Your suggestion to frame the paper as a practical improvement to D-Adaptation makes sense from the perspective of Prodigy being useful in practice. Our perspective is that it's not the end yet as many components are not yet understood, including momentum, weight decay, and schedulers. We believe the way forward is to develop the theory further, which is why we emphasized it in the paper. For instance, we are working on estimating $\beta_1$ and $\beta_2$ adaptively, and it is all based on the theory rather than the empirical improvements. The main value of the empirical improvements, in our opinion, is to verify that the developed theory is a step in the right direction.  
Your perspective is nevertheless very helpful to us to see how the paper can be improved and we'll definitely take it into account.

Reviewer comment (Theory):
The lower bounds are interesting, but the fact that the function $f$ (and, thus, $D$) depend on $n$ in such a way that $\log \log D/d_0$ is roughly of order $\sqrt{n}$ is important and a big reason these lower bounds do not rule out the possibility of improved asymptotic convergence rates, even in the stochastic case. Although this is mentioned before the lower bounds, adding this to the theorem statements would improve understandability. Mentioning that the function is of the form $f(x) = |x - x^*|$ could also be interesting if space allows.

Author response:
Thank you for your detailed review! We found your feedback to be very insightful and we we hope we can clarify some of the confusing parts:
4. We agree with your comments on the lower bounds.

Reviewer comment (Theory):
Do the authors have an explanation of why/how the improved convergence rates should impact practical performance of the algorithm? In practice, convergence rates much faster than $O(1/\sqrt{n})$ are observed, and it is unclear why removing the $\sqrt{\log D / d_0}$ factor matters, especially since these rates are only proven for the deterministic setting while empirical performance is studied in the stochastic case.

Author response:
Thank you for your detailed review! We found your feedback to be very insightful and we we hope we can clarify some of the confusing parts:
1. Our main contribution is modifying D-Adaptation to adapt the learning rate faster. D-Adaptations at its core is a method that builds directly on top of Adam by estimating the learning rate, with the theory based on Adagrad. While the reviewer is right that the theoretical rate is slower than what we see in practice, D-Adaptation's main contribution is the theory-based estimate of the learning rate, and we are improving this specific aspect of the theory. The fact that D-Adaptation works well in practice suggests that this aspect of the theory is useful, so our $\sqrt{\log(D/d_0)}$ improvement might translate to a better practical performance. As we then validated in the paper using an extensive suite of benchmarks, there is indeed an improvement in some cases (ImageNet, CIFAR-10) and in other cases it works the same way as D-Adaptation and hand-tuned Adam.

Reviewer comment (Theory):
Could the authors expand on the relevance of the convergence rates given the comparisons with other works in more general settings? Since the paper emphasizes convergence rates, but these do not seem relevant compared to other works, clarification is needed. This currently feels like an interesting contribution for deep learning optimization, but much of the paper seems aimed at framing it as a theoretical contribution.

Author response:
We appreciate your candid feedback, and thank you for the advice!  
Your suggestion to frame the paper as a practical improvement to D-Adaptation makes sense from the perspective of Prodigy being useful in practice. Our perspective is that it's not the end yet as many components are not yet understood, including momentum, weight decay, and schedulers. We believe the way forward is to develop the theory further, which is why we emphasized it in the paper. For instance, we are working on estimating $\beta_1$ and $\beta_2$ adaptively, and it is all based on the theory rather than the empirical improvements. The main value of the empirical improvements, in our opinion, is to verify that the developed theory is a step in the right direction.  
Your perspective is nevertheless very helpful to us to see how the paper can be improved and we'll definitely take it into account.

Reviewer comment (Evaluation):
In terms of practical impact, the improvements made by Prodigy reported in Figure 1/2/3 are small. Often the differences between Adam, Prodigy, and D-Adaptation are much smaller than their standard errors. It doesn’t always do better either, sometimes being superseded by Adam, and sometimes by D-Adaptation.

Author response:
We believe that the improvements made by Prodigy on Imagenet and CIFAR-10 are significant enough (roughly 2% in test accuracy) to suggest the use of Prodigy over D-Adaptation. Moreover, in cases where the improvements are small, in particular when D-Adaptation already matches the hand-tuned Adam, there is no downside to using Prodigy.

Reviewer comment (Evaluation):
The authors start to make claims about the test accuracy / generalization that I don’t think they should or need to include. Specifically, they make this claim about CIFAR10 on ResNet and ImageNet on Vision Transformers. In both scenarios, the models are overfitting, trained for hundreds of epochs on the training data. Lower training loss here does not necessarily mean higher test accuracy.

Author response:
The claims we made about test accuracy are only coming from the observed values. Figures 1, 2, 3 all report the test accuracy/perplexity in the left column as the main metric. We did not intend to make any claims about the generalization properties of Prodigy beyond stating that we observe better values and that we observe larger learning rates, we will make it clear that we are not claiming a better implicit bias from the larger learning rate. We also highlight that we used weight decay and standard data augmentations, which is why the test accuracy keeps improving even after 100 epochs.

Reviewer comment (Evaluation):
Under Figure 1, they write “Prodigy estimates a larger step size than D-Adaptation, which helps it reach test accuracy closer to the one of Adam.” This statement is not about any implicit bias of large learning rates, but strictly that a large learning rate leads to better training loss convergence.

Author response:
The claims we made about test accuracy are only coming from the observed values. Figures 1, 2, 3 all report the test accuracy/perplexity in the left column as the main metric. We did not intend to make any claims about the generalization properties of Prodigy beyond stating that we observe better values and that we observe larger learning rates, we will make it clear that we are not claiming a better implicit bias from the larger learning rate. We also highlight that we used weight decay and standard data augmentations, which is why the test accuracy keeps improving even after 100 epochs.

Reviewer comment (Presentation):
Can Figure 2 be plotted in log scale? It’s hard to see the difference between the lines.

Author response:
The point of Figure 2 is mostly that there is no difference between the lines, it illustrates that we get the same performance without tuning the learning rate. In fact, Figure 2 looks almost the same way in log scale (the curves' shapes remain the same) since the values are all above 1 and do not go to 0.

Reviewer comment (Presentation):
There is a text spacing issue on Page 3.

Author response:
Thank you for pointing this out, we will fix the formatting issues on page 3.