Paper: Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization (ICLR-2024)

Reviewer comment (Experiments):
To my understand $c$ in Algorithm 1 is a hyperparameter. If so, does it mean that this method introduces a parameter to eliminate learning rate? How sensitive the proposed method to this parameter?

Author response:
We apologize for the confusion. Below is our response to weakness 1:

Although Alg. 1 introduces a hyperparameter, $c$, it demonstrates robustness to this hyperparameter. Specifically, we consistently employed $c=0.5$ across all experiments, including supervised classification, self-supervised learning, fine-tuning, and reinforcement learning.

Reviewer comment (Theory):
Algorithm 1 also requires $f^*$, which I think is the loss value at the optimum. For deep models, obtaining such a value sounds quite challenging.

Author response:
Yes. $f^*$ represents the loss value at the optimum, and achieving this value can be challenging in deep learning applications. However, we can simply use the theoretically minimum value of the loss function as $f^*$. For instance, we set $f^*=0$ for cross-entropy losses and the SimCLR loss. On the other hand, since we cannot determine the minimum loss values for reinforcement learning models, we have omitted the experiments in Table 3. (SPS and PS-SPS in reinforcement learning tasks)

Reviewer comment (Experiments):
How to tune $\gamma_k$?

Author response:
$\gamma_k$ is an optional hyperparameter for learning rate annealing. We can simply employ the same learning rate annealing schedule used in previous methods, such as a step decay or a cosine annealing. It is worth noting that the annealing should have a base learning rate of $1$. For example, a step decay should start from $1$ and decay to $0.1$ and $0.01$.

We would like to emphasize that SPS amd PS-SPS (Alg. 1) do not employ an additional annealing strategy ($\gamma_k$), apart from D-Adaptation-based methods like D-Adapt SGD, D-Adapt Adam, Prodigy, and PS-DA-SGD (Alg. 2), which include $\gamma_k$.

To ensue a fair comparison, we applied the same annealing schedule used for training baseline models. Specifically, we used a step decay in CIFAR-100 experiments, a constant learning rate in reinforcement learning models, and a cosine annealing in ImageNet, fine-tuning, and SSL. We will provide further clarification on this point in the Appendix.

It is true that a careful annealing strategy enables an optimizer to converge with arbitrary learning rates. For instance, in Fig. 1 (a), Prodigy exhibits a sudden increase in the learning rate, which can be handled by a rapidly decreasing schedule. However, we did not further tuned the schedule and utilized the ones provided by baseline codes.

Reviewer comment (Theory):
While the approach is intuitive and a convergence proof is given, the approach exhibits heuristic qualities and doesn't discuss the resulting dynamic of the adaptation.

Author response:
W1) We apologize for the confusion. Our proposed method estimates one step-size for one network. 

For example, when employing Adam to train a network, there exists one step-size hyperparameter, and our method is designed to estimate this value.

The parameter-wise scaling of the proposed method corresponds to the gradient scaling of Adam, where the gradient of each parameter is divided by the square root of the moving average of the second moment of the gradient. Similarly, the proposed method scales each parameter by the fourth root of the moving average of the second moment of the gradient. The effects of these scaling are equivalent as shown in Eqs. 1-4.

Reviewer comment (Theory):
Especially, in consideration of the potential complex resulting dynamics by applying parameter wise step-size adaptation.

Author response:
W1) We apologize for the confusion. Our proposed method estimates one step-size for one network. 

For example, when employing Adam to train a network, there exists one step-size hyperparameter, and our method is designed to estimate this value.

The parameter-wise scaling of the proposed method corresponds to the gradient scaling of Adam, where the gradient of each parameter is divided by the square root of the moving average of the second moment of the gradient. Similarly, the proposed method scales each parameter by the fourth root of the moving average of the second moment of the gradient. The effects of these scaling are equivalent as shown in Eqs. 1-4.

Reviewer comment (Experiments):
Is there something that can be said about label noise sensitivity/robustness of the proposed method?

Author response:
Q1) Thank you for the insightful suggestion. To measure the label noise sensitivity of the proposed method, we conducted an additional experiment using the ResNet-32 network on the CIFAR-100 dataset. To introduce label noise, we randomly shuffled some labels in the training dataset. The Adam optimizer with a learning rate of $0.003$ was chosen as the baseline, which is the optimal learning rate value found by grid-search on the dataset without label noise.

| Label noise | 0.0   | 0.2   | 0.4   | 0.6   | 0.8   |
|-------------|-------|-------|-------|-------|-------|
| Adam        | 66.31 | 50.95 | 39.75 | 25.10 | 9.65  |
| PS-DA-SGD   | 64.49 | 52.57 | 41.34 | 25.26 | 10.41 |

The proposed method demonstrates robustness to the label noise by surpassing the baseline when the label noise is injected. It is due to the optimal learning rate depends on the level of label noise. We used a learning rate of $0.003$ for Adam across all label noise levels. While this value was the optimal value in the absence of label noise, it was suboptimal when the level of label noise changed. Contrarily, the proposed method found the optimal learning rate regardless of the label noise level.

Reviewer comment (Novelty):
The contribution of the paper is unclear. In the first sentence in the abstract, the authors claim that they "address the challenge of estimating the learning rate for adaptive gradient methods." The issue is important, but after reading the paper, it is not clear how they addressed the issue.

Author response:
In our paper, we present two algorithms to estimate the learning rate of adaptive gradient methods. In Alg. 1, line 6 represents the update rule of an optimizer, with the learning rate $\eta_k$ estimated in line 5. Similarly, in Alg. 2, line 11 represents the update rule of an optimizer, with the learning rate $\eta_k$ estimated in line 10.

Reviewer comment (Writing):
The authors claim they "propose learning rate-free adaptive gradient methods." It appears that Algorithm 2 is the method they propose. However, in Algorithm 2, there are many hyper-parameters, including $\eta_k$, $\gamma_k$, and even $\alpha_k$. It is not clear why Algorithm 2 is "learning rate-free." The explanation about the notations in Algorithm 2 should be clearer.

Author response:
Training a deep neural network with an adaptive gradient optimizer involves various design choices:
1. Selecting a gradient rescaling strategy, such as AdaGrad, Adam, or AMSGrad.
2. Choosing a base learning rate within the range of $0.0003$ to $0.01$ or broader.
3. Selecting a learning rate annealing schedule, such as constant, step decay, or cosine annealing.

In this context, Alg. 1 relieves the second and third design choices, while Alg. 2 addresses the second design choice. These algorithms are termed learning-rate-free methods as they alleviate the manual tuning of the base learning rate.

The $\alpha_k$ term in Alg. 1 and Alg. 2 corresponds to the first design choice. For example, if we choose Adam as the optimizer, then $\alpha_k$ will be $v_k^{1/4}$ (see Table 1 and Eq. 4). The $\gamma_k$ of Alg. 2 corresponds to the third design choice, representing the learning rate annealing schedule. For example, if we choose a step decay schedule, then $\gamma_k$ will start from $1$ and decay by a factor of $10$.

The remaining parameters, $c$ of Alg. 1 and $d_0$ of Alg. 2, are hyperparameters. However, the algorithms are robust to their values. We used $c=0.5$ and $d_0=10^{-6}$ across all experiments, including supervised classification, self-supervised learning, fine-tuning, and reinforcement learning.

Reviewer comment (Evaluation):
In Section 5.2, authors should report the metrics for their reinforcement learning experiment. It is hard to understand the value of the proposed PS-SPS and PS-DA-SGD from Table 3.

Author response:
In our experiments evaluating reinforcement learning, we utilized the average of the reward output of the OpenAI Gym environment as the performance metric. The values in parentheses represent the training success rate. It is important to note that SPS and PS-SPS were omitted from these experiments due to their reliance on the loss value at optimum, denoted as $f^*$, which is challenging to estimate before training within the context of reinforcement learning models.

We have removed all batch normalization layers from VGG networks, while ResNet networks remain unchanged. This modification aims to assess the impact of batch normalization on learning-rate-free algorithms. Our rationale for investigating this impact of batch normalization stems from the observation that many learning-rate-free methods exhibit subpar performance in reinforcement learning experiments, where batch normalization is not employed.

Reviewer comment (Experiments):
The authors mention that they removed all the batch normalization layers in the CIFAR-100 experiment. The benefits of such removal are not explained.

Author response:
We have removed all batch normalization layers from VGG networks, while ResNet networks remain unchanged. This modification aims to assess the impact of batch normalization on learning-rate-free algorithms. Our rationale for investigating this impact of batch normalization stems from the observation that many learning-rate-free methods exhibit subpar performance in reinforcement learning experiments, where batch normalization is not employed.

Reviewer comment (Theory):
Authors should clearly write their assumptions and conclusions into a formal theorem in Section 4.2.

Author response:
A formal theorem of Sec. 4.2 is as follows.

AMGRrad scaling rule:

$\alpha_k=(\max_{i\le k}v_i)^{1/4}$, where $v_{i+1}=\beta v_i + (1 - \beta) g^2_i$, $0<\beta<1$.

For a convex G-Lipschitz function $f$ and AMSGrad scaling rule, Alg. 1 converges to the minimum of $f$.

For a convex G-Lipschitz function $f$, AMSGrad scaling rule, and a decreasing sequence $\gamma_k$ satisfying $\sum^\infty_{k=1}\gamma_k=\infty$ and $\sum^\infty_{k=1}\gamma_k^2<\infty$, Alg. 2 converges to the minimum of $f$.

Reviewer comment (Theory):
The equations are also not clear. The authors claim that the adaptive methods can be viewed as applying the steepest descent to parameter-scaled networks based on Eqs 1-4. However, the notations are not clear, what's the formal definition of $f'$, why $f'$ needs to be introduced, and how to leverage it?

Author response:
W2) Sorry for the ambiguous explanation. Below is our response to weakness 2:

Given a locally convex function $y=f(w)$, an optimizer tries to find its local minimum $w_*$ with an initial guess of $w_0$. Here, we stretch $f$ in the $w$-axis by a factor of $\alpha$. Therefore, the formal definition of $f'$ is given as $f'(w)=f(w/\alpha)$. [Eq. (1)]

The reason we introduce $w'$ is to emphasize the relationship between the local minimum and the initial guess of $f'$ and $f$, where the local minimum of $f'$ becomes $\alpha w_* = w'_*$ and our initial guess becomes $\alpha w_0 = w'_0$.

The above observation leads to our main idea: stretching $f$ by $\alpha$ **increases** the distance to the solution by the factor of $\alpha$, but the gradient **decreases** by the factor of $\alpha$. Therefore, if we use a learning rate of $\eta$ to find the local minimum on $f'$, its behavior will be equivalent to using a learning rate of $\eta/\alpha^2$ to find the local minimum on $f$. [Eqs. (2-4)]

In the above discussion, we focused on the case of a scalar-valued $\alpha$. However, the same principles can be applied to transform adaptive gradient methods into steepest descent by stretching $f$ with different values for each axis.

&nbsp;

Reviewer comment (Theory):
In Section 3, equation (7) is also confusing, in Adam, $\alpha$ is also depending on the randomness of the stochastic gradients, when why $E u = \nabla f(w)/\alpha^2$ can hold?

Author response:
W3) Thank you for pointing out the ambiguity. Below is our response to weakness 3:

Eq. 7 holds true only when the coefficient for computing the moving average of second moment (denoted as $\beta_2$ in the case of Adam) is sufficiently large so that the stochasticity of gradient can be ignored.

Reviewer comment (Theory):
The reasoning from (9)-(11) is also not clear to me, if you only want to mention that the learning rate should not depend on $\alpha$, why do you still need equations (10) and (11)?

Author response:
W4) Sorry for the confusion. Below is our response to weakness 4:

While the learning rate may depend on $\alpha$, we would like to emphasize that the factor that should not depend on $\alpha$ is the trajectory of parameter. Assume an optimizer updates the parameter of $f$ with $w_0$, $w_1$, $w_2$, ..., $w_t$ and updates $f'$ with $w'_0$, $w'_1$, $w'_2$, ..., $w'_t$. Here, we used the stretched function $f'(w)=f(w/\alpha)$. If a learning-rate-free optimizer actually estimates the optimal learning rate, then the trajectory should be identical regardless of $\alpha$, i.e., $w'_i=w_i/\alpha$ for all $i$.

However, without employing our parameter scaling approach, simply scaling gradients while leaving parameters intact (which is the case of previous adaptive gradient methods) results in trajectories that depends on $\alpha$. [Eqs. (9-11)]

In contrast, it can be easily shown that our parameter scaling approach ensures identical trajectories as following. In this case, we estimate the learning rate on $\alpha$-stretched $f’(w’)$, which means $f'(w')=f(w)$, $\nabla f'(w')=\nabla f(w)/\alpha$, and $w’=\alpha w$. Therefore, $E[\eta u]$, which means the estimated step, becomes proportional to $\alpha$. Since $\alpha$-stretched function leads to an estimated step that is proportional to $\alpha$, out parameter scaling approach ensures identical parameter trajectories across different scalings.

&nbsp;

Reviewer comment (Theory):
The convergence analysis is also not clear, the authors just provide a very simple proof in the appendix, in the main part, I actually do not see anything that is related to the convergence. Additionally, the proof is also not clear, many notations such as D,G are not clearly presented; the assumption that $\alpha_k$ coverges to $\alpha$ is also not presented; Eq. (15) is also not well justified.

Author response:
W5) Sorry for the insufficient explanation. Below is our response to weakness 5:

We omit the details about the proof in the main text, because they are simple and constitute only a minor part of our contribution. $G$ denotes the Lipschitz constant of function $f$ and $D$ denotes the distance to optimum from the initial guess. While Alg. 1 and Alg. 2 accept arbitrary scaling factors $\alpha_k$, we employed the AMSGrad [3] scaling rule for the convergence proof, denoted as $\alpha^2_k=\sqrt{\max_{i\le k} v_i}$. As $\alpha_k$ is a non-decreasing sequence with an upper bound of $\sqrt{G}$, it converges to a constant and we denote the constant as $\alpha$.

Further details regarding Eq. 15 are provided below.

If we consider two vectors $x=[x_0, x_1, ..., x_n]$ and  $y=[y_0, y_1, ..., y_n]$ with $x_i \ge y_i$ for all $i$, we can define the element-wise division $x \circ y^-1=[x_0/y_0,...,x_n/y_n]$. Since $x_i/y_i \ge 1$, it follows that $\max (x \circ y^-1) \ge \prod_i x_i / \prod_i y_i$. We define $\beta_k$ as the product of all elements of $\alpha_k$. Since $\alpha_k$ is a bounded vector with a finite number of elements, $\beta_k$ is also bounded. Finally, we define $\beta_n \cdot \beta_m ^{-1}=A^2$.

Additionally, as $\alpha_k$ is a non-decreasing sequence, it follows $\max (\alpha_k \circ \alpha^{-1}_{k-1}) \ge 1$.

&nbsp;

[1] Learning-Rate-Free Learning by D-Adaptation, Aaron Defazio, Konstantin Mishchenko, ICML (2023)

[2] Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence, Loizou, Nicolas, et al., AISTATS (2021)

[3] On the Convergence of Adam and Beyond, Reddi, Sashank J., Satyen Kale, and Sanjiv Kumar., ICLR (2018)