Paper: Constrained Parameter Regularization (ICLR-2024)

Reviewer comment (Theory):
Although it is a very interesting paper, I feel like the analysis is not good enough. There are many interesting aspects that could be further explored IMHO. For example: If $\lambda$ is used to accumulate constraint violations, shouldn't it be $\lambda_{t+1}=\lambda_t + \mu(R(\theta_t) - \kappa)^+$?

Author response:
**W1:** *If $lambda$  is used to accumulate constraint violations, shouldn't it be $\lambda_{t+1} = \lambda_t + \mu (R(\theta_t )-\kappa)^+ $?*

Unfortunately, this wording was imprecise. $\lambda$ accumulates constraint function values (not constraint violations, otherwise $\lambda$ would never decrease). We adjusted the wording in the manuscript (Section 4.1).

Reviewer comment (Experiments):
Although it is a very interesting paper, I feel like the analysis is not good enough. There are many interesting aspects that could be further explored IMHO. For example: Have the authors considered adjusting $\lambda$ in a non-linear way? For example $\lambda_{t+1}=\lambda_t + \mu f((R(\theta_t) - \kappa))^+$, where $f$ is a non-linear function. Maybe it leads to faster convergence if tuned properly?

Author response:
**W2:** *Have the authors considered adjusting $\lambda$  in a non-linear way? For example $\lambda_{t+1} = \lambda_t + \mu f ((R(\theta_t )-\kappa))^+ $, where $f$ is a non-linear function. Maybe it leads to faster convergence if tuned properly?*

We have not tried a non-linear adjustment. The update rule is derived as the closed form solution for the maximization in $\hat{F}$, see Equation (2). Note that this can also be seen as a (projected) gradient ascent step on $f(x) + \lambda \cdot c(x)$ (see right side of Equation (1)) with respect to $\lambda$ and a learning rate $\mu$. Additionally, for a different choice of $\hat{F}$, a different update rule is obtained. 
Nevertheless, it should be noted that we use the method as a means to implement a computationally cheap adjustment procedure for the individual $\lambda_i$. It is not immediately clear if a faster convergence of the $\lambda_i$ would lead to more favorable training dynamics and thus an overall better training result.

Reviewer comment (Theory):
Although it is a very interesting paper, I feel like the analysis is not good enough. There are many interesting aspects that could be further explored IMHO. For example: Should $\lambda$ have a self decay factor? Similar to a diverging behavior with large learning rates, when a feasible point $\theta_{t}$ is found at a very large $\lambda_t$. In the next iteration, $\theta_{t+1}$ may overshoot in the opposite direction and becomes infeasible again because $\lambda_t$ has not yet become small enough.

Author response:
**W3:** *Should $\lambda$ have a self decay factor? Similar to a diverging behavior with large learning rates, when a feasible point $\theta_t$  is found at a very large $\lambda_t$. In the next iteration,  $\theta_{t+1}$  may overshoot in the opposite direction and becomes infeasible again because $\lambda_t$ has not yet become small enough.*

If the constraint violations are growing moderately, and since we initialize $\lambda=0$, $\lambda$ should also not be growing too quickly. Additionally, the procedure seeks to find a stationary point where $\eta  \cdot \nabla L = - \lambda \cdot \nabla c$ (in case of gradient descent). Instabilities may arise, for example, if one of the balancing forces, in this case, the update direction for $L$ may drastically change. However, this should be unlikely when training with momentum on $L$. 
We thus assume that only a very unstable training progress could potentially cause such issues, and we never observed them in any of our experiments.

Reviewer comment (Evaluation):
How significant is the performance difference between CPR and weight decay? As the absolute performance difference is not huge, it would be great if the standard deviation of different runs of the same setting can be provided.

Author response:
**W5:** *How significant is the performance difference between CPR and weight decay? As the absolute performance difference is not huge, it would be great if the standard deviation of different runs of the same setting can be provided.*

We performed additional experiments for three random seeds in the language modeling experiment (similar to the image classification experiments) and added the standard deviation for experiments on image classification and language modeling. We added the standard deviation for the LLM experiments in the plot and in Tables 6 and 7 in Appendix G and for the image classification experiments in the heat map plots in Appendix F. 
To assess the significance of our results, we also performed a Welch’s t-test on the GPT2s experiments with the best AdamW and best AdamCPR configuration, showing a significant difference with a p-value of 0.018.

Reviewer comment (Experiments):
Maybe I missed it, but how about a baseline of weight decay factor dependent on the magnitude of the parameters, e.g. larger weights leads to a larger decay coefficient (linear or non-linear dependence)? I wonder how well this could achieve the goal of the proposed method.

Author response:
**W6:** *Maybe I missed it, but how about a baseline of weight decay factor dependent on the magnitude of the parameters, e.g. larger weights leads to a larger decay coefficient (linear or non-linear dependence)? I wonder how well this could achieve the goal of the proposed method.*

We are unsure if we understand this idea of a baseline correctly. Since the decoupled weight decay update already contains the parameter in the update step: $\theta_i = \theta_{i-1} - \eta \cdot \operatorname{Opt}(L)  - \eta \cdot \gamma \cdot \theta_{i-1}$. Should the weight decay value $\gamma$ additionally be a function of $\theta_0$ or  $\theta_{i-1}$?

Reviewer comment (Experiments):
The authors claim that the method is not sensitive to $\mu$, but it would still be great to see the analysis on it.

Author response:
**WM2:** *The authors claim that the method is not sensitive to $\mu$  but it would still be great to see the analysis on it.*

We agree that the paper missed so far to show experiments on the update rate $\mu$. We added a new section in the Appendix (Appendix D) with two plots. One on the modular addition task and one on the image classification task which compares different $\mu$ values and demonstrates the low sensitivity of the update rate.

Reviewer comment (Presentation):
Better to use consistent notations. E.g. in Figure E1, fix -> Kappa-K, dependent -> Kappa_kI0, warm started ->Kappa_Is.

Author response:
**WM3:** *Better to use consistent notations. E.g. in Figure E1, fix -> Kappa-K, dependent -> Kappa_kI0, warm started ->Kappa_Is*

Thanks for pointing us to this inconsistency, we corrected the figures accordingly.

Reviewer comment (Experiments):
Larger scale evaluation, e.g. on image-net, would be strongly advised.

Author response:
**WM4:** *Larger scale evaluation, e.g. on image-net, would be strongly advised.*

We expected the GPT2m experiment to be a large-scale experiment since the GPT2m model has 354M parameters (in contrast to 124M in GPT2s). As far as we know, a ResNet152, which is often used for imagenet training, only has ~64M parameters. 

If we have addressed (some of) your concerns, we would be very thankful if you considered raising your score. We would be very happy to answer any follow-up questions and concerns.

Reviewer comment (Writing):
The authors should provide a clearer exposition of the problem statement and research motivation in the abstract.

Author response:
We thank the reviewer for the constructive comments. We adjusted the abstract accordingly
and tried to clearly emphasize the research question. We moved the motivation to the beginning of the abstract and reformulated it to have a clearer and more trenchant abstract.

Reviewer comment (Novelty):
In the related work section, the authors should provide a more comprehensive review of prior research.

Author response:
We added an additional related work on adaptive weight decay [Nakamura & Hong (2019)] and compared it experimentally. We also added baseline experiments on weight decay schedule and rescaling. Please find the experimental results in the appendix (Figures E2, E3, E4). To the best of our knowledge, we mentioned all relevant related work. Does the reviewer know any additional related work we missed so far? We would gladly add it.

Reviewer comment (Experiments):
The experimental results conducted by the authors did not show a significant improvement in optimization performance, and they did not provide additional experimental results or comparisons related to existing work.

Author response:
We performed additional experiments for three random seeds in the language modeling experiment and added the standard deviation for experiments on image classification and language modeling. We added the standard deviation for the LLM experiments in the plot and in Table G.6 and G.7 in Appendix G and for the image classification experiments in all heat map plots in Appendix F. 
To assess the significance of our results, we also performed a Welch’s t-test on the GPT2s experiments with the best AdamW and best AdamCPR configuration, showing a significant difference with a p-value of 0.018. 
Furthermore, we provide additional experiments for comparison to existing work (see comment on the previous point). Does this address the concerns of the reviewer regarding the significance and comparison to existing work?

Reviewer comment (Novelty):
While applying different regularization strengths to each parameter group may seem intriguing, the authors have not discussed the necessity and potential advantages of this approach compared to traditional methods.

Author response:
Choosing individual regularization strength is a more general case of choosing one strength for all parameter groups. Our hypothesis was that this additional flexibility leads to better performance. E.g., we do not regularize parameters that do not need regularization. We clarify this in the modified abstract too. In the introduction, we already write: "However, not all parameters in a neural network have the same role or importance and different weights could benefit from different regularizations. Similarly, it is unclear if a single weight decay value is optimal for the entire duration of optimization, especially for large-scale training.” Does this address your concern?

Reviewer comment (Experiments):
It’s difficult to find a good value of the upper bound for each group of the parameters.

Author response:
Indeed, there is still no hyperparameter-free regularization technique. Similar to the weight decay value, one needs to choose e.g. the warm start steps. These could be more  interpretable or transferable than the weight decay value, but we agree that so far this is a weakness and should be addressed in future works.

Reviewer comment (Evaluation):
The accuracy gain for the language model is quite small, so it’s not clear to say that AdaCPR is better than AdamW on this task.

Author response:
We now performed three random seeds on the GPT2 experiments and added the standard deviation in the plot and in Table G.6 and G.7 in Appendix G. We also performed a Welch’s t-test on the GPT2s experiments with the best AdamW and best AdamCPR configuration, showing a significant difference with a p-value of 0.018.

Reviewer comment (Experiments):
“We can also close the gap between training and validation performance by increasing the weight decay regularization but this comes at the price of unstable training.” Shouldn’t we compare the AdamAdaCPR with AdamW with a bigger regularization parameter? Since our concern here is the grokking effect.

Author response:
We agree, we can also compare it to an AdamW training with a higher weight decay, therefore we added a new plot (E.7) in Appendix E. We see the ability of weight decay to close the gap as well. We also see the instability in comparison to a more stable behavior with AdamCPR and AdamAdaCPR.

Reviewer comment (Evaluation):
There are large performance drops in Table 3 when the number of warm-start steps increases from 3k to 4k. This increment is not large compared to the total number of training steps (25k). Does this imply that CPR with Kappa-I_s is highly sensitive to the warm-start parameter?

Author response:
Thank you for mentioning this issue. The problem arises because the datasets are very small. Consequently, unregulated optimization tends to overfit quickly which leads to NaNs. When CPR regularization starts too late, as was the case after 4000 steps in this instance, the model is already overfitted, resulting in some broken folds. These are counted as zero in the mean, which explains the low number in the table.