Paper: Analysis of a class of stochastic component-wise soft-clipping schemes (ICLR-2024)

Reviewer comment (Theory):
Although the authors mention in the abstract that theoretical guarantees can provide guidelines for us to choose what algorithms to use, there is little discussion about this aspect in the paper. More specifically, it is unclear through the theoretical results why or when soft clipping is better than hard clipping or vanilla SGD. For the non-convex case, the convergence rate is only $O(1/\log K)$ which is much slower that standard results for GD/SGD or (hard) gradient clipping, though the assumptions seem to be a little different.

Author response:
With "a theoretical foundation" and "motivates their usage", we only intended to convey that we have proved that they converge under reasonable conditions, not that theory necessarily recommends a specific method. In fact, like we mentioned in the responses to the other referees, we do not yet have a full understanding of which method would be the optimal choice in a given situation. The benefit of soft clipping over hard clipping is mostly empirical, see e.g. the comments in [Zhang et al., 2020a]. As to why clipping should be preferred over vanilla SGD, see e.g.\ the references [3] and [4] provided by the referee xFTp. These should obviously have been referenced in our paper already and have been added to the revised version.

Reviewer comment (Theory):
It seems to me that although soft and hard clipping are not exactly equivalent, they are basically the same since the induced step size only differs by a constant scale. In the introduction, the authors motivate the study of soft clipping by saying that "A drawback of the rescaling in (4) is that it is not a differentiable function of the gradient", which looks confusing, since it is unclear why being differentiable here is favorable.

Author response:
That hard and soft clipping are "basically the same" is essentially the argument in the paper by Zhang et al. (2020a), which has been propagated from a similar argument in previous papers with some of the same authors. This seems plausible when stated in this vague form, but when doing the details it becomes apparent that it is in fact not true. That is, the convergence proof for hard clipping does not translate to a proof of convergence for soft clipping. We have now added a few sentences commenting on this.

As stated above, the fact that soft clipping is preferable to hard clipping is mostly empirical and we do not have a rigorous proof of why differentiability is desirable. However, experience from control theory and ODE time-stepping strongly suggests that this is the case. Essentially, it makes the errors more predictable, and the method more robust.

Reviewer comment (Experiments):
I think there are some existing works on either component-wise or non component-wise clipping, and some of them are cited in this paper, but there is little comparison between these papers and the current paper.

Author response:
In the revised version, we have improved the sections on contributions and literature overview. This includes more comparisons between our work and that in related papers.

Reviewer comment (Experiments):
In the experiments, do you compare component-wise clipped SGD with the original clipping method (non component wise)?

Author response:
As stated in the text, we are comparing to the component-wise clipped SGD. We thought this was most relevant.

Reviewer comment (Theory):
Can you briefly discuss why the theoretical results in this paper imply that component-wise clipped SGD is superior to other popular methods?

Author response:
We do not claim that these methods are superior, and the analysis does not suggest this. It does, however, prove convergence for a general class of methods in a unified framework. In particular, we observe the somewhat unexpected result that the trigonometric rescaling converges. This doesn't mean that we expect it to be a great method. In fact, we included it mostly to show that the analysis is applicable to a large class of functions.

Reviewer comment (Theory):
From the theoretical perspective, the existing results under considered assumptions are either stronger or match the derived ones. Indeed, under bounded variance and smoothness assumptions, standard SGD also converges (and the known rates are even better), e.g., see [1, 2].

Author response:
W.1.1: We might be missing something here, but as far as we can see, our rates are the same as for SGD. The result in [2] for strongly convex problems is $\mathcal{O}(1/k)$ which is the same as in our Corollary 2 (3 in the revised version). The main result Theorem 2.1 in [1] says that $\|\nabla f(w)\| = \mathcal{O}(1 / \sum \alpha_k)$, where $\alpha_k = 1/k$ would give a $\log k$ term in the denominator. One main difference is that the results for SGD require a step size restriction, which can be hard to evaluate a priori in practice. The clipped schemes do not have such a restriction.

Reviewer comment (Theory):
Assumption 6 is quite restrictive: it is unclear apriori whether it holds in the considered setup. Indeed, it can be the case that $\mathbb{E}[\|\|w^k - w^\ast \|\|^3] = \infty$ while Assumptions 3 and 4 are satisfied, e.g., for simple quadratic function with linear noise $f(v,\xi) = \|\|v\|\|^2 + \langle \xi, v \rangle$, where $\xi$ is zero mean random vector with a bounded variance but an unbounded third moment (e.g., one can take shifted Pareto distribution with tails decaying as $x^{-4}$) we have Assumption 3 with $L_{\xi} \equiv 2$, Assumption 4 is also satisfied with some $\sigma$, but Assumption 6 is not satisfied. The authors refer to the work of Eisenmann & Stillfjord (2022), but in that work, it is assumed that the fourth moment of the gradient is bounded (and the fourth moment of $L_\xi$) to derive Assumption 6. Such assumptions are even stronger than for the standard analysis of SGD.

Author response:
W2.1: We agree that the assumption is somewhat non-standard and may not be fulfilled in every problem. But in practice, when there is only a finite amount of data and thus also finite possibilities for choosing the stochastic approximation, the third moment will be bounded if the second moment is. The setting in Eisenmann & Stillfjord (2022) indeed does not perfectly match our setting, in that it requires strong convexity. We mainly wanted to state an example included in our setting where such a moment bound on the increments can be observed. Note that the fourth moment of the approximate gradient is not assumed to be bounded for the basic convergence result, it is only used to improve the rate

Reviewer comment (Theory):
The result of Theorem 1 seems to be not tight: when full gradients are used the theorem does not recover the rate of a deterministic counterpart -- Gradient Descent. This happens because $B_i \neq 0$ even when full gradients are used in the method. A similar issue is present in Theorem 2. Moreover, one can improve the rate from Corollary 1 to $\mathcal{O}(1/\sqrt{K})$ (instead of $\mathcal{O}(\frac{1}{\ln(K)})$) if one takes $\alpha_k \equiv 1/\sqrt{K}$.

Author response:
W2.2 (second comment): Yes, this is true. Our emphasis in the article is on the decreasing step size regime, but a slight extension yields $\mathcal{O}(1/\sqrt{K})$-convergence in the non-convex case with the mentioned step size. We have added a second corollary about this in the non-convex part, and a similar remark in the convex case.

Reviewer comment (Theory):
Assumption 1 does not include some interesting special cases as standard gradient clipping (the second part of the assumption does not hold).

Author response:
W2.3 We have updated our main assumption on the methods in line with referee pggZ's comments, and in the new version this method is included.

Reviewer comment (Novelty):
The main question is about the motivation of this work: why and when SGD-type methods with soft clipping should be used? In the current shape, the theory shows no benefit over the existing results. Moreover, the derived results are even weaker since they rely on an additional restrictive assumption. The numerical experiments also do not provide enough evidence that the methods with soft-clipping can outperform the existing algorithms (in both experiments standard Clipped-SGD performs better than other methods with soft clipping).

Author response:
As stated in the responses to the other referees, we do not agree that the Clipped-SGD outperforms the other methods. They essentially perform equally well. We have tried to state more clearly now that the point of the paper is not to suggest new methods that always outperform the state of the art.

Reviewer comment (Novelty):
Although the state of the art is well stated which allows one to dig further into the appropriate references, I believe a more explicit *comparison* with the state of the art, gathered at one place in the paper (perhaps in Appendix) would improve the quality of the paper. For instance, I think the two most related papers are the ones from Eisenman & Stillfjord (2022), and the one from Zhang et al. (2020a). Even though by checking those references one may get some idea on the difference between those papers and the submitted one (i.e. Eisenman & Stillfjord (2022) only deal with a strongly convex setting, while Zhang et al. (2020a) have slightly different assumptions, and also, the submitted paper studies a general form of clipping)), I still think it would be good to compare the method exposed in the paper with those two papers more systematically (perhaps in Appendix, in a structured form such as a table) (for instance, saying which algorithms is a special case of which (some algorithms may have momentum, some may have general component-wise functions etc), what assumptions they make in the papers, and what is the resulting convergence rate. For instance, in the strongly convex setting, does the results in the paper retrieve the results from Eisenman & Stillfjord (2022)? Also, it would be interesting if possible to compare how taking different assumptions and/or considering more general settings actually impacts the proofs compared to the previous ones in the literature.

Author response:
We have slightly extended the literature review and contributions section, and hope that the differences the referee points out are now more clear. Given how many different settings that could be considered, and how slightly different assumptions lead to significantly different rates, we do not think an exhaustive comparison is feasible. The presented analysis is the first attempt at considering this general class of methods, and we think it is also somewhat unfair to compare this to e.g. highly refined proofs for SGD in special settings. However, in line with the comments by the referee xFTp, we have added some straightforward extensions of the presented analysis on certain fixed step sizes.

Reviewer comment (Novelty):
Although a general form of clipping is analyzed, if I am not mistaken, I didn’t see any motivation or related works related to the new gradient clipping schemes from the Appendix A (i.e. Examples 2-4). And in practice it seems that those do not make a big difference (most curves are superimposed so it is hard to differentiate them). Therefore, even though having a general convergence proof is always good, I think it would be even better to further motivate the use of any of the new Examples 2-4 from Appendix A, for instance by mentioning whether they are present in the literature, discussing how their shape depart from the classic soft-clipping (and why it is advantageous), or through experimental results that would show their advantage in some cases.

Author response:
As mentioned in the responses to the other referees, we do not yet have a full understanding of which rescalings are best in each situation. Precisely like the referee comments, the error curves are essentially superimposed in both our numerical examples. This indicates that the different rescalings often behave similarly in practice, so that doing some rescaling is more important than what kind of rescaling. The idea of the paper is to simultaneously analyze most such rescalings in a common framework, and finding the optimal choice in a particular situation is still an open problem.

Reviewer comment (Novelty):
The comparative analysis with other literatures is insufficient, and it is difficult to see the innovation of the convergence results or proofs in this paper.

Author response:
We have slightly extended the literature review and contributions section, in line with the comments by referees pggZ and GtoB. If there is some specific additional reference which the referee thinks should also be included, please let us know.

Reviewer comment (Theory):
This paper lacks some intuitive understanding and analysis of the theorems and corollaries given. Especially for the symbols $\w_k(w)$ without interpretation in Corollary 2, it’s hard for readers to understand and what insight the corollary hopes to provide.

Author response:
We apologize for the confusing statement of Corollary 2. We did not realize that after some reordering of the material, it was no longer specified that $\omega$ was an element of the sample space $\Omega$. It has now been reformulated. The idea of Corollary 2 is just that the norm of the gradient of $w_k$ will almost surely (i.e. for essentially every random path) go to zero. The min over the $K$ iterations is needed for technical reasons, similar to e.g. the results for standard SGD.

Reviewer comment (Novelty):
Have similar convergence analyses been conducted in other literature? If they exist, can you provide a comprehensive comparison to highlight the advantages of the results presented in this paper?

Author response:
The general class of methods is, to our knowledge new, and there is no convergence existing analysis. With some specific choices of G, we acquire previously analyzed methods like SGD or the TSGD. We have commented on this in the contributions and literature overview sections.

Reviewer comment (Theory):
Is it reasonable to assume that $\sum_{k=1}^\infty \alpha_k^2 < \infty$ and $\sum_{k=1}^\infty \alpha_k = \infty$ as stated in Theorem 1? Could you offer a more intuitively understandable explanation for this assumption?

Author response:
Yes, this is a standard assumption that goes back all the way (at least) to the paper by Robbins and Monro. It means that the step-size sequence goes to zero, but not too quickly. The idea is that we have to take large enough step sizes to actually get closer to the minimum. But once we get close enough, the noise in the stochastic gradient would prevent us from getting closer. By decreasing the step size, the noise is reduced. The assumed step size condition adequately balances these two conflicting demands.

Reviewer comment (Novelty):
While the topic is clearly of interest, there is uncertainty about the effective novelty of the contribution. Theorem 3.1 and Theorem 3.2 in Zhang et al. (2020a) already provide convergence results for a hard-clipping algorithm, and Appendix F of the same paper gives a reasonable explanation of why such results should generalize to the soft-clipping version of their algorithm.

Author response:
We are aware of the argument in the paper by Zhang et al. (2020a), which has been propagated from a similar argument in previous papers with some of the same authors. As the referee notes, the argument seems "fairly reasonable" when stated in this vague form, but when doing the details it becomes apparent that it is in fact not true. That is, the convergence proof for hard clipping does not translate to a proof of convergence for soft clipping. We have now added a few sentences commenting on this.

Reviewer comment (Presentation):
Figure 1 and Figure 2 lack confidence bars around the lines to represent uncertainty. They are difficult to read, and it would be more interesting to plot those lines in log-scale to better compare the performance of the methods.

Author response:
1. We unfortunately do not see an easy way to add confidence bars in our computational setup. We also suspect that doing so would further decrease the readability of the figures.

[...]

Finally, Figure 1 and 2 are already on a log-scale ("semilogy"), which means that the spacing near 100% and 90% (1 vs. 0.9) is smaller than between e.g. 30% and 40%. We have now changed it to be not logarithmic, which separates the error curves a bit more.

Reviewer comment (Experiments):
None of the experiments presented are intended to verify the theoretical insights provided in Section 2, particularly Theorem 1, Corollary 1, and Theorem 2. It would be relevant to verify these results, even with a simple setup such as a quadratic landscape. This would validate the theoretical results empirically and assess how loose the bounds are. At least verifying Corollary 1 should be feasible given the explicit shape of the bound.

Author response:
The experiments are intended to illustrate the performance of the methods on real-world applications. The verification of the theorems and corollaries are their proofs in the respective appendices. We aimed for rather complex experiments as this seemed to be what this community wanted to see, but we're happy to include a more simple setup as well. The revised version has two experiments with $\nabla F(w) = Aw$, where we choose $A$ to be a stiff diagonal matrix (large ratio between largest and smallest eigenvalue). The first one provides some motivation for why it is beneficial to use a componentwise method in the stiff case, and the second one compares the different generalized clipping functions.

Reviewer comment (Presentation):
The text before Section 1 should be reformulated by (1) adding an “Introduction” section that discusses the problem, contributions, and relevance of the paper; (2) adding a “Related Works” or “Literature Review” section; and (3) ensuring the exposition is fluent and divided into paragraphs rather than a list of points.

Author response:
We have now rearranged the first section along the lines of the referee's suggestion. We are sorry to hear that the referee thinks the text is unpleasant to read, but several of the other referees seem to think that it is easy to follow.

Reviewer comment (Presentation):
Section 1 (“Setting”) should be moved to the appendix, with its content summarized briefly in the main text. Standard assumptions can be noted in a sentence at the start of Section 2, with any non-standard assumptions stated clearly in the main paper.

Author response:
In the new version, we have followed the referee's suggestion to put the assumptions in the supplementary material. We also moved much of the setting description as well as some of the details about the experiment setups there.

Reviewer comment (Presentation):
In Section 3, there is no need to report all technical details of the numerical experiments in the main paper. Only the most important parts should remain, with the rest moved to the appendix.

Author response:
In the new version, we have followed the referee's suggestion to put the assumptions in the supplementary material. We also moved much of the setting description as well as some of the details about the experiment setups there.

Reviewer comment (Experiments):
None of the theoretical results are experimentally validated. Although the paper proposes an interesting class of optimizers, the theoretical analysis is too restricted for a substantial contribution and the experimental side is lacking and needs more attention. The manuscript also requires significant rewriting for better organization and clarity.

Author response:
The experiments are intended to illustrate the performance of the methods on real-world applications. The verification of the theorems and corollaries are their proofs in the respective appendices. We aimed for rather complex experiments as this seemed to be what this community wanted to see, but we're happy to include a more simple setup as well. The revised version has two experiments with $\nabla F(w) = Aw$, where we choose $A$ to be a stiff diagonal matrix (large ratio between largest and smallest eigenvalue). The first one provides some motivation for why it is beneficial to use a componentwise method in the stiff case, and the second one compares the different generalized clipping functions.