Paper: Clip21: Error Feedback for Gradient Clipping (ICLR-2024)

Reviewer comment (Theory):
While gradient clipping is originally known to solve the exploding gradient problem, it seems not reflected in theoretical analysis of this paper. This paper still considers the standard $L$-smooth setting rather than $(L_0,L_1)$-smoothness, so the benefit of using clipped gradients is unclear from the theory.

Author response:
We agree that Clip-GD enjoys stronger convergence guarantees than GD for problems under the $(L_0, L_1)$ smoothness assumption, which implies the exploding gradient condition in deep neural network training [1]. However, existing results of gradient clipping algorithms are limited for the centralized case (one master and one client). 

Even under the $L$-smoothness assumption, we show that distributed clipping algorithms do not even converge towards the optimal solution (see Example 1). This motivates our investigation to modify standard clipping methods to attain strong convergence for distributed settings. We also prove theoretically and empirically that Clip21-GD outperforms these standard gradient clipping algorithms. 

Furthermore, we believe that our analysis is sufficient to prove the convergence under the $(L_0, L_1)$ smoothness. We prove this by finding the upper-bound for $\Vert \nabla^2 f(x_k) \Vert$ and our results apply as follows: 

1. By the $(L_0, L_1)$ smoothness and Assumption 3 of [A1],  

$$
\Vert \nabla^2 f(x_k) \Vert \leq L_0 + L_1 \Vert \nabla f(x_k) \Vert \leq L_0 + L_1 \sqrt{2L[f(x_k)-f_{\inf}]}.
$$

2. We then have $\Vert \nabla^2 f(x_k) \Vert \leq L_0 + L_1 \sqrt{2L\phi_0}$ for all $k \geq 0$ by proving that $f(x_k) \leq f_{\inf}+\phi_0$ by the induction and by setting the step-size $\gamma$ sufficiently small. 

We also refer the reviewer to our general response, where we explain why the current analysis from our paper holds under $(L_0, L_1)$-smoothness.

[A1]: Khaled, Ahmed, and Peter Richtárik. "Better theory for SGD in the nonconvex world." arXiv preprint arXiv:2002.03329 (2020).

Reviewer comment (Theory):
This paper only considers the case for deterministic gradients. Although it is expected that similar results hold for stochastic gradients, in the gradient clipping literature more restrictive assumptions are made on noise (bounded noise rather than the more standard bounded variance, see e.g. [1]). It is unclear whether gradient noise is an issue in the distributed setting.

Author response:
Stochastic optimization algorithms suffer from instability for neural network training, which is in the presence of heavy-tailed stochastic gradient noises. This has been observed empirically for training neural network models by centralized stochastic gradient descent in Zhang et al. (2020b), and distributed methods in [M1]. Furthermore, deriving the result for Clip21-GD with stochastic gradients is non-trivial, because of (1) the dependency of the clipping threshold $\tau$ on the noise variance parameters, and (2) the required condition on the bounded norm of the input vector before clipping. This may be proved by the strong bounded noise condition, e.g. $\Vert g - \nabla f(x) \Vert^\alpha \leq \sigma^\alpha$ with probability 1. 

[M1] Yang, Haibo, Peiwen Qiu, and Jia Liu. "Taming Fat-Tailed (“Heavier-Tailed” with Potentially Infinite Variance) Noise in Federated Learning." Advances in Neural Information Processing Systems 35 (2022): 17017-17029.

Reviewer comment (Theory):
(related to weaknesses 1) What is the motivation of considering gradient clipping at the client level? Because in a distributed setting, one only wants to optimize the averaged objective function, can we just clip the averaged gradient at the server to accelerate training? If you clip the server's gradient instead of the clients, then the problem demonstrated by Example 1.1 does not exist, and there may be a more straightforward approach.

Author response:
Implementing the client-side gradient algorithm is necessary for designing federated averaging algorithms that stabilize neural network training while saving communication costs, e.g. CELGC by Liu et al. (2022). Clip21-GD can be modified for these federated learning applications to accelerate the convergence of CELGC by Liu et al. (2022). This is because distributed gradient clipping algorithms can be viewed as a special case of CELGC by Liu et al. (2022) with the number of local steps $I=1$. Also note that the implementation of clipping the average gradient of clients can be viewed as Clip21-GD with the $n=1$ case supported by our theory.

Reviewer comment (Theory):
The article's motivation may need further clarity. While it claims to address the importance of deep neural network (DNN) training, the use of gradient descent (GD) in Clip-GD may appear unsuitable for DNNs. While Clip-SGD is briefly mentioned in the appendix for VGG 11 training, its theoretical convergence rate is not thoroughly explored in the main text, leaving room for ambiguity.

Author response:
We believe that Clip21-GD can be modified to obtain similar convergence guarantees when it uses stochastic gradients. This is because EF21 that inspires Clip21-GD attains the $\mathcal{O}(K^{-1})$ convergence rate when it uses stochastic gradients for nonconvex problems (see Theorem 3 in Appendix D.1 of [L1]). From this theorem, the residual error term of running EF21-SGD decreases when we increase the mini-batch size of stochastic gradients.  

However, deriving the result for Clip21-GD with stochastic gradients is non-trivial because of the required condition on the bounded norm of the input vector before clipping. This condition can be proved by the strong bounded noise condition, e.g. $\Vert g - \nabla f(x) \Vert\leq \sigma$ with probability 1 [L2]. 

[L1] Fatkhullin, Ilyas, Igor Sokolov, Eduard Gorbunov, Zhize Li, and Peter Richtárik. "EF21 with bells & whistles: Practical algorithmic extensions of modern error feedback." arXiv preprint arXiv:2110.03294 (2021).

[L2] Zhang, Bohang, Jikai Jin, Cong Fang, and Liwei Wang. "Improved analysis of clipping algorithms for non-convex optimization." Advances in Neural Information Processing Systems 33 (2020): 15511-15521.

Reviewer comment (Experiments):
The necessity of a client-side gradient clipping algorithm is not sufficiently explained. The implementation of clipping the average gradient of clients on the server side may seem more straightforward and effective in managing explosive gradients to prevent convergence issues.

Author response:
We believe that implementing the client-side gradient algorithm is necessary for designing federated algorithms that stabilize neural network training while saving communication costs, e.g. CELGC by Liu et al. (2022). Clip21-GD can be extended for these federated set-ups to further accelerate the convergence of CELGC by Liu et al. (2022). This is because distributed gradient clipping algorithms can be viewed as a special case of CELGC by Liu et al. (2022) with the number of local steps $I=1$. Also note that the implementation of clipping the average gradient of clients can be viewed as Clip21-GD with the $n=1$ case supported by our theory.

Reviewer comment (Theory):
Element-wise gradient clipping is typically used in DP-SGD (Abadi et al, 2016), where clipping controls the sensitivity of gradient steps and Gaussian noise is added to the clipped gradient to create a Gaussian mechanism. In Algorithm 1, the clipped vectors are aggregated, but the influence of a noise $\zeta_i$ on the mean estimate $\frac{1}{n}\sum_iv_i$ is not studied, making it unclear whether this algorithm would work in the context of differential privacy.

Author response:
Although DP-Clip-SGD for distributed settings (Abadi et al, 2016) has been studied empirically, there is no theoretical convergence for this method. Even without DP noise, distributed Clip-GD does not converge towards the optimum for quadratic problems in Example 1.1. This motivates us to redesign distributed Clip-GD that leverages the error-feedback framework to ensure convergence. Deriving DP guarantees for a version of Clip21-GD is a prominent research direction for future work.

Reviewer comment (Theory):
The paper notes it is not using clipping as a tool for taming the exploding gradients problem and is fully complementary to such literature. However, clipping after averaging, as presented, does not cause the severe bias and divergence issues addressed in their work.

Author response:
We agree that clipping helps stabilize neural network training which has exploding gradient problems, which can be modeled by the $(L_0,L_1)$ smoothness condition. We believe that our analysis is sufficient to prove the convergence under the $(L_0, L_1)$ smoothness. We prove this by finding the upper-bound for $\Vert \nabla^2 f(x_k) \Vert$ and our results apply as follows: 

1. By the $(L_0, L_1)$ smoothness and Assumption 3 of [A1],  

$$
\Vert \nabla^2 f(x_k) \Vert \leq L_0 + L_1 \Vert \nabla f(x_k) \Vert \leq L_0 + L_1 \sqrt{2L[f(x_k)-f_{\inf}]}.
$$

2. We then have $\Vert \nabla^2 f(x_k) \Vert \leq L_0 + L_1 \sqrt{2L\phi_0}$ for all $k \geq 0$ by proving that $f(x_k) \leq f_{\inf}+\phi_0$ by the induction and by setting the step-size $\gamma$ sufficiently small. 
We also refer the reviewer to our general response, where we explain why the current analysis from our paper holds under $(L_0, L_1)$-smoothness.

[A1]: Khaled, Ahmed, and Peter Richtárik. "Better theory for SGD in the nonconvex world." arXiv preprint arXiv:2002.03329 (2020).

Reviewer comment (Experiments):
The paper positions itself against the Clip-GD algorithm (among others) in all experiments, but experiments comparing against vanilla GD are lacking.

Author response:
Our theory and experiments suggest that Clip21-GD outperforms the Clip-GD for both deterministic and stochastic gradients since it can handle data heterogeneity and ensures strong convergence guarantees. We believe that Clip21-GD would outperform the naive clipping algorithm, which is faster than vanilla SGD for neural network trainin, e.g., in (Zhang et al.,2020a).

Reviewer comment (Experiments):
It is unclear what use-cases exist in which Clip21-GD outperforms vanilla GD (and not only Clip-GD). The algorithm's practical differences from vanilla GD are not clarified, and experiments comparing against vanilla GD are missing.

Author response:
Our theory and experiments suggest that Clip21-GD outperforms the Clip-GD for both deterministic and stochastic gradients since it can handle data heterogeneity and ensures strong convergence guarantees. We believe that Clip21-GD would outperform the naive clipping algorithm, which is faster than vanilla SGD for neural network trainin, e.g., in (Zhang et al.,2020a).

Reviewer comment (Writing):
There is a typo and LaTeX citation formatting issue. For citations, such as at the top of page 5, it is recommended to use `\citep{}` instead of `\cite{}`, for example in `Richt ´arik et al. (2021); Fatkhullin et al. (2021); Richt ´arik et al. (2022),`. Additionally, there is a missing `)` in the phrase: “it will become 'inactive' in at most k⋆ steps (i.e., ∇fi(xk) − vik−1 ≤ τ”.

Author response:
We thank the reviewer for this recommendation and for spotting this typo. We will correct this in our finalized manuscript.