Paper: Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods (ICLR-2024)

Reviewer comment (Theory):
It will be good to add discussion about your techniques to prove the last iterate SGD convergence for non convex objectives.

Author response:
**Q1**: We remark that the key in our proof is to utilize the convexity of the objective as described at the beginning of Section 4 (before Lemma 4.1), which means this technique cannot be generalized to non-convex objectives. Additionally, as far as we know, the stochastic gradient methods generally cannot guarantee convergence in the function value gap without convexity unless some special assumptions (e.g., the Polyak-Lojasiewicz condition) hold. Hence, we think proving the convergence of the last iterate for the non-convex functions is beyond the scope of this paper.

Reviewer comment (Theory):
Bounded variance: This assumption contradicts strong convexity. I understand it is in several works in the literature. Take for instance F(x,y,z) = zx^2 + (1-z)y^2, z follows Bernoulli (1/2), then E_z(norm(\nabla F(x,y,z) - \nabla E_z(F(x,y,z)))^2) = (x-y)^2 and this is not bounded for all x and y in R.

Author response:
**Q2**: (We sincerely apologize for the following mistake in the first version of our response) ~~We do not understand your example very well due to the following two reasons. Could you explain more about your point?~~

~~1. In your example, we can find $\nabla^{2}F(x,y,z)=\mathrm{diag}(2z,2(1-z))$, which is only PSD for any realization of $z\in\left\\{ 0,1\right\\}$. Hence $F(x,y,z)$ itself is only convex but not strongly convex.~~

~~2. Moreover, it only requires the expectation of $F(x,y,z)$ to be strongly convex in our setting. However, there is $f(x,y)\coloneqq\mathbb{E}_{z}\left[F(x,y,z)\right]=0$, which is not strongly convex.~~

We remark that it is always possible to construct unbiased stochastic gradients with unbounded variance regardless of what conditions are required to hold for the objective function $F(x)$. For example, one can always consider the stochastic gradient as $\widehat{\nabla}F(x)\coloneqq\nabla F(x)+\xi$ where $F(x)$ can be any objective and $\xi$ is a centered r.v. only admitting finite $\alpha$-th moment where $\alpha\in(1,2)$ (e.g., $\alpha$-stable distribution). However, this counterexample does not mean the bounded variance condition should not be assumed. Arguably, it is the most standard and common condition when proving the expected convergence and appeared in lots of research related to stochastic optimization. Hence, we think the finite variance assumption is reasonable.

---

We highly appreciate the reviewer for the feedback!

- Thanks for pointing it out. Your example is correct and we sincerely apologize for our mistake. We agree that your construction gives a counterexample when considering the bounded variance and strong convexity assumptions. We will delete the corresponding part in the response accordingly.

- However, as mentioned in our response. It is indeed possible to construct stochastic gradients without bounded variance for any function. But this does not mean that bounded variance should not be assumed. Arguably, it is the most standard assumption.

- To summarize, we think both your example and bounded variance assumption are meaningful. In this work, our goal is to show the last-iterate convergence under the bounded variance assumption. For future work, we will try to relax it and prove the last-iterate convergence under more general assumptions (e.g., your example).

---

Thanks for the question. Here we give some practical examples where the finite variance assumption holds. 

Let's consider the general linear classification problem in the form of $f(x)\coloneqq\mathbb{E}_{(Z, Y)\sim\mathcal{D}}[\ell(\langle x, Z \rangle, Y)]$ where $\ell$ is some loss function, $Z\in\mathbb{R}^d$ denotes the feature, $Y\in\\{\pm1\\}$ is the label, and $\mathcal{D}$ is the distribution of the data. We additionally assume the feature $Z$ has bounded support, i.e., $\\|Z\\|_2\leq r$ almost surely for some $r>0$. Now we consider the following two popular loss functions:

- Logistic Loss: In this case, we have $f(x)=\mathbb{E}_{(Z,Y)\sim\mathcal{D}}[\log(1+\exp(-Y\langle x, Z \rangle))]$. For any given $(Z_i,Y_i)$, we can find the stochastic gradients as $\widehat{\nabla}f(x)=-Y_iZ_i/(1+\exp(Y_i\langle x,Z_i\rangle))$. Note that $\\|\widehat{\nabla}f(x)\\|_2\leq \\|Z_i\\|_2\leq r$, hence there is $\mathbb{E}[\\|\widehat{\nabla}f(x)-\nabla f(x)\\|_2^2]\leq 4r^2,\forall x\in\mathbb{R}^d$, which implies a bounded variance.

- Hinge Loss: In this example, we have $f(x)=\mathbb{E}_{(Z,Y)\sim\mathcal{D}}[\max(0,1-Y\langle x,Z\rangle)]$. For any given $(Z_i,Y_i)$, we can find the stochastic gradients as $\widehat{\nabla}f(x)=-Y_iZ_i\mathbb{I}[Y_i\langle x,Z_i\rangle\leq 1]$. Note that $\\|\widehat{\nabla}f(x)\\|_2\leq \\|Z_i\\|_2\leq r$, hence there is $\mathbb{E}[\\|\widehat{\nabla}f(x)-\nabla f(x)\\|_2^2]\leq 4r^2,\forall x\in\mathbb{R}^d$, which implies a bounded variance.

Reviewer comment (Novelty):
You mentioned in your criticism to the SOTA in the abstract that " the existing results are all limited to a single objective..", as far as I can check you are also considering only a single objective, not multi objectives. Composite optimization problem still a single objective?

Author response:
**Q3**: Thanks for pointing it out. The phrase "a single objective" has been changed to "a non-composite objective" in the first revision.

Reviewer comment (Novelty):
Before Lemma 4.1, the authors state "This brand new analysis is where we different from all the previous works." It is unclear what "previous works" are or what is "brand new." If the difference from Zamani and Glineur, 2023 exists, specify it; if not, clarify or modify the sentence. It is acceptable to use tools from another paper, but the usage and modifications should be clearly described.

Author response:
**Q2**: Before Sec. 4 the authors say "to the best knowledge, our results are entirely new and the first to prove ...." $\cdots$.

**A2**: These two sentences have been modified as the reviewer suggested.

Reviewer comment (Novelty):
Orabona, 2020’s blog post clearly left Bregman extension, smoothness, and composite cases as "exercises." These extensions should be straightforward; if not, the authors should explain why Orabona’s approach does not generalize. Otherwise, the contribution is enough without over-justifying. The authors can remark that these were not explicit in Orabona, 2020, but should not overemphasize them, focusing instead on the main contribution.

Author response:
**Q7**: Many places, such as the footnote in page 6 says that "Harvey et al. (2019a) claims their proof can extend do sub-Gaussian noises ...." $\cdots$.

**A7**: Thanks for the suggestion! In the second revision, we have further clarified the related discussion when talking about the assumptions on bounded noises and compact domains. Instead of saying the existing works rely on both these assumptions, we restate it as the existing works rely on one of these two assumptions. Consequently, we remove the previous footnote/sentence about [1].

Reviewer comment (Theory):
Are the results of Zamani, Glineur, 2023, and Liu et al., 2023 easily combined to get the high probability rate on the last iterate without bounded domains, or do additional difficulties arise? If there are difficulties, specify them. Even if no difficulties arise, clearly explanation is needed.

Author response:
The following two questions are about [1] and [2]
>**Q1**: Are the results of Zamani, Glineur 2023 and Liu et al., 2023 work well together to get the high probability rate on the last iterate $\cdots$.

>**Q8**: What are the main technical novelties in addition to Liu et al., 2023 $\cdots$.

**A1&8**: We list the key differences from [1] and [2] here:

**Compared with [1]**: The starting point of the proof is very different. The proof of [1] can be viewed as beginning from the convex inequality $f(x^t)-f(z^t)\leq\langle g^t,x^t-z^t\rangle$ where $g^t\in\partial f(x^t)$. However, if we still start from convexity when considering composite optimization (for simplicity, we assume there are no noises in the gradient, $\mu_f=\mu_h=0$, $\\|\cdot\\|=\\|\cdot\\|_2$ and $\psi(x)=\frac{1}{2}\\|x\\|_2^2)$, there will be
\begin{align*}
F(x^t)-F(z^t)&=f(x^t)-f(z^t)+h(x^t)-h(z^t)\leq\langle g^t,x^t-z^t\rangle+h(x^t)-h(z^t)\\\\
&=\langle g^t,x^{t+1}-z^{t}\rangle+\langle g^t,x^t-x^{t+1}\rangle+h(x^t)-h(z^t)\\\\
&\overset{(a)}{\leq}\frac{\\|z^t-x^t\\|^2-\\|z^t-x^{t+1}\\|^2-\\|x^{t+1}-x^t\\|^2}{2\eta_t}+h(z^t)-h(x^{t+1})+\langle g^t,x^t-x^{t+1}\rangle+h(x^t)-h(z^t)\\\\
&=\frac{\\|z^t-x^t\\|^2-\\|z^t-x^{t+1}\\|^2}{2\eta_t}+\langle g^t,x^t-x^{t+1}\rangle-\frac{\\|x^{t+1}-x^t\\|^2}{2\eta_t}+h(x^t)-h(x^{t+1})\\\\
&\overset{(b)}{\leq}\frac{\\|z^t-x^t\\|^2-\\|z^t-x^{t+1}\\|^2}{2\eta_t}+\frac{\eta_t\\|g^t\\|^2}{2}+h(x^t)-h(x^{t+1})\\\\
\Rightarrow\eta_t(F(x^t)-F(z^t))&\leq\frac{\\|z^t-x^t\\|^2-\\|z^t-x^{t+1}\\|^2}{2}+\frac{\eta^2_t\\|g^t\\|^2}{2}+\eta_t(h(x^t)-h(x^{t+1})),
\end{align*}

where $(a)$ is by the optimality condition for $x^{t+1}=\mathrm{argmin}_{x\in\mathcal{X}}h(x)+\langle g^t,x-x^t\rangle+\frac{\\|x-x^t\\|^2}{2\eta_t}$ (for a detailed explanation of this step, please refer to how to bound term II on page 13 of the paper) and $(b)$ is due to Cauchy-Schwarz inequality and AM-GM inequality to get $\langle g^t,x^t-x^{t+1}\rangle\leq \\|g^t\\|\\|x^t-x^{t+1}\\|\leq \frac{\\|x^{t+1}-x^t\\|^2}{2\eta_t}+\frac{\eta_t\\|g^t\\|^2}{2}$. 

Note that when $h=0$ and $\\|g^t\\|\leq G$, one can follow [1] to finish the following proof. However, in our setting, there will be two problems.

1)  The term of $\\|g^t\\|^2$ can not be bounded since we are considering the general $(L,M)$-smoothness assumption. Even for $M=0$ (i.e., the standard smooth case), $\\|g^t\\|^2$ does not admit a uniform bound.

2) The term $\eta_t(h(x^t)-h(x^{t+1}))$ can not be bounded. Note that it cannot be telescoping summed either since we need to multiply both sides by $v_t$ (or $v_t^2$ if we follow [1] rigorously) before summing from $t=1$ to $T$.

Due to these two reasons, we start our proof from the $(L,M)$-smoothness as mentioned at the beginning of the proof of Lemma 4.1 on page 13 instead of employing convexity directly. This difference from [1] is very important to make our proof be able to work.

**Compared with [2]**: The main difference is in how to find $w_t$. That is saying that if we follow the way in [2], then one can find the following definition: $$w_t\coloneqq\left(\sum_{s=2}^{t}\frac{2\gamma_s\eta_sv_s\sigma^2}{1-\mu_f\eta_s}+\sum_{s=1}^T2\gamma_s\eta_sv_s\sigma^2\right)^{-1},\forall t\in[T].$$ However, recalling that $v_t$ is defined as $v_t\coloneqq\frac{w_T\gamma_T}{\sum_{s=t}^Tw_s\gamma_s},\forall t\in[T]$ and $v_0\coloneqq v_1$, this means that we need $w_t$ to define $v_t$ and also need $v_t$ to define $w_t$, which is like the problem of "Chicken or the egg". 

A potential way to deal with this issue is to plug $v_t$ into the above definition of $w_t$ (or do it reversely) to obtain a set of equations w.r.t. $w_t,\forall t\in[T]$ and prove that there exists a sequence of $w_t,\forall t\in[T]$ satisfying the corresponding equations. However, this approach is rather complicated. 

Instead, what we did in the paper is to use a new proxy sequence $\bar{v}_t\coloneqq\frac{\gamma_T}{\sum\_{s=t}^T\gamma_s},\forall t\in[T]$ and $\bar{v}_0\coloneqq\bar{v}_1$ to define $$w_t\coloneqq\left(\sum\_{s=2}^t\frac{2\gamma_s\eta_s\bar{v}_s\sigma^2}{1-\mu_f\eta_s}+\sum\_{s=1}^T2\gamma_s\eta_s\bar{v}_s\sigma^2\right)^{-1},\forall t\in[T].$$ As the reviewer can see, this difference helps us get rid of the above problem and makes the proof relatively simple. 

Moreover, a paragraph for [2] has been added in Section 2.1 following the reviewer's advice in the second revision.

---
**References**

[1] Zamani et al. Exact convergence rate of the last iterate in subgradient methods. arXiv preprint arXiv:2307.11134.

[2] Liu et al. High probability convergence of stochastic gradient methods. ICML 2023.

Reviewer comment (Novelty):
Before Sec. 4, the authors claim results are "entirely new," which is unnecessary. Use "to our best knowledge" and avoid exaggerated terms such as "brand new analysis." It is unclear whether this refers to Zamani and Glineur, 2023 or this paper.

Author response:
**Q2**: Before Sec. 4 the authors say "to the best knowledge, our results are entirely new and the first to prove ...." $\cdots$.

**A2**: These two sentences have been modified as the reviewer suggested.

Reviewer comment (Writing):
The authors’ use of notation such as $\eta_{t\in [T]} \leq \frac{1}{2L \vee \mu_f}$ is convoluted and non-standard. Use simpler, standard notations like $\eta_t \leq \frac{1}{2L \vee \mu_f}$ for $t \in [T]$.

Author response:
**Q4**: Throughout, the authors use notation such as (this one is for example from Lemma 4.1 but it is like this all around the paper and the proofs) $\eta_{t\in[T]}\leq\frac{1}{2L\lor\mu_f}$ $\cdots$.

**A4**: The notations have been changed throughout the paper accordingly.

Reviewer comment (Novelty):
Clearly describe how Lemma 4.2 improves over Orabona, 2020, given that Bregman and smooth cases were left as exercises in that work.

Author response:
**Q3&6&14&15**: We give a detailed comparison with [1] in this answer. Let us first recall the key lemma in [1]:

>**Lemma 1  in [1]**:  Let $\eta_t>0,\forall t\in[T]$ be a non-increasing sequence and $q_t\geq0,\forall t\in[T]$. Then $\eta_Tq_T\leq\frac{1}{T}\sum_{t=1}^T\eta_tq_t+\sum_{k=1}^{T-1}\frac{1}{k(k+1)}\sum_{t=T-k+1}^{T}\eta_t(q_t-q_{T-k})$.

Now, we list the differences:

**Can [1] be extended to smooth or/and composite case straightforwardly?** Our answer is **no** [...] (full content from R7 and R8)

Reviewer comment (Writing):
The authors sometimes cast doubt on claims in Harvey et al., 2019a without verifying with the original authors. For example, questioning the extension to sub-Gaussian noises can be clarified with direct communication. The current phrasing is not ideal and distracts from the main contribution.

Author response:
**Q7**: Many places, such as the footnote in page 6 says that "Harvey et al. (2019a) claims their proof can extend do sub-Gaussian noises ...." $\cdots$.

**A7**: Thanks for the suggestion! In the second revision, we have further clarified the related discussion when talking about the assumptions on bounded noises and compact domains. Instead of saying the existing works rely on both these assumptions, we restate it as the existing works rely on one of these two assumptions. Consequently, we remove the previous footnote/sentence about [1].

Reviewer comment (Writing):
Page 3 mentions "smoothness with respect to optimum"—this phrase should be clarified.

Author response:
>**Q9**: Page 3 mentions "smoothness with respect to optimum", what do the authors refer to by this phrase?

**A9**: It means $f(x)-f(x^{\*})\leq\frac{L}{2}\\|x-x^{\*}\\|^2, \forall x\in\mathcal{X}$ for some $L>0$ (the original definition can be found in Eq. (2) in [1]. Here we state it in our notations). With this extra assumption, [1] transferred the well-known expected rate $\mathbb{E}\left[\\|x^{T}-x^{*}\\|\right]=O(\frac{1}{T})$  for the Lipschitz and strongly convex functions into the convergence w.r.t. the function value gap. However, this smoothness with respect to optimum condition may not hold in general. We have added a footnote to clearly state the definition in the first revision.

Reviewer comment (Writing):
On page 4, the authors could use $\mathcal{X}=\mathbb{R}^d$ and standard domain definitions for $f,g$ to avoid overly complicated discussions.

Author response:
>**Q10**: page 4, you can just use $\mathcal{X}=\mathbb{R}^{d}$ and define the domains of $f,g$ in standard ways $\cdots$.

**A10**: We refer the reviewer to the answer **A17** for why we consider $\mathcal{X}$ to be a general closed convex set instead of setting it to be $\mathbb{R}^{d}$ directly.

Reviewer comment (Theory):
On page 4 and elsewhere, the authors refer to the book Lan, 2020 without specifying sections or results. Specific references should be provided.

Author response:
**Q11**: page 4 and many places of the paper refer to the book Lan, 2020 for some results $\cdots$.

**A11**: We have clarified them following the reviewer's advice.

Reviewer comment (Presentation):
Section 2.1 is unnecessary in the main text and should be moved to the appendix. The sentence starting with "We also would like to emphasize that ..." reads awkwardly and should be rewritten.

Author response:
**Q12**: Sec 2.1 is unnecessary and should be moved to the appendix $\cdots$.

**A12**: We put it there mainly to connect and contrast the two common convergence criteria. For optimization experts, this is certainly redundant. However, since ICLR has a broad audience, some may not be very familiar with the concepts. We will make the paragraph as concise as possible so that it doesn't take much space. Besides, we have rephrased the corresponding sentence following the reviewer's advice.

Reviewer comment (Theory):
Theorem 3.1’s step size choice depends on $x^*$, which is undesirable. If the result works with $\eta$ independent of $x^*$, it should be written that way.

Author response:
>**Q13**: Theorem 3.1 has a step size choice depending on $x^*$, this is not good and should be avoided $\cdots$.

**A13**: Sorry for the confusion. Here we mean that the algorithm can still guarantee the convergence of the last iterate for any $\eta>0$ rather than any $\eta>0$ leads to the same convergence rate as in Theorem 3.1. In other words, even if we don't know the Bregman divergence w.r.t. $x^*$, the last iterate still provably converges. But with $\eta_*$ (which depends on the knowledge of $M$,$\sigma$ and $D_{\psi}(x^*,x^1))$, we will get better dependence on the parameters in the final rate. We refer the reviewer to Theorem C.1 for a detailed result. The discussion here has been clarified in the first revision.

Moreover, we would like to keep the current form of Theorem 3.1 since the full version (Theorem C.1) is rather long but we only have limited space for the main text.

Reviewer comment (Theory):
The discussion after Theorem 3.1 claims "Orabona 2020 cannot be applied to composite optimization" without explanation. This should be clarified or rephrased.

Author response:
**Q3&6&14&15**: We give a detailed comparison with [1] in this answer. Let us first recall the key lemma in [1]: [...] (full R7-R8 content)

Reviewer comment (Novelty):
On page 5, the claim that the results are the first improvement over Moulines and Bach, 2011 for smooth problems should be reconsidered, particularly since Orabona, 2020 left this extension as an exercise. If the authors think otherwise, they should explain why.

Author response:
**Q3&6&14&15**: We give a detailed comparison with [1] in this answer. Let us first recall the key lemma in [1]: [...] (full R7-R8 content)

Reviewer comment (Theory):
Before Theorem 3.2, the claim about recovering $L/T$ rate in the noiseless case should be clarified, especially if it requires $\eta$ depending on $x^*$.

Author response:
>**Q16**: Before Theorem 3.2, the authors mention recovering rate $L/T$ in the noiseless case $\cdots$.

**A16**: From Theorem C.1, we have the following convergence rate for smooth functions (taking $M=0$) with the step size $\eta_t=\frac{1}{2L}\land\frac{\eta}{\sqrt{t}}$ where $\eta>0$ can be any number, $\mathbb{E}\left[F(x^{T+1})-F(x^*)\right]\leq O\left(\frac{LD_{\psi}(x^*,x^1)}{T}+\frac{1}{\sqrt{T}}\left(\frac{D_{\psi}(x^*,x^1)}{\eta}+\eta\sigma^2\log T\right)\right)$. [...]

Reviewer comment (Theory):
The strongly convex case is problematic since strongly convex and bounded gradient assumptions contradict each other. The significance of such a case should be explained separately from the main contribution.

Author response:
>**Q17**: The case of strongly convex is strange since in this case the bounded gradient assumption $\cdots$.

**A17**: We do not think the strong convexity contradicts the bounded gradient assumption due to the following reasons. [...]

Reviewer comment (Writing):
The term "last-iterative convergence" is incorrect and should be replaced with "last-iterate convergence" throughout the paper.

Author response:
**Q18**: Paper uses the phrase at many places (including the title) "last-iterative convergence" $\cdots$.

**A18**: Thanks for pointing it out. They have been all changed according to the suggestion.

Reviewer comment (Theory):
Explain the 5th line in the chain of equations in the beginning of the proof of Lemma 2.1.

Author response:
>**Q19**: Please explain the 5th line in the chain of equations in the beginning of the proof of Lemma 2.1.

**A19**: From the 4th line to the 5th line, what we did was relabelling the index $k$ and aggregating the same $k$ in the summation. [...]

Reviewer comment (Writing):
After eq. (7), when using convexity of $F$ with $z^t$, properly express $z^t$ as a convex combination for the reader’s understanding, and recall that $v_s$ is monotone and weights sum to $1$.

Author response:
**Q22**: After eq. (7), when authors use convexity of $F$ with $z^t$, $z^t$ is not written in a proper way $\cdots$.

**A22**: The explanation is added following the reviewer's suggestion.

Reviewer comment (Theory):
In the last inequality chain on page 15, explain the last step.

Author response:
>**Q23**: Last inequality chain in page 15, please explain the last step.

**A23**: What we did was relabelling the index t, changing the order of the summation and aggregating the same t in the summation. [...]

Reviewer comment (Theory):
On page 17, explain the first inequality in the estimate of $2w_t\gamma_t \eta_t v_t$ and refer to $\bar v$ for readability.

Author response:
>**Q24**: Page 17, the estimation of $2w_t\gamma_t\eta_tv_t$, please explain the first inequality $\cdots$.

**A24**: The inequality holds due to $\sum_{s=2}^t\frac{2\gamma_s\eta_s\bar{v}_s\sigma^2}{1-\mu_f\eta_s}\geq0,\forall t\in[T]$ and $\sum\_{s=1}^T2\gamma_s\eta_s\bar{v}_s\sigma^2\geq2\gamma_t\eta_t\bar{v}_t\sigma^2$. [...]

Reviewer comment (Writing):
Beginning of page 19: $w_1$ is undefined and should be properly defined.

Author response:
>**Q26**: Beginning of page 19: $w_1$ is undefined since contains a sum starting from $2$ $\cdots$.

**A26**: We believe $w_1$ is explicitly well-defined since there is $w_1=\left(\sum\_{s=2}^1\frac{2\gamma\_s\eta\_s\bar{v}\_s\sigma^2}{1-\mu\_f\eta\_s}+\sum\_{s=1}^T2\gamma\_s\eta\_s\bar{v}\_s\sigma^2\right)^{-1}=\left(\sum\_{s=1}^T2\gamma\_s\eta\_s\bar{v}\_s\sigma^2\right)^{-1}$ [...]

Reviewer comment (Theory):
On page 19, explain the bound of $w_1/w_T$.

Author response:
>**Q27**: Page 19, bound of $w_1/w_T$, please explain the inequality.

**A27**: Let $\rho\coloneqq\max_{2\leq t\leq T}\frac{1}{1-\mu_{f}\eta_{t}}$. Clearly, there is $\rho\geq\frac{1}{1-\mu_f\eta_s},\forall2\leq s\leq T$. [...]

Reviewer comment (Novelty):
After Lemma 4.2 and Lemma 4.3, the repeated emphasis on being the "first unified inequality" can feel tedious—novelties should be stated concisely without overselling.

Author response:
**Q28**: After Lemma 4.2, the authors again say "first unified inequality for the in-expectation" and after Lemma 4.3 $\cdots$.

**A28**: We have rephrased these two sentences to make them more concise. Please let us know if the reviewer still does not think they are proper.