Paper: The Implicit Bias of Stochastic AdaGrad-Norm on Separable Data (ICLR-2024)

Reviewer comment (Theory):
Proof of Lemma A.4. Could you please describe in more details, why $\|\nabla^2 g(\theta)\| = \Theta(\|\nabla g(\theta)\|)$ in the neighborhood of the stationary point?

Author response:
We thank the reviewer for the feedback. We are glad to have our contributions and novelties acknowledged. The detailed comments greatly help us to revise our manuscript. Amendments have been made according to the comments. We provide a response to the question below.


> Question/Comment 1: Proof of Lemma A.4. Could you please describe in more details, why $$\\|\nabla^2 g\\|=\Theta(\\|\nabla g\\|)$$ in the neighborhood of the stationary point?

In this problem, the stationary point of the loss function is at infinity, and it is not all infinite points, but rather infinite points along a certain margin vector direction. Therefore, vectors near the stationary point must be a margin vector of the dataset, resulting in a loss function form similar to $e^{-x}$ (x is sufficiently large). Hence, this conclusion can be drawn.

Reviewer comment (Presentation):
Maybe move problem formulation to introduction, because it is rather hard to understand the introduction part, if you are not very familiar with topic.

Author response:
We thank the reviewer for the follow-up comments. We are currently working on moving the formulation to the introduction. We fully agree that the problem needs to be clearly defined before discussing the related works and the contributions. Especially, the contribution part even uses the term $\theta^*/\\|\theta^*\\|$. We will update the manuscript again in a few hours. We wanted to respond to the rest of the questions first.

> Page 2, row 8. Seems like, you forgot $\mathbb{E}$ sign in front of $\frac{\alpha_{0}^{2}}{S_{n}}\big\|\nabla g(\theta_{n},\xi_{n})\big\|^{2}$

The sentence was equivalently "the expectation of $\frac{\alpha_{0}^{2}}{S_n}\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}$" cannot be $\frac{\alpha_{0}^{2}}{S_n}{\mathbb{E}_\xi}_n\Big(\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}\Big)$. We did not put the $\mathbb{E}$ sign before the first term because of the phrase "the expectation of ". The sentence was indeed confusing, and now we have revised it by breaking it into two sentences.

> Same place, you forgot transpose sign

Indeed we forgot the transpose sign. We have fixed that (in the same highlighted sentence).

> Page 5. Maybe remove index $n$ in $\psi_{n,i}$, because it does not depend on $n$ and only makes equations messier.

It was deliberately left as it is. We wanted to explain the reason. In fact, this quantity is only independent of $n$ when $\theta_n$ is a margin vector. However, during training, $\theta_n$ is not initially a margin vector. Therefore, considering the overall update process, it is rigorous to include the $n$ index.

Reviewer comment (Writing):
Page 2. You haven't introduced $f$ and $g$.

Author response:
We wanted to follow up on our revision of the formulation. We now break down the introduction into 5 parts: the beginning part, "Formulation", "Challenges", "Related Works", and "Contributions". The definition of the $\mathcal{L}^{2}$ max-margin solution $\theta^*/\\|\theta^*\\|$ and the loss function $g(\theta)$ are defined in the "Formulation" part of the introduction (highlighted in blue). Up to the point of (1) plus "Formulation", we should have everything needed in the introduction section defined. Because of this change we contracted some definitions in Section 2 so they are not restated (also highlighted in blue).

We also merged two paragraphs in the introduction with the technical contribution part as the "Contributions" paragraph. Now we clearly state the main results before briefly introducing our main techniques in response to the main challenges (highlighted in blue).

We thank the reviewer again for suggesting this edition and we are happy to answer further questions and make further updates :)

Reviewer comment (Theory):
Page 2, row 8. Seems like, you forgot $\mathbb E$ sign in front of $\frac{\alpha_0}{\sqrt S_n} f(\theta_n) \nabla g(\theta_n)$.

Author response:
We thank the reviewer for the follow-up comments. We are currently working on moving the formulation to the introduction. We fully agree that the problem needs to be clearly defined before discussing the related works and the contributions. Especially, the contribution part even uses the term $\theta^*/\\|\theta^*\\|$. We will update the manuscript again in a few hours. We wanted to respond to the rest of the questions first.

> Page 2, row 8. Seems like, you forgot $\mathbb{E}$ sign in front of $\frac{\alpha_{0}^{2}}{S_{n}}\big\|\nabla g(\theta_{n},\xi_{n})\big\|^{2}$

The sentence was equivalently "the expectation of $\frac{\alpha_{0}^{2}}{S_n}\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}$" cannot be $\frac{\alpha_{0}^{2}}{S_n}{\mathbb{E}_\xi}_n\Big(\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}\Big)$. We did not put the $\mathbb{E}$ sign before the first term because of the phrase "the expectation of ". The sentence was indeed confusing, and now we have revised it by breaking it into two sentences.

> Same place, you forgot transpose sign

Indeed we forgot the transpose sign. We have fixed that (in the same highlighted sentence).

> Page 5. Maybe remove index $n$ in $\psi_{n,i}$, because it does not depend on $n$ and only makes equations messier.

It was deliberately left as it is. We wanted to explain the reason. In fact, this quantity is only independent of $n$ when $\theta_n$ is a margin vector. However, during training, $\theta_n$ is not initially a margin vector. Therefore, considering the overall update process, it is rigorous to include the $n$ index.

Reviewer comment (Writing):
Page 5. Maybe remove index $n$ in $\psi_{n,i}$, because it does not depend on $n$ and only makes equations messier.

Author response:
We thank the reviewer for the follow-up comments. We are currently working on moving the formulation to the introduction. We fully agree that the problem needs to be clearly defined before discussing the related works and the contributions. Especially, the contribution part even uses the term $\theta^*/\\|\theta^*\\|$. We will update the manuscript again in a few hours. We wanted to respond to the rest of the questions first.

> Page 2, row 8. Seems like, you forgot $\mathbb{E}$ sign in front of $\frac{\alpha_{0}^{2}}{S_{n}}\big\|\nabla g(\theta_{n},\xi_{n})\big\|^{2}$

The sentence was equivalently "the expectation of $\frac{\alpha_{0}^{2}}{S_n}\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}$" cannot be $\frac{\alpha_{0}^{2}}{S_n}{\mathbb{E}_\xi}_n\Big(\big\\|\nabla g(\theta_n,\xi_n)\big\\|^{2}\Big)$. We did not put the $\mathbb{E}$ sign before the first term because of the phrase "the expectation of ". The sentence was indeed confusing, and now we have revised it by breaking it into two sentences.

> Same place, you forgot transpose sign

Indeed we forgot the transpose sign. We have fixed that (in the same highlighted sentence).

> Page 5. Maybe remove index $n$ in $\psi_{n,i}$, because it does not depend on $n$ and only makes equations messier.

It was deliberately left as it is. We wanted to explain the reason. In fact, this quantity is only independent of $n$ when $\theta_n$ is a margin vector. However, during training, $\theta_n$ is not initially a margin vector. Therefore, considering the overall update process, it is rigorous to include the $n$ index.

Reviewer comment (Theory):
As far as I understand, Assumption 3.1 holds for the standard estimator that samples data points uniformly at random. If this is the case I think it would be better to remove Assumption 3.1 and provide a Lemma stating that the assumption is satisfied for the standard estimator.

Author response:
We very much appreciate the suggestion. In fact, by removing Assumption 3.1 and putting the standard estimator into the setting, the presentation is clean and becomes easier to follow. Nevertheless, the noise model we work on is broader than the mini-batch setting. An example is reinforcement learning with bounded rewards, where the gradient is then bounded by a constant that is independent of $g$ [1]. This could potentially broaden the applicable cases of our results.  
For a better presentation, we have revised the manuscript, and added in the beginning of Section 3 the main purpose of the noise model as well as the section. We believe that with your suggestion, the revised presentation could bring up more clarity.

Reviewer comment (Theory):
Are you aware of a first-order method that does not exhibit implicit bias to max-margin solutions in such linear classification problems? In the case there is such a method, I think it would be worth mentioning it so as to emphasize that implicit bias is not a general property of first-order methods.

Author response:
Answer: Implicit bias is indeed not a general property of first-order methods. For example, the authors in [2] have observed that by using mirror descent, it is possible to obtain different L-p max margins.

Reviewer comment (Theory):
The result does not pass an eye test. In [1, 2], the implicit bias of the diagonal variant of AdaGrad depends on various factors such as initialization. Therefore, it is very surprising that the more complex normed version variant would have an implicit bias that does not depend on such conditions.

Author response:
The algorithm we consider in our work is the Adagrad-Norm algorithm, which is different from the AdaGrad-diagonal algorithm. In this algorithm, all random gradient components share the same learning rate $-\frac{\alpha_{0}}{\sqrt{S_{n}}}$ during updates. Therefore, this algorithm, intuitively, is supposed to have the same implicit bias as GD and SGD. We specifically explained this on above Theorem 4.3 the reason AdaGrad-diagonal has a different implicit bias compared to AdaGrad-Norm.

Reviewer comment (Theory):
In the equation block in the "Intuition of the theorem" part of Section 4, it is unclear what the second gradient is taken with respect to. If it is taken with respect to $\theta_n$, then the author clearly forgot to apply the chain rule to $\theta_n / \lVert {\theta_n} \rVert$. Also, the value of $i_n$ is in general not unique.

Author response:
The intuition block, as its name suggests, is just an intuitive understanding, and the expressions used here are not rigorous. For a rigorous derivation using the chain rule, please refer to Equation (41) in the appendix. Additionally, regarding $i_n$, it is indeed not unique. We deliberately skipped this detail in the intuition part. For a rigorous definition of $i_{n}$, please refer to step 2 in the proof sketch of Theorem 4.2, where $i_n$ is rigorously defined. We provide the complete process of the derivative here for your reference.

$$\nabla \Bigg\\|\frac{\theta}{\\|\theta\\|}-\frac{\theta^*}{\\|\theta^*\\|}\Bigg\\|^2=2\bigg(\frac{\theta}{\\|\theta\\|}-\frac{\theta^*}{\\|\theta^*\\|}\bigg)^{\top}\nabla\frac{\theta}{\\|\theta\\|}=2\bigg(\frac{\theta}{\\|\theta\\|}-\frac{\theta^*}{\\|\theta^*\\|}\bigg)^{\top}\frac{\textbf{1}\\|\theta\\|-\frac{\theta\theta^{\top}}{\\|\theta\\|}}{\\|\theta\\|^{2}}=-2\Bigg(\frac{\frac{\theta^*\\|\theta\\|}{\\|\theta^*\\|}-\frac{\theta\theta^{\top}\frac{\theta^*}{\\|\theta^*\\|}}{\\|\theta\\|}}{\\|\theta\\|^{2}\}\Bigg)=2\frac{\theta-\frac{\theta^*\\|\theta\\|}{\\|\theta^*\\|}}{\\|\theta\\|^{2}}-\frac{\theta}{\\|\theta\\|^2}\Bigg\\|\frac{\theta}{\\|\theta\\|}-\frac{\theta^*}{\\|\theta^*\\|}\Bigg\\|^{2}.$$

$$-\Bigg(\nabla \Bigg\\|\frac{\theta}{\\|\theta\\|}-\frac{\theta^*}{\\|\theta^*\\|}\Bigg\\|^2\Bigg)^{\top}\nabla g(\theta)=2\frac{1}{N}\sum_{i=1}^{N}\text{sgn}(y_{i}-0.5)f_{x_{i}}(\theta,x_{i})\Bigg(\frac{{\theta}^{\top}x_{i}-{(\frac{\theta^*}{\\|\theta^*\\|})^\top}x_{i}\\|\theta\\|}{\\|\theta\\|^{2}}-\frac{{\theta}^{\top}x_{i}}{2\\|\theta_{n}\\|^{2}}\bigg\\|\frac{\theta}{\\|\theta\\|}-{\hat{\theta}}^*\bigg\\|^{2}\Bigg)\le 2\frac{1}{N}\sum_{i=1}^{N}\text{sgn}(y_{i}-0.5)f_{x_{i}}(\theta,x_{i})\Bigg(\frac{{\theta}^{\top}x_{i}-{(\frac{\theta^*}{\\|\theta^*\\|})^\top}x_{i}\\|\theta\\|}{\\|\theta\\|^{2}}\Bigg).$$


We wanted to note that we do agree that writing an informal equation, even in the intuition block, is not ideal. In fact, in the revised version, the intuition block specifically states the drop of the gradient chain and refers the reader to the appendix if they are looking for a full, rigorous statement. We thank the reviewer very much for pointing this out.

Reviewer comment (Theory):
In the second equation of the proof of Theorem 4.3, the gradient of $\lVert \theta \rVert^\alpha$ is computed incorrectly, again seeming to be an omission of the chain rule. Also, the dimensions in the middle and right expressions do not match.

Author response:
We did make a mistake in calculating the gradient of $\\|\theta\\|^\alpha$, and we thank the reviewer for pointing this out. Fortunately, this mistake is fixable in a few lines (revised in the updated manuscript) and does not affect the overall derivation of the proof.

Reviewer comment (Writing):
The first two sentences in Section 4 seem to suggest that Theorem 4.1 is original, but it is in fact not. They need to be rephrased to avoid confusing the readers.

Author response:
It has been revised.

Reviewer comment (Writing):
Lemmas A.8 and A.10 were invoked in Section 3 without being stated first. In Lemma A.8, the term "margin vector" was not defined.

Author response:
The term 'margin vector' refers to a vector that satisfies the conditions $\\{\theta\mid\text{sgn}(y_i-0.5)\cdot\theta^{\top}x_i>0\\}$. The definition has been updated in the manuscript.

Reviewer comment (Theory):
The set defined at the top of page 4 is not rigorous. In fact, the global minimum is simply not attainable.

Author response:
The global optimal points lie at infinitely far points in the directions parallel to at least one margin vector. This can be rigorously represented as follows: 
$$\lim_{\\|\theta\\|\rightarrow+\infty,\ \theta/\\|\theta\\|\in S}g(\theta)=0,$$ where $$S:=\\{\alpha\|\\|\alpha\\|=1,\ \text{sgn}(y_i-0.5)\cdot\alpha^{\top}x_i>0\\}.$$