Paper: Risk Bounds of Accelerated SGD for Overparameterized Linear Regression (ICLR-2024)

Reviewer comment (Theory):
Please provide a more detailed explanation of the challenges encountered in theoretical analysis and how to address them in the submission.

Author response:
The most significant challenge is that, different from previous methods analyzing SGD (Zou et al. ,2021; Wu et al. ,2022), the monotonicity of the second moment for the bias error $\mathbf{B}_t$ is violated in ASGD. Therefore, we need to calculate the explicit expression of $\mathbf{A}_i^k$, and analyze quantities like those in Lemma K.5, Lemma K.6 and Lemma K.7.

Another challenge is the identification of the eigenvalue cutoffs $k^\dagger$, $k^\ddagger$, $\hat k$ and $k^*$. This is much more complicated than SGD because $\lambda_i$ significantly affects the decay rate of $\mathbf{A}_i^k$. Specifically, the eigenvalues of $\mathbf{A}_i$ can be complex or real, which also depends on $\lambda_i$. As a comparison, in the SGD results (Zou et al. ,2021; Wu et al. ,2022), there are only two cutoffs for eigenvalue.

Last but not least, we develop a new choice of parameters (Eq. (4.2)) for ASGD in the overparameterized regime. This choice of parameters is obtained through a fine-grained analysis of the effect of fourth-moment (see details in Appendix F.2), where we manage to avoid the model dimension $d$ in our choice of parameters.

Reviewer comment (Experiments):
The experiments are not comprehensive enough, for example, the paper finds that the variance error of ASGD is always larger than that of SGD, but there is not enough experimental support for this claim.

Author response:
Thank you for your suggestion. We have provided additional experiments to justify that the variance error of ASGD is larger than that of SGD. We have also provided additional experiments that compare both the bias error and the variance error of ASGD and SGD for linear regression instances with different spectra of $\mathbf{H}$, including $\lambda_k=k^{-2}$, $\lambda_k=k\log (k+1)$ and $\lambda_k=e^{-k/2}$. In these experiments, we use the same hyperparameters as those used in Section 6. $\mathbf{w}_0$ is initialized as $\mathbf{w}_0=10\cdot\\mathbf{e}\_1$, representing the case where $\mathbf{w}_0-\mathbf{w}^*$ is refined mainly to the subspace of large eigenvalues, or $\mathbf{w}_0=10\cdot\mathbf{e}\_{10}$, representing the case where $\mathbf{w}_0-\mathbf{w}^*$ is refined mainly to the subspace of small eigenvalues. The experimental results are shown in Figures 3-5 in Appendix A. From our experiments, we can observe that the variance error of ASGD is larger than that of SGD, and that the bias error of ASGD is smaller when $\mathbf{w}_0-\mathbf{w}^*$ is refined mainly to the subspace of small eigenvalues. Please refer to Appendix A in the revision for more details.