Paper: A Coefficient Makes SVRG Effective (ICLR-2024)

Reviewer comment (Theory):
Section 3 introduces ``optimal coefficient’’ by minimizing the sum of variances for each component of $\bf g^t$. However, this formulation does not consider the potential correlation between different components of $\bf g^t$. In other words, it implicitly assumes the components of $\bf g^t$ are uncorrelated, which looks too strong.

Author response:
We sincerely thank you for your constructive comments. We are encouraged that you find our topic is important and our paper is easy to follow and presents sufficient numerical experiments to support $\alpha$-SVRG. We would like to address the comments and questions below.
> Can you design some strategy to improve SVRG by considering the correlation between the components of the gradient estimator?

1. **We have provided a detailed rigorous proof that considers the correlation between the components of the gradient estimator.** If we consider the correlation, we need to add a matrix $A^t$ instead of a coefficient to control the variance reduction term in SVRG. For notation, $A^t$ denotes the coefficient matrix; $(a^t_k)^T$ represents the row of the coefficient matrix; $a^t_{k, n}$ as the entry of the coefficient matrix at the $k$-th row and $n$-th column; and $K$ represents the covariance matrix. Formally, the generalization of SVRG using the coefficient matrix can be written as:
\begin{flalign}
     g_i^t = \nabla f_i(\theta^t)-A^t(\nabla f_i(\widetilde{\theta})-\nabla f(\widetilde{\theta}))\end{flalign}
We adopt the same gradient variance definition in Section3 and aim to determine the optimal coefficient matrix \(A^{t*}\) that minimizes it at each iteration. 
\begin{flalign}
     min_{A^t} \sum_{k=1}^d Var(g^t_{\cdot, k}) =\sum_{k=1}^d min_{a^t_k} Var(g^t_{\cdot, k})\end{flalign}
The order of minimization and summation can be switched because the variance of the $k$-th component of the gradient only depends on the $k$-th row of the coefficient matrix.
We then expand the variance reduced gradient with the definition of the variance:
\begin{flalign}
     &=\sum_{k=1}^d min_{a^t_k} Var(\nabla f_{i, k}(\theta^t)-(a^t_k)^T\nabla f_i(\widetilde{\theta})) \\\\
     &= \sum_{k=1}^d min_{a^t_k} Var(\nabla f_{i, k}(\theta^t)-\sum_{n=1}^da^t_{k, n}\nabla f_{i, n}(\widetilde{\theta})) \\\\
     &= \sum_{k=1}^d min_{a^t_k} \bigg(Var(\nabla f_{i, k}(\theta^t)) + \sum_{n=1}^d(a^t_{k, n})^2Var(\nabla f_{i, n}(\widetilde{\theta})) - 2\sum_{n=1}^da^t_{k, n}Cov(\nabla f_{i, k}(\theta^t), \nabla f_{i, n}(\widetilde{\theta})) + 2\sum_{n=1}^d\sum_{m\neq n}a^t_{k, n}a^t_{k, m}Cov(\nabla f_{i, m}(\widetilde{\theta}), \nabla f_{i, n}(\widetilde{\theta}))\bigg)
\end{flalign}
Differentiating the objective with respect to $a^t_{k, n}$, we can determine the optimal matrix $A^{t*}$ satisfies:
\begin{flalign}
    &\forall k, n:  2a^t_{k, n}Var(\nabla f_{i, n}(\widetilde{\theta})) - 2Cov(\nabla f_{i, k}(\theta^t), \nabla f_{i, n}(\widetilde{\theta})) + 2\sum_{m\neq n}a^t_{k, m}Cov(\nabla f_{i, m}(\widetilde{\theta}), \nabla f_{i, n}(\widetilde{\theta})) = 0 \\\\      &\implies A^{t*} = K_{\nabla f_i(\theta^t), \nabla f_i(\widetilde{\theta})}K_{\nabla f_i(\widetilde{\theta}), \nabla f_i(\widetilde{\theta})}^{-1}
\end{flalign}
This full derivation is also included in Appendix G of the revision.

2. **Using the optimal coefficient (the diagonal entries of the optimal matrix) is a more feasible choice for empirical analysis.** For instance, a logistic regression model taking a 32x32 RGB image and outputting 10 classes has an optimal matrix with $3082^2$ entries, which cannot be easily accommodated by any modern GPU. This has similar implications in second-order optimization. Computing the Hessian matrix in deep learning is often computationally intractable, but it is possible to estimate the diagonal entries of the Hessian matrix [1], making the number of elements only scale linearly with the number of parameters in the model.

Reviewer comment (Theory):
The optimal coefficient only considers the current variable $\theta^t$, but it is unclear how it affects the convergence rate of the algorithm in theoretical. The existing analysis only provide a greedy view, while I am more interested in the global theoretical guarantees of proposed algorithms.

Author response:
> The optimal coefficient only considers the current variable $\theta^t$, but it is unclear how it affects the convergence rate of the algorithm in theoretical. The existing analysis only provides a greedy view, while I am more interested in the global theoretical guarantees of proposed algorithms. Can you provide convergence analysis to show the advantage of proposed method?

Our primary aim with the development of $\alpha$-SVRG is to enhance SVRG for practical applications, particularly in training neural networks. **It's crucial to note that while theoretical analyses in SVRG offer valuable insights [2, 3], we should clarify there is a gap between theoretical analysis and empirical results in SVRG. Our work is to bridge this gap by providing a deeper understanding of why SVRG may not work in practice, and how we can adapt it to better suit real-world applications.**

1. Theoretically, SVRG has shown faster convergence than SGD in non-convex settings [2, 3]. **However, empirical studies, including our own, reveal that SVRG does not effectively reduce gradient variance or achieve lower training losses compared to SGD.** This observation holds even for simple models like MLP-4 or LeNet in non-convex settings [4].

2. Theoretically, SVRG has the same convergence rate even if the number of iterations between two consecutive snapshot scales with the number of data points [3]. **Yet, empirically, as the model moves away from the snapshot, the correlation between the snapshot gradients and the model gradients decreases, rendering the variance reduction mechanism in SVRG failed.**

3. Theoretically, the variance of the gradient estimator in SVRG is upper bounded by the iterated distance between the snapshot model and the current model up to a constant factor [2]. **But, the work [4] shows the model iterate "moves too quickly through the optimization landscape," undermining the utility of the variance upper bound.**

Reviewer comment (Experiments):
It is well-known that stochastic recursive gradient methods has the optimal stochastic first-order oracle (SFO) complexity for nonconvex optimization. For example, SVRG can find $\epsilon$-stationary point within $n+n^{2/3}\epsilon^{-2}$ SFO calls (Allen-Zhu & Hazan, 2016; Reddi et al., 2016), while SPIDER only requires $n+n^{1/2}\epsilon^{-2}$ (Fang et al., 2018), where $n$ is the number of individual functions. Compared with SVRG, the study on SPIDER for training neural networks is more interesting.

Author response:
> It is well-known that stochastic recursive gradient methods have the optimal stochastic first-order oracle (SFO) complexity for nonconvex optimization. For example, SVRG can find $\epsilon$-stationary point within $n+n^\frac{2}{3}\epsilon^{-2}$ SFO calls [2, 3], while SPIDER only requires $n+n^\frac{1}{2}\epsilon^{-2}$ [5], where $n$ is the number of individual functions. Compared with SVRG, the study on SPIDER for training neural networks is more interesting.

1. You are absolutely right about the potential of stochastic recursive gradients in training neural networks, **but we found that our current framework based on control variate cannot be applied directly to the family of stochastic recursive gradients**. The main issue is that the variance reduction term is not zero-mean in expectation. We can analyze one of the stochastic recursive gradients methods SpiderBoost as an example. Formally, SpiderBoost defines its variance reduced gradient as:
$$g^{t} = \nabla f_i(\theta^t) - (\nabla f_i(\theta^{t-1}) - g^{t-1}), g^{t_0} = \nabla f(\widetilde{\theta}).$$
If conditioning the expectation on all the randomness from $t_0$ to $t-1$, the expectation of the first term in the variance reduction term becomes:
$$
    \mathbb{E}[\nabla f_i(\theta^{t-1})] = \nabla f(\theta^{t-1}).$$
However, the expectation of $g^{t-1}$ could only be $\nabla f(\theta^{t-1})$ if we condition the expectation on all the randomness from $t_0$ to $t-2$. 

2. **Despite the incompatibility of the control variates method on SpiderBoost, we still find intriguing empirical results when applying a linearly decreasing coefficient to SpiderBoost. We refer to this approach as $\alpha$-SpiderBoost.** We employ $\alpha$-SpiderBoost with three different initial coefficients $\alpha_0$ from \{0.5, 0.75, 1\} to train ConvNeXt-Femto on a variety of small datasets. All the experiments use the same training recipe and the hyper-parameters as before (detailed in Appendix B). We present the result [here](https://drive.google.com/uc?id=1fcf9O_ZhZW25ETFEYHYT6WIIRkVukqKf) and compare it to the baseline, the standard SVRG, $\alpha$-SVRG, and the standard SpiderBoost. 
3. **For small datasets (Pets, STL-10, and DTD), the standard SpiderBoost achieves a lower train loss than the baseline, but $\alpha$-SpiderBoost could have a higher train loss.** Intuitively, for small datasets, the number of iterations within an epoch is small. As a result, the correlation between the model stochastic gradients and the snapshot stochastic gradients is still high enough for SpiderBoost to reduce gradient variance. This might also suggest SpiderBoost does optimize models better than SVRG on small datasets where SVRG consistently increases the train loss. 
4. **However, for relatively large datasets (CIFAR100, Food-101, and SVHN), $\alpha$-SpiderBoost could achieve a substantially lower train loss than the standard SpiderBoost and even $\alpha$-SVRG.** This is likely because as the number of iterations within an epoch increases, the strength of the variance reduction term should decrease and a linearly decreasing coefficient helps weaken the strength. 

The above paradoxical results of $\alpha$-SpiderBoost warrants future research to further understand how to make SpiderBoost more effective in training neural networks. **And our work on $\alpha$-SVRG could be a good starting point for this.** All the above results and analysis have been included in Appendix F.

We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

References:

[1] Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, Anonymous, 2023.\
[2] Variance reduction for faster non-convex optimization. Allen-Zhu & Hazan, 2016.\
[3] Stochastic Variance Reduction for Nonconvex Optimization. Reddi et al., 2016.\
[4] On the ineffectiveness of variance reduced optimization for deep learning. Defazio et al., 2019.\
[5] SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator. Fang, 2018.

Reviewer comment (Novelty):
Can you design some strategy to improve SVRG by considering the correlation between the component of gradient estimator?

Author response:
We sincerely thank you for your constructive comments. We are encouraged that you find our topic is important and our paper is easy to follow and presents sufficient numerical experiments to support $\alpha$-SVRG. We would like to address the comments and questions below.
> Can you design some strategy to improve SVRG by considering the correlation between the components of the gradient estimator?

1. **We have provided a detailed rigorous proof that considers the correlation between the components of the gradient estimator.** If we consider the correlation, we need to add a matrix $A^t$ instead of a coefficient to control the variance reduction term in SVRG. For notation, $A^t$ denotes the coefficient matrix; $(a^t_k)^T$ represents the row of the coefficient matrix; $a^t_{k, n}$ as the entry of the coefficient matrix at the $k$-th row and $n$-th column; and $K$ represents the covariance matrix. Formally, the generalization of SVRG using the coefficient matrix can be written as:
\begin{flalign}
     g_i^t = \nabla f_i(\theta^t)-A^t(\nabla f_i(\widetilde{\theta})-\nabla f(\widetilde{\theta}))\end{flalign}
We adopt the same gradient variance definition in Section3 and aim to determine the optimal coefficient matrix \(A^{t*}\) that minimizes it at each iteration. 
\begin{flalign}
     min_{A^t} \sum_{k=1}^d Var(g^t_{\cdot, k}) =\sum_{k=1}^d min_{a^t_k} Var(g^t_{\cdot, k})\end{flalign}
The order of minimization and summation can be switched because the variance of the $k$-th component of the gradient only depends on the $k$-th row of the coefficient matrix.
We then expand the variance reduced gradient with the definition of the variance:
\begin{flalign}
     &=\sum_{k=1}^d min_{a^t_k} Var(\nabla f_{i, k}(\theta^t)-(a^t_k)^T\nabla f_i(\widetilde{\theta})) \\\\
     &= \sum_{k=1}^d min_{a^t_k} Var(\nabla f_{i, k}(\theta^t)-\sum_{n=1}^da^t_{k, n}\nabla f_{i, n}(\widetilde{\theta})) \\\\
     &= \sum_{k=1}^d min_{a^t_k} \bigg(Var(\nabla f_{i, k}(\theta^t)) + \sum_{n=1}^d(a^t_{k, n})^2Var(\nabla f_{i, n}(\widetilde{\theta})) - 2\sum_{n=1}^da^t_{k, n}Cov(\nabla f_{i, k}(\theta^t), \nabla f_{i, n}(\widetilde{\theta})) + 2\sum_{n=1}^d\sum_{m\neq n}a^t_{k, n}a^t_{k, m}Cov(\nabla f_{i, m}(\widetilde{\theta}), \nabla f_{i, n}(\widetilde{\theta}))\bigg)
\end{flalign}
Differentiating the objective with respect to $a^t_{k, n}$, we can determine the optimal matrix $A^{t*}$ satisfies:
\begin{flalign}
    &\forall k, n:  2a^t_{k, n}Var(\nabla f_{i, n}(\widetilde{\theta})) - 2Cov(\nabla f_{i, k}(\theta^t), \nabla f_{i, n}(\widetilde{\theta})) + 2\sum_{m\neq n}a^t_{k, m}Cov(\nabla f_{i, m}(\widetilde{\theta}), \nabla f_{i, n}(\widetilde{\theta})) = 0 \\\\      &\implies A^{t*} = K_{\nabla f_i(\theta^t), \nabla f_i(\widetilde{\theta})}K_{\nabla f_i(\widetilde{\theta}), \nabla f_i(\widetilde{\theta})}^{-1}
\end{flalign}
This full derivation is also included in Appendix G of the revision.

2. **Using the optimal coefficient (the diagonal entries of the optimal matrix) is a more feasible choice for empirical analysis.** For instance, a logistic regression model taking a 32x32 RGB image and outputting 10 classes has an optimal matrix with $3082^2$ entries, which cannot be easily accommodated by any modern GPU. This has similar implications in second-order optimization. Computing the Hessian matrix in deep learning is often computationally intractable, but it is possible to estimate the diagonal entries of the Hessian matrix [1], making the number of elements only scale linearly with the number of parameters in the model.

Reviewer comment (Theory):
Is it possible to apply the idea of this paper to improve stochastic recursive gradient methods?

Author response:
> It is well-known that stochastic recursive gradient methods have the optimal stochastic first-order oracle (SFO) complexity for nonconvex optimization. For example, SVRG can find $\epsilon$-stationary point within $n+n^\frac{2}{3}\epsilon^{-2}$ SFO calls [2, 3], while SPIDER only requires $n+n^\frac{1}{2}\epsilon^{-2}$ [5], where $n$ is the number of individual functions. Compared with SVRG, the study on SPIDER for training neural networks is more interesting.

1. You are absolutely right about the potential of stochastic recursive gradients in training neural networks, **but we found that our current framework based on control variate cannot be applied directly to the family of stochastic recursive gradients**. The main issue is that the variance reduction term is not zero-mean in expectation. We can analyze one of the stochastic recursive gradients methods SpiderBoost as an example. Formally, SpiderBoost defines its variance reduced gradient as:
$$g^{t} = \nabla f_i(\theta^t) - (\nabla f_i(\theta^{t-1}) - g^{t-1}), g^{t_0} = \nabla f(\widetilde{\theta}).$$
If conditioning the expectation on all the randomness from $t_0$ to $t-1$, the expectation of the first term in the variance reduction term becomes:
$$
    \mathbb{E}[\nabla f_i(\theta^{t-1})] = \nabla f(\theta^{t-1}).$$
However, the expectation of $g^{t-1}$ could only be $\nabla f(\theta^{t-1})$ if we condition the expectation on all the randomness from $t_0$ to $t-2$. 

2. **Despite the incompatibility of the control variates method on SpiderBoost, we still find intriguing empirical results when applying a linearly decreasing coefficient to SpiderBoost. We refer to this approach as $\alpha$-SpiderBoost.** We employ $\alpha$-SpiderBoost with three different initial coefficients $\alpha_0$ from \{0.5, 0.75, 1\} to train ConvNeXt-Femto on a variety of small datasets. All the experiments use the same training recipe and the hyper-parameters as before (detailed in Appendix B). We present the result [here](https://drive.google.com/uc?id=1fcf9O_ZhZW25ETFEYHYT6WIIRkVukqKf) and compare it to the baseline, the standard SVRG, $\alpha$-SVRG, and the standard SpiderBoost. 
3. **For small datasets (Pets, STL-10, and DTD), the standard SpiderBoost achieves a lower train loss than the baseline, but $\alpha$-SpiderBoost could have a higher train loss.** Intuitively, for small datasets, the number of iterations within an epoch is small. As a result, the correlation between the model stochastic gradients and the snapshot stochastic gradients is still high enough for SpiderBoost to reduce gradient variance. This might also suggest SpiderBoost does optimize models better than SVRG on small datasets where SVRG consistently increases the train loss. 
4. **However, for relatively large datasets (CIFAR100, Food-101, and SVHN), $\alpha$-SpiderBoost could achieve a substantially lower train loss than the standard SpiderBoost and even $\alpha$-SVRG.** This is likely because as the number of iterations within an epoch increases, the strength of the variance reduction term should decrease and a linearly decreasing coefficient helps weaken the strength. 

The above paradoxical results of $\alpha$-SpiderBoost warrants future research to further understand how to make SpiderBoost more effective in training neural networks. **And our work on $\alpha$-SVRG could be a good starting point for this.** All the above results and analysis have been included in Appendix F.

We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

References:

[1] Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, Anonymous, 2023.\
[2] Variance reduction for faster non-convex optimization. Allen-Zhu & Hazan, 2016.\
[3] Stochastic Variance Reduction for Nonconvex Optimization. Reddi et al., 2016.\
[4] On the ineffectiveness of variance reduced optimization for deep learning. Defazio et al., 2019.\
[5] SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator. Fang, 2018.

Reviewer comment (Reproducibility):
Additional parametrization (initialization, linear decay) of the proposed $\apha$-SVRG

Author response:
**We emphasize that these additional parametrization are designed to be robust, meaning they do not require extensive tuning in practical settings.** In Table 6 in Appendix B, we assess $\alpha$-SVRG across various datasets with different initial coefficients $\alpha_0 \in\\{0.5,0.75,1\\}$. The results are [here](https://drive.google.com/uc?id=1gZ46eLCA1_8CZZMMArL31lCWDF_ogfQv) and have been included in Appendix C. The consistent reduction in training loss across most datasets, irrespective of the initial coefficient choice, underscores $\alpha$-SVRG's robustness and its effectiveness.

Reviewer comment (Evaluation):
Results of $\apha$-SVRG often very close to the baseline: no clear improvement in most cases (except on Pets, STL-10 and DTD datasets)

Author response:
1. In Appendix E, we have utilized three different random seeds to run $\alpha$-SVRG and compare it to the baseline. The results (below) demonstrate that $\alpha$-SVRG consistently decreases training loss and increases validation accuracy compared to the baseline. Importantly, the mean difference in these metrics for $\alpha$-SVRG is greater than its standard deviation in most cases. **This indicates that the improvement offered by our method is consistent.**

**Train Loss**:
|                    | CIFAR-100              | Pets                  | Flowers               | STL-10                | Food-101              | DTD                   | SVHN                  | EuroSAT               |
|--------------------|------------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| baseline           | 2.645 ± 0.013          | 2.326 ± 0.088         | 2.436 ± 0.038         | 1.660 ± 0.017         | 2.478 ± 0.021         | 2.072 ± 0.066         | 1.583 ± 0.005         | 1.259 ± 0.017         |
| + $\alpha$-SVRG $      | **2.606 ± 0.017**      | **2.060 ± 0.071**     | **2.221 ± 0.042**     | **1.577 ± 0.022**     | **2.426 ± 0.007**     | **1.896 ± 0.075**     | **1.572 ± 0.011**     | **1.239 ± 0.016**     |

**Validation Accuracy:**
|                    | CIFAR-100              | Pets                  | Flowers               | STL-10                | Food-101              | DTD                   | SVHN                  | EuroSAT               |
|--------------------|------------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| baseline           | 81.02 ± 0.07           | 70.61 ± 1.55          | 80.33 ± 1.01          | 80.80 ± 1.46          | 85.29 ± 0.47          | 56.21 ± 1.19          | 94.29 ± 0.67          | 97.91 ± 0.12          |
| + $\alpha$-SVRG $       | **81.07 ± 0.22**       | **76.37 ± 1.06**      | **84.15 ± 1.15**      | **83.65 ± 0.92**      | **85.45 ± 0.43**      | **61.44 ± 0.35**      | **94.94 ± 0.60**      | **98.13 ± 0.07**      |

2. When comparing $\alpha$-SVRG to the baseline, we maintain the same hyperparameters, including regularization strength. While $\alpha$-SVRG effectively lowers training loss, it may lead to overfitting, as indicated by the less significant improvements in validation accuracy in some datasets. This suggests a need for adjusting regularization parameters to enhance model generalization. **We believe this falls outside the scope of $\alpha$-SVRG as an optimization method.** However, it presents an important area for future research, particularly in how optimization and regularization can be co-adapted for better performance.

Reviewer comment (Experiments):
No clear explanation on how $\apha$-SVRG scales to larger datasets: no explanations about the feasibility of taking full batch gradient every epoch (cf mega-batch size of [Defazio & Bottou, 2019])

Author response:
In response to your concern about the feasibility of taking full batch gradients, especially for large datasets like ImageNet21k, we emphasize that it is indeed feasible. **By partitioning the dataset into smaller mini-batches, we can compute gradients for each mini-batch independently. After computing the gradients for each mini-batch, we accumulate them to obtain the full gradient for the entire dataset.**

Reviewer comment (Writing):
$\theta^{past}$ is not a standard notation, I would recommend $\tilde{\theta}$ instead

Author response:
Thank you for pointing out this notation convention. **We have updated the notation for snapshot to $\boldsymbol{\widetilde{\theta}}$ throughout the paper.**

Reviewer comment (Evaluation):
An explanation of the three metrics in Table 1 is required. How are they different?

Author response:
**In the revision, we replace the original mathematical description with an intuitive explanation for each metric**:
Metric 1 captures the directional variance of the gradients; metric 2 sums the variance of gradients across each component; metric 3 focuses on the magnitude of the most significant variation.

Reviewer comment (Writing):
$g_i,k$ in metric 2 of Table 1 is not defined

Author response:
1. **To address this, we have included the mini-batch index $i$ when formulating the variance-reduced gradient in the revision.
$g_i^t = \nabla f_i(\theta^t)-(\nabla f_i(\widetilde{\theta})-\nabla f(\widetilde{\theta}))$.** This change has been applied throughout the paper wherever the variance-reduced gradient is mentioned, including the gradient variance paragraph at the bottom of page 2, where we describe how $g_i^t$ is collected from model checkpoints.

2. **We have added the clarification $g_{i,k}^t$ in the caption of** [Table 1](https://drive.google.com/uc?id=1dMku3LfTmax2wZZBMSzPCjXfR2ZPXn_g)**: $k$ indexes the $k$-th component of gradient $g_{i, k}^t$.**

Reviewer comment (Writing):
The notations $\textbf{g}_{i \cdot}$ is unclear

Author response:
1. **To address this, we have included the mini-batch index $i$ when formulating the variance-reduced gradient in the revision.
$g_i^t = \nabla f_i(\theta^t)-(\nabla f_i(\widetilde{\theta})-\nabla f(\widetilde{\theta}))$.** This change has been applied throughout the paper wherever the variance-reduced gradient is mentioned, including the gradient variance paragraph at the bottom of page 2, where we describe how $g_i^t$ is collected from model checkpoints.

2. **We have added the clarification $g_{i,k}^t$ in the caption of** [Table 1](https://drive.google.com/uc?id=1dMku3LfTmax2wZZBMSzPCjXfR2ZPXn_g)**: $k$ indexes the $k$-th component of gradient $g_{i, k}^t$.**

Reviewer comment (Presentation):
Precising the baseline in the legends (SGD or AdamW) would be better

Author response:
Thank you for pointing this out. **We have made the following improvements in the revision to ensure it is explicitly specified:** 
1. In each plot of the revision, we have changed legend from baseline to SGD or AdamW. 
2. Furthermore, throughout sections 2, 3, and 4, we have specified the baseline optimizer in the main text of each experiment.
3. In section 5, under the training paragraph of the first subsection, we have explicitly mentioned that the base optimizer employed is AdamW.

Reviewer comment (Writing):
The "snapshot interval" is better known as "inner loop size"

Author response:
Thank you for pointing out this notation convention. **We have changed this in section 5.3 of the revision.**

Reviewer comment (Reproducibility):
How large are the mini-batches for $\alpha$-SVRG for the different experiments?

Author response:
**We have included all hyper-parameters, including mini-batch sizes, along with the training recipe, in Appendix B.**
|            | C-100 | Pets | Flowers | STL-10 | Food | DTD  | SVHN | EuroSAT | ImageNet1K |
|------------|-------|------|---------|--------|------|------|------|---------|------|
| batch size | 1024  | 128  | 128     | 128    | 1024 | 128  | 1024 | 512     | 4096 |

Reviewer comment (Theory):
Page 5, is the standard deviation ratio of equation (6) constant across iterations?

Author response:
**Overall, the std ratio is relatively stable over iterations, but the correlation has a similar trend as the optimal coefficient**. The comparison is [here](https://drive.google.com/file/d/1jraqxte7uLsr_4_H1TX94mJv1fvn9Ri_/view?usp=sharing). We have also included this comparison in Appendix D.

Reviewer comment (Evaluation):
Table 3, validation accuracy of SVRG, datasets Pets, STL-10 : have you double checked this results? The accuracy gap is very important. How can this be explained?

Author response:
We have double-checked the results. We have also rerun these experiments with three different seeds. The results are displayed as below:

**Train Loss**:
|                    | CIFAR-100              | Pets                  | Flowers               | STL-10                | Food-101              | DTD                   | SVHN                  | EuroSAT               |
|--------------------|------------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| baseline           | 2.645 ± 0.013          | 2.326 ± 0.088         | 2.436 ± 0.038         | 1.660 ± 0.017         | 2.478 ± 0.021         | 2.072 ± 0.066         | 1.583 ± 0.005         | 1.259 ± 0.017         |
| + $\alpha$-SVRG      | **2.606 ± 0.017**      | **2.060 ± 0.071**     | **2.221 ± 0.042**     | **1.577 ± 0.022**     | **2.426 ± 0.007**     | **1.896 ± 0.075**     | **1.572 ± 0.011**     | **1.239 ± 0.016**     |

**Validation Accuracy:**
|                    | CIFAR-100              | Pets                  | Flowers               | STL-10                | Food-101              | DTD                   | SVHN                  | EuroSAT               |
|--------------------|------------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| baseline           | 81.02 ± 0.07           | 70.61 ± 1.55          | 80.33 ± 1.01          | 80.80 ± 1.46          | 85.29 ± 0.47          | 56.21 ± 1.19          | 94.29 ± 0.67          | 97.91 ± 0.12          |
| + $\alpha$-SVRG       | **81.07 ± 0.22**       | **76.37 ± 1.06**      | **84.15 ± 1.15**      | **83.65 ± 0.92**      | **85.45 ± 0.43**      | **61.44 ± 0.35**      | **94.94 ± 0.60**      | **98.13 ± 0.07**      |

1. **The results show $\boldsymbol{\alpha}$-SVRG’s consistent decrease in train loss and improvement in validation accuracy.**

2. **We believe the significant validation accuracy improvement can be explained by the fact that the models trained on these datasets are still well within the underfitting regime.** From the values of validation accuracy, we can see they still have space to improve. Given that $\alpha$-SVRG accelerates convergence by reducing gradient variance, it can significantly impact the validation accuracy for very underfitting models.

Reviewer comment (Novelty):
To enrich the bibliography, SVRG has also been extended to policy learning ([Papini et al. "Stochastic variance-reduced policy gradient." 2018] & [Du et al. "Stochastic variance reduction methods for policy evaluation." 2017]) other examples are given in [Gower, et al. "Variance-reduced methods for machine learning." 2020]

Author response:
**For the works [7, 8], we have added them at the end of the second paragraph in the introduction. For the work [9], we believe it is a good survey for readers to review related background information about variance reduction methods. Therefore, we add it at the first paragraph in the related works section.**

Reviewer comment (Experiments):
The baseline is a bit unclear in some scenarios. See questions.

Author response:
Regarding the clarity of the baseline in our experiments, **we have made improvements in the revision to ensure it is explicitly specified:** 
1. In each plot of the revision, we have changed legend from baseline to SGD or AdamW. 
2. Furthermore, throughout sections 2, 3, and 4, we have specified the baseline optimizer in the main text of each experiment.
3. In section 5, under the training paragraph of the first subsection, we have explicitly mentioned that the base optimizer employed is AdamW.

Reviewer comment (Writing):
It seems that by saying that $\alpha_t$ decreases linearly, the authors mean that $\alpha_t=O(1/t)$. However, in some literature, a linearly decreasing sequence is a geometric sequence. The authors may want to be more clear about this.

Author response:
We want to clarify that our linearly decreasing schedule for $\alpha$-SVRG is not represented as $\alpha_t=O(1/t)$. **Instead, we use a coefficient that linearly decays across epochs while remaining constant between consecutive snapshots**. For clarity, we decompose the global iteration index $t$ into the epoch-wise index $s$ and the iteration index $i$ within an epoch. We also denote the total training epochs as $T$ and the number of iterations in one epoch $M$. So the value of the default linearly decreasing coefficient at iteration $t$ can be formally written as:
$$\alpha_\text{linear}^t = \alpha_0(1-\frac{s}{T}).$$
Due to the limited space in the main text, we have included this formula in Appendix F of the revision and added a reference to it in the $\alpha$-SVRG paragraph in Section 4. **The reason behind choosing this schedule will also be discussed in response to your next question.**

Reviewer comment (Novelty):
The part where the proposed method is applied to AdamW is a bit unclear. Could the authors guide me to previous works where SVRG is combined with adaptive momentum methods, if there is any? If there is, then the authors may want to refer to these works and build the proposed method based on the previous ones; otherwise it is worth discussing in greater detail how SVRG can be combined with AdamW, as the effect of the variance estimator could be more complicated due to the existence of moments.

Author response:
**The key principle behind combining SVRG with adaptive methods, such as AdamW, is that SVRG only modifies the gradient to reduce its variance and does not update the model weights. The variance reduced gradient is then directly fed into the base optimizer to perform the weights update.** This approach ensures that the core mechanism of SVRG is independent of the baseline optimizer. These conventions are followed from previous works [1, 2]. We have included an additional paragraph on Page 2, above the gradient variance paragraph, to clarify how SVRG can be effectively combined with adaptive methods.

Reviewer comment (Evaluation):
The performance of the proposed method applied to ImageNet-1K is mixed compared to the baseline.

Author response:
For smaller models, a lower training loss usually directly translates to a higher validation accuracy. However, in larger models (Mixer-S, ViT-B, and ConvNeXt-B), while $\boldsymbol{\alpha}$-SVRG lowers the training loss, it pushes the model toward the regime of overfitting, thus lowering the validation accuracy. Additional adjustments to the regularization strength may be required to achieve better generalization. **We believe that addressing the issue of overfitting and fine-tuning regularization parameters may fall outside the immediate scope of $\alpha$-SVRG as an optimization technique.** However, we recognize it is important to understand how optimization methods and regularization strategies can be co-adapted for improved model performance and generalization for future research.

Reviewer comment (Experiments):
Can the authors provide more details about the baseline? In some sections of this work, the baseline is AdamW, but I'm not sure whether this applies to all experiments.

Author response:
Regarding the clarity of the baseline in our experiments, **we have made improvements in the revision to ensure it is explicitly specified:** 
1. In each plot of the revision, we have changed legend from baseline to SGD or AdamW. 
2. Furthermore, throughout sections 2, 3, and 4, we have specified the baseline optimizer in the main text of each experiment.
3. In section 5, under the training paragraph of the first subsection, we have explicitly mentioned that the base optimizer employed is AdamW.

Reviewer comment (Evaluation):
Can the authors explain more about why even the only seems to suppress variance more effectively in the early stages of training, but is less effective in later stages?

Author response:
Assume that you are referring to SVRG. **We attribute the diminishing effectiveness of SVRG in later stages to its constant default coefficient of one.**

1. **The optimal coefficient derived in our paper reduces the variance in SVRG optimally.** The coefficient in SVRG controls the strength of the variance reduction term added to the model's stochastic gradient. This coefficient essentially balances between the unmodified stochastic gradient and a modified gradient with potentially higher variance. A coefficient of 0 means no variance reduction, whereas a very high coefficient value can lead to a significant increase in the variance of the modified gradient. 

2. **The weakening correlation requires a lower optimal coefficient to effectively reduce variance without adding unwanted risk.** As illustrated in Figure 4, the optimal coefficient typically remains well below one. Therefore, using a coefficient value of one throughout the entire training process is far from optimal and poses a risk of introducing additional gradient variance. In addition, as training advances, the weakening correlation between snapshot gradients and model gradients leads to a lower optimal coefficient, further increasing the risk of introducing unwanted variance.