Paper: Adaptive Federated Learning with Auto-Tuned Clients (ICLR-2024)

Reviewer comment (Novelty):
I believe $\Delta$-SGD is algorithmically similar to the Adaptive SGD method in the paper "Adaptive Gradient Descent without Descent" (Malitsky and Mishchenko, 2020). While there are some attempts to cite and discuss this paper, I think the authors should be more upfront about it, as it changes the perceived contribution entirely.

Author response:
**A:** You are right. $\Delta$-SGD is indeed inspired by [1]. We will make the connection more explicit when we introduce $\Delta$-SGD. However, as we detail in the following answer, extending the proof to a distributed nonconvex setting is nontrivial. Further, any empirical studies on how [1] can be developed and how it works in FL settings were missing from the literature; this is the purpose of this work, among other goals.

Reviewer comment (Theory):
To the best of my knowledge, Malitsky and Mishchenko did not deal with the FL setting, so I believe theorem 1 and its proof are novel. However, I think the authors should provide a proof sketch to help readers understand the nuance of the proving strategy. What is hard/non-trivial about applying Adaptive SGD in the FL setting?

Author response:
**A:** We are glad you asked this question so we can explain in detail the nontriviality of extending proof in [1] to the FL setting. Before delving into this, we first kindly remind you that Theorem 1 is not our only theoretical extension of [1]: 
- As you are aware, in Appendix A.1, we provide the proof of Theorem 1, which is for distributed nonconvex settings and includes stochasticity and local iterations.
- Appendix A.2 provides proof that extends [1] to a distributed setting where $f_i$ is convex.
- Appendix A.3 provides a simple proof in the centralized nonconvex setting.

That being said, there are several challenges in extending the theoretical result in [1] to the FL setting, as we briefly wrote before Assumption 1 in Section 3.1. A primary obstacle is that since $\eta_{t, k}^i$ is prescribed using known quantities, we cannot simply *choose* a *common constant step size* to work for all clients. Our proof has to deal with adaptive step size, which is different across all clients. Moreover, we have to account for the "model drift" between the server and the client model parameters due to local iterations performed by each client, which is trickier to handle on top of stochasticity and the individual adaptive step size we use.

We can add this explanation before introducing Assumption 1 in the revision.

Reviewer comment (Experiments):
Despite the claim that $\Delta$-SGD is complementary to other FL methods that perform server adaptation this is not demonstrated in the experiments.

Author response:
**A:** Thank you for the comment. *We added FedAdam experiments in Appendix B.1* of the updated supplementary material.

Reviewer comment (Experiments):
I think Table 2b should be expanded to become one of the main result (conduct over all datasets, rather than as an ablation study on 2 datasets).

Author response:
**A:** Thank you for the suggestion. *We have added a new table in Appendix B.2* where we have conducted experiments using the FedProx loss function over all datasets, per the suggestion.

Reviewer comment (Evaluation):
The gain in performance is not significant enough in many cases. The authors should provide some error bar or standard deviation.

Author response:
**A:** Thank you for the suggestion! *We have included additional results in Appendix B.3* of the updated supplementary material, plotting and reporting the average and standard deviation of three independent trials for CIFAR-10 classification trained with a ResNet-18.

Reviewer comment (Presentation):
The result presentation is a little bit confusing. The small number on the right of Table 1 is the performance difference from the best result, but what is this "best result" referring to? Best among \alpha = 1.0/ 0.1/ 0.01 ?

Author response:
**A:** *Best result* refers to the highest accuracy within a fixed experiment (including a fixed $\alpha$) among *different client optimizers*. For example, for CIFAR-10 classification trained with a ResNet-18 with $\alpha=1$, the best result is achieved by $\Delta$-SGD with the achieved accuracy 89.8. For this setting, SGD achieves 87.7$_{\downarrow(2.1)}$, and $89.8 = 87.7 + 2.1$. We remind the reviewer that $\alpha$ changes client data heterogeneity (c.f., Figure 7 in Appendix B.6), so different $\alpha$ constitute a different experimental setup.

Reviewer comment (Experiments):
For all methods, the authors performed grid search on CIFAR-10 and apply it on other test scenarios. Why not performing grid search on each individual setting? Is it because of the expensive cost of hyperparameter tuning?

Author response:
**A:** This is precisely why we are proposing $\Delta$-SGD, as it can perform well ***without any additional tuning across different scenarios***. To provide a bit more perspective on how much computing/time effort is needed for step size tuning, let us focus on CIFAR-10 classification trained with a ResNet-18, with a fixed $\alpha$. For this task, SGD takes about 22 seconds per communication round. Since we run 2000 epochs, this amounts to 44,000 seconds, roughly 12 hours. Since we tried four different step sizes, it takes 48 hours *just for SGD*. Now consider adding learning rate decay, on top of that adding momentum, and consider performing grid-search of other client optimizers, not to mention different $\alpha$'s and larger-scale experiments using CIFAR-100 and ResNet-50.

Reviewer comment (Experiments):
If hyperparameter tuning is the bottleneck (and the motivation), how will this method compare to the method FedEx proposed by the paper Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing (Khodak et al., 2021)

Author response:
**A:** Thank you for the reference. While FedEx and $\Delta$-SGD can be loosely connected in the sense of hyperparameter tuning, we clarify that FedEx *requires solving yet another optimization problem (Eq. (6) in [2])*. It is unclear whether solving that optimization problem is computationally more accessible or more challenging than performing a grid search. Our proposed method, $\Delta$-SGD, is fundamentally different because it removes the need for client step size tuning while achieving outstanding performance.

Reviewer comment (Theory):
The bounded gradient assumption can be seen as a strong assumption, and it doesn’t hold for many common functions. Can those terms be bounded using the other two assumptions, bounded variance and dissimilarity?

Author response:
**A:** In general, gradient norm cannot be bounded with bounded variance and function dissimilarity because bounded variance measures the departure of the stochastic gradients $\tilde{\nabla} f_i(\cdot)$ from their deterministic mean $\nabla f_i(\cdot)$, and the function dissimilarity measures the difference in gradients of the local cost functions $\nabla f_i(\cdot)$ and the average cost function $\nabla f(\cdot)$. We also kindly remind the reviewer that bounded gradient is quite a common assumption in non-convex optimization [2, 3, 4, 5], especially when using an adaptive optimizer. Still, removing the bounded gradient assumption can be an interesting future work to improve the theoretical analysis of our proposed method.

Reviewer comment (Writing):
In (4), client $i$ seems to use exact gradients to choose the step size. Shouldn’t it be $\tilde{\nabla}{f_i(x_t^i)}- \tilde{\nabla}{f_i(x_{t-1}^i)}$, or how does client $i$ know true gradient? I guess, it is just a notation thing because there doesn’t seem any problem in Algorithm 1.

Author response:
**A:** Yes, you are right: we used the exact gradients in Eq. (4) for more explicit exposition with simpler notations. As we write right below Eq. (5) (now highlighted in blue color in the updated supplementary material), “For the FL settings, we extend (4) by including stochasticity and local iterations, as summarized in Algorithm 1.”

Reviewer comment (Theory):
I couldn’t get the intuition behind setting the initial step $\eta_{t,0}^i$ to an arbitrary $\eta_0$ at the beginning of each local training after the first one (line 6 in Algorithm 1). Wouldn’t it be good to use the final step size of each client's latest local training?

Author response:
**A:** You are correct that using the final step size of each client’s latest local training would be good. We used an arbitrary $\eta_0$ due to how we implemented the federated learning experiments. Regardless, the reason why arbitrary $\eta_0$ is fine is that, with some reasonable $\eta_0$ (e.g., 0.01), $x_{t, 1}^i$ will be reasonably close to $x_{t, 0}^i$. Then $\eta_{t, 1}^i$ is already a good estimate of the (local) smoothness of $f_i$, only with one local iteration (from $k=0$ to $k=1$).

Reviewer comment (Experiments):
What parameter(s) did you tune for $\Delta-SGD$ except $\gamma$?

Author response:
**A:** *We want to emphasize that we did not tune $\gamma$. As we write right below the pseudocode in pg. 5, $\gamma$ is only needed for Theorem 1.* In practice, we keep the default value $\gamma=2$ from the original implementation; tuning for $\gamma$ might result in even better performance. For other parameters, as we wrote in the paragraph starting with **Hyperparameters** in pg. 6 of the original submission, we append $\delta$ in front of the second condition of $\eta_{t,k}^i$. Still, we use $\delta=0.1$ for all experiments. We also kindly refer the reviewer to Appendix B.3, where we compare the performance of $\Delta$-SGD for different values of $\delta$ to demonstrate the unnecessity of tuning for $\Delta$-SGD.

Reviewer comment (Evaluation):
“This is a counter-intuitive behavior, as one would expect to get better accuracy by using a more powerful model.” → Here, mentioning Adagrad, Adam, and SPS is finetuned on ResNet18 can be useful. I guess, it should be one of the reasons. Also, it would highlight your method doesn’t need any additional tuning.

Author response:
**A:** Thank you! We took your suggestion and modified the supplementary material accordingly (highlighted in blue on pg. 8).

Reviewer comment (Experiments):
In (B) and (C) of Figure 3, have the parameters been tuned on MNIST+CNN and CIFAR-10+ResNet-18, respectively?

Author response:
**A:** No, the parameters are tuned using Figure 1 (A). Then, as the caption reads, “The same step size from (A) is intentionally used in settings (B) and (C).” Similar to our main results in Table 1, the purpose of Figure 1 was to highlight two points: 
$i)$ $\Delta$-SGD works well without any tuning across different datasets, model architectures, and degrees of heterogeneity; 
$ii)$ other optimizers perform suboptimally without additional tuning.

Reviewer comment (Experiments):
You may enrich the study by reporting how the learning rate changes across time and across clients in $\Delta-SGD$. Also, you can compare this with the best learning rate of the other methods.

Author response:
**A:** Thank you for the great suggestion! *We plotted the learning rate conditions of $\Delta$-SGD in Figure 5 of Appendix B.4 of the updated supplementary material*. Our proposed step size adapts to the local smoothness and oscillates between the values as large as 0.3 and as small as $\sim$ 0.0002.

Reviewer comment (Experiments):
The two parts of local learning rate calculation (line 9 in Algorithm 1) are clear in how they are used in the theoretical analysis. It may be good to include an experimental report to see in how many iterations the learning rate is updated based on the first part $\left(\frac{\gamma \lVert x^i_{t, k} - x^i_{t, k-1} \rVert}{2 \lVert \tilde{\nabla} f_i(x^i_{t, k}) - \tilde{\nabla} f_i(x^i_{t, k-1}) \rVert}\right)$ or the second part $\left(\sqrt{1 + \theta^i_{t,k-1}} \eta^i_{t,k-1}\right)$. It may show that the minimum of those two is useful in practice as well.

Author response:
**A:** Again, we appreciate your great suggestion. We took your recommendation and *plotted the first and second parts and their minimum in Figure 5 of Appendix B.4 in the updated supplementary material.* 

From that figure, we can see that *both conditions for $\eta_{t,k}^i$ are necessary.* The first condition, plotted in green, approximates the local smoothness of $f_i,$ but can get quite oscillatory. The second condition, plotted in blue, effectively restricts the first condition from taking too large values. We believe this enriches our paper, as you suggested.

Reviewer comment (Theory):
For the proof of Lemma 2: $\eta_t^{i}$ is used for updating from $x_t$ to $x_{t+1}$, but in the proof it seems like the $\eta_t^{i}$ is used for updating from $x_{t-1}$ to $x_t$. If the latter case is necessary for the proof, how can we get $x_t$ and $\nabla f(x_t)$ without knowing $\eta_t^{i}$.

Author response:
**A**: For simplicity, let us explain with deterministic gradients without local updates in this response. (We will answer about stochasticity in $(ii)$ below.)

It is true that in the algorithm $\eta_t^i$ is used to update $x_{t+1},$ as $x_{t+1}^i = x_t^i - \eta_t^i \nabla f_i (x_t^i).$ 

**In the proof, we are not *updating* $x_t^i$ *from* $x_{t-1}^i$ *using* $\eta_t^i$, but just *utilizing the definition* of $\eta_t^i$, as we clarify below.**

$\eta_t^i$ is defined as:
\begin{align}
    \eta_t^i = \min ( \tfrac{||x_t^i - x_{t-1}^i ||}{2 || \nabla f_i(x_t^i) - \nabla f_i (x_{t-1}^i)  ||},  \sqrt{ 1 + \theta_{t-1}^i } \eta_{t-1}^i ) , \quad \theta_{t-1}^i = \eta_{t-1}^i/\eta_{t-2}^i.
\end{align}
Based on the definition above, we have
\begin{align}
    \eta_t^i \leq \tfrac{||x_t^i - x_{t-1}^i ||}{2 || \nabla f_i(x_t^i) - \nabla f_i (x_{t-1}^i)  ||} \implies 
    || \nabla f_i(x_t^i) - \nabla f_i(x_{t-1}^i) || \leq \frac{1}{2\eta_t^i} ||x_t^i - x_{t-1}^i ||.
\end{align}

We hope the above clarifies the original confusion. That said, we slightly modified the statement and the proof of Lemma 2 in the updated supplementary material, where the source of the initial confusion is removed.

Reviewer comment (Theory):
For the proof of Lemma 2: In the algorithm $\eta_x^{i}$ is based on the gradient of minibatch estimation, how to change them into the true gradient.

Author response:
**A:** We are confused about what you mean by "change to true gradient". We assume by $\eta_x^i$, you mean $\eta_{t,k}^i$ used in Algorithm 1. In the updated supplementary material, we clarified the proof of Lemma 2 for readability, which we also explain below.

Since by Assumptions (1a) (bounded variance) and (1b) (bounded gradient) as well as the independence of samples $z\in\mathcal{B}$, we have that stochastic gradients are bounded:
\begin{align*}
    \mathbb{E} || \tilde{\nabla}f_i(x) ||^2 = \mathbb{E} || \frac{1}{|\mathcal{B}|} \sum_{z\in\mathcal{B}} \nabla F_i(x,z)  ||^2 \leq \mathbb{E} || \frac{1}{|\mathcal{B}|} \sum_{z\in\mathcal{B}} \nabla F_i(x,z) - \nabla f_i(x)  ||^2 + || \nabla f_i(x) ||^2.
\end{align*}

Therefore, as we explain at the beginning of the proof of Lemma 2, Eq. (9) holds in the presence of stochastic gradients, and thus Eq. (12) holds.

Reviewer comment (Theory):
For the proof in A.1: Inequality from (12) to (13) does not seem to be correct. Because $\eta_{t,k}^i$ depends on $\tilde{\nabla} f_i (x_{t,k}^t)$, we need to show that the expected value of $\eta_{t,k}^i \tilde{\nabla} f_i (x_{t,k}^t)$. Since they are not independent, it is trivial to see that $E [\eta_{t,k}^i \tilde{\nabla} f_i (x_{t,k}^t])] = E [\eta_{t,k}^i E[\tilde{\nabla} f_i (x_{t,k}^t)]]$.

Author response:
**A:** We appreciate your question. As stated at the end of page 4, we use the notation
\begin{align*}
    \tilde{\nabla} f_i(x) = \frac{1}{|\mathcal{B}|} \sum_{z\in\mathcal{B}} \nabla F_i(x,z)
\end{align*}
as a shorthand for the stochastic gradients with batch $\mathcal{B}$. When this term appears in the algorithm/proof, we sample a new independent batch with respect to data distribution $\mathcal{D}_i$.

Therefore, the stepsize and stochastic gradient in line (12) of the proof in A.1 are independent:

\begin{align*}
    E_{z \sim D_i} [\eta_{t,k}^i \tilde{\nabla}f_i(x_{t,k}^i)] &= 
    \eta_{t,k}^i E_{z \sim D_i} [\tilde{\nabla}f_i(x_{t,k}^i)] ,
\end{align*}

as $\eta_{t,k}^i$ does not depend on $z,$ but on another independently sampled minibatch.
This is a well-known proof technique that is commonly used in the following works:
- Malitsky, et al (2020). "Adaptive gradient descent without descent." 37th International Conference on Machine Learning 
- Fallah, et al (2020). "Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach." Advances in Neural Information Processing Systems 33 
- Fallah, et al (2020). "On the convergence theory of gradient-based model-agnostic meta-learning algorithms." International Conference on Artificial Intelligence and Statistics.
- Kayaalp, et al (2022). "Dif-MAML: Decentralized multi-agent meta-learning." IEEE Open Journal of Signal Processing 3 (2022): 71-93.

We clarified this part in the updated supplementary material (in blue color).

We hope the above answers clarify your concerns, and if so, we hope you could consider reflecting on the score. Thank you again for your time in reviewing our manuscript, and please don't hesitate to follow up if anything is unclear.

Reviewer comment (Novelty):
Novelty and technical challenge: Client-level optimization is orthogonal to server-level optimization. Therefore, using auto-tuning, which has been studied extensively in the context of centralized computing, in client level is not a very challenging transferral, as we could directly use any auto-tuner directly in client level and check the performance of FL tasks.

Author response:
**A:** We are confused about what you mean by “auto-tuner.” To clarify and help the discussion, assuming by auto-tuning you mean *adaptive methods*, Adam and Adagrad (popular adaptive methods) perform poorly (c.f., Table 1). SPS, a recently proposed adaptive method, also performs quite suboptimally in most cases. We kindly remind the reviewer of two works we cite [1, 2] that study the inefficacy of naively employing adaptive methods like Adagrad to the client optimizer, leading to suboptimal performance (similarly to what we show in Table 1 of our manuscript), and hence required server-side modification. 

The key difference is that our proposed method, *$\Delta$-SGD, performs well across all the settings without additional tuning or modification to the server-side aggregation.*

Reviewer comment (Novelty):
I am wondering whether the auto-tuner, used or partially inspired by any practice in centralized computing. And is there any unique challenge if we simply combine any centralized auto-tuner to FL clients?

Author response:
We would like to point out that all the methods from the references either require users to set/tune multiple hyperparameters or solve multiple subroutines per iteration, as we detail below. 
*Importantly, what we describe below should be done for each client separately, in order for each client to use their own step size, as $\Delta$-SGD does.*

---

> [1] SGDR: Stochastic Gradient Descent with Restarts

This method requires the user to set $\eta_{min}^i, \eta_{max}^i, T_0,$ and $T_{mult}$, for the $i$-th run (Eq (5), pg. 4). 

> [2] Cyclical Learning Rates for Training Neural Networks

This method similarly requires the user to set lower (base_lr) and upper (max_lr) bounds on the learning rate, as well as the frequency to cycle between the lower and the upper bounds (Figure 2, pg. 2)

> [3] A Stagewise Hyperparameter Scheduler to Improve Generalization

This method requires the user to set the "Quasi-Hyperbolic Momentum triplets" {$\alpha_i, \beta_i, v_i$} from $i=1$ to $M$, as well as the training length {$T_i$} from $i=1$ to $M$ (Algorithm 1, pg. 1532), where $M$ is the number of stage (another hyperparameter you have to set.) 

> [4] YellowFin and the Art of Momentum Tuning

This method requires computing four different subroutines per iteration: (i) CURVATURE_RANGE, (ii) VARIANCE, (iii) DISTANCE, and (iv) SINGLE_STEP, as can be seen in Algorithm 1 (pg. 8).

---

We do not have enough time to perform additional experiments to compare to these methods within the discussion period, but we will happily include these references in the related work, and try to add comparison results for the final version.

Reviewer comment (Experiments):
Insufficient comparison with auto-tuner baseline: As I mentioned in the last point, it should not be very challenging to directly deploy auto-tuners to FL clients. However, there is limited comparison to this important baseline, i.e., some representative autotuners developed in centralized computing directly used in FL clients.

Author response:
We would like to point out that all the methods from the references either require users to set/tune multiple hyperparameters or solve multiple subroutines per iteration, as we detail below. 
*Importantly, what we describe below should be done for each client separately, in order for each client to use their own step size, as $\Delta$-SGD does.*

---

> [1] SGDR: Stochastic Gradient Descent with Restarts

This method requires the user to set $\eta_{min}^i, \eta_{max}^i, T_0,$ and $T_{mult}$, for the $i$-th run (Eq (5), pg. 4). 

> [2] Cyclical Learning Rates for Training Neural Networks

This method similarly requires the user to set lower (base_lr) and upper (max_lr) bounds on the learning rate, as well as the frequency to cycle between the lower and the upper bounds (Figure 2, pg. 2)

> [3] A Stagewise Hyperparameter Scheduler to Improve Generalization

This method requires the user to set the "Quasi-Hyperbolic Momentum triplets" {$\alpha_i, \beta_i, v_i$} from $i=1$ to $M$, as well as the training length {$T_i$} from $i=1$ to $M$ (Algorithm 1, pg. 1532), where $M$ is the number of stage (another hyperparameter you have to set.) 

> [4] YellowFin and the Art of Momentum Tuning

This method requires computing four different subroutines per iteration: (i) CURVATURE_RANGE, (ii) VARIANCE, (iii) DISTANCE, and (iv) SINGLE_STEP, as can be seen in Algorithm 1 (pg. 8).

---

We do not have enough time to perform additional experiments to compare to these methods within the discussion period, but we will happily include these references in the related work, and try to add comparison results for the final version.

Reviewer comment (Theory):
Non-standard Assumptions used in theory: bounded gradient is strong assumption, and a bit contradictory to the heterogeneous setting the paper is motivated from. But it may be understandable to use bounded gradient as many existing literature especially who study adaptive optimizers also use it. Strong growth of dissimilarity is less standard and few papers in FL use it. Not sure whether it has any realistic and intuitive correspondence in FL tasks.

Author response:
**A:** As the reviewer acknowledged, bounded gradient is quite a common assumption in nonconvex optimization [3, 4, 5, 6], especially when using an adaptive optimizer. We agree that strong growth of dissimilarity is non-standard, yet $\Delta$-SGD enables each client to use its own step size, which is a significant difference and advantage compared to other FL methods. Therefore, the local iterates can diverge from each other more rapidly (by using a larger step size), which is intuitively why we need a slightly non-standard assumption. We note that relaxing the assumptions can be an interesting future work.

Reviewer comment (Theory):
Though it may be true there is no convergence guarantee given to the varying step size across clients, there are various papers that give convergence guarantee to the scenario where clients can have different number of local iterations, which seems to be a bit related. Is there any connection or difference in proving these two scenarios?

Author response:
**A:** We thank the reviewer for this comment. We do not see a clear connection between our proposed auto-tuned step size selection and existing analyses based on the number of local iterations. *A major difference is that using a different number of local iterations implies computing more gradients, whereas $\Delta$-SGD enables a simple step size calculation without additional computation.* 
Hyperparameters in gradient-based routines might indeed be connected. E.g., some works connect step size selection in stochastic cases with the mini-batch size per gradient calculation. We consider this question an interesting open question to study in the near future, but as of now, it is a bit orthogonal to this submission.