Paper: Locally Adaptive Federated Learning (ICLR-2024)

Reviewer comment (Theory):
The connection to the Polyak stepsize and the rationale behind the specific choices of (\gamma_1) and (\gamma_2) in Example 1 could be clarified by referring to the definition in Loizou et al. 2021.

Author response:
For convex and strongly convex functions (such as the ones in Example 1), the optimal local stepsizes can be calculated using the Polyak stepsize (PS). For the above example, we have $f_1(x) = \frac{a}{2}x^2$ and corresponding $\gamma_1^\star = \frac{\frac{a}{2}x_0^2 - 0}{\left(ax_0\right)^2} = \frac{1}{2a}$. Similarly, $f_2(x) = \frac{1}{2}x^2$ and corresponding $\gamma_2^\star = \frac{\frac{1}{2}x_0^2 - 0}{\left(x_0\right)^2} = \frac{1}{2}$. Since this analytical Example considers deterministic gradient descent (GD), we use the deterministic version of the Polyak stepsizie. For Figure 1, we consider the stochastic scenario (SGD) and hence use the SPS stepsize from Loizou et al., 2021.

Reviewer comment (Writing):
The paper should provide a clear definition of ( f^* ) in Eq. 5, addressing whether it refers to the global minimum of the finite sum or average sum of each f_i. They are essentially different.

Author response:
$f^{\star}$ has been defined in the Introduction section at Equation (1), and refers to the global minimum of the finite sum. In the same paragraph, we also clarify that the global minimum $x^{\star}$ and the local minima $x_i^{\star}$ can be different.

Reviewer comment (Writing):
It would be very hard to parse the sentence that \sigma_f^2 is stronger than (zeta_*, sigma_*) but weaker than (zeta, sigma), and it is very hard to connect that to the inequalities in Proposition 1.

Author response:
Proposition 1 makes it clear that $\sigma_f$ is a stronger assumption than ($\zeta_{\star}, \sigma_{\star}$). This is because any existing convergence rate for FedAvg using ($\zeta_{\star}, \sigma_{\star}$), will also hold for $\sigma_f$ due to Proposition 1. Moreover, it should also be obvious that uniformly bounded noise ($\zeta, \sigma$) is a stronger assumption than bounded noise at optimum ($\zeta_{\star}, \sigma_{\star}$). We would try to rephrase this comment in the more understandable way in our revised version.

Reviewer comment (Experiments):
The paper claims to compare the proposed methods with FedADAM, but this comparison is not present in the paper. Including this missing comparison, more extensive comparisons with other non-iid FL papers, would strengthen the paper:  

pFedMe: Personalized Federated Learning with Moreau Envelopes Dinh et al., 2020  
PerFedAvg: Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach Fallah et al., 2020  
APFL: Adaptive Personalized Federated Learning Deng et al., 2020  
Ditto: Fair and Robust Federated Learning Through Personalization Li et al., 2022

Author response:
We do compare our method to FedADAM as claimed in our manuscript, and refer the reviewer to Figure 3(c). For the other experiments we chose to include the comparison to only FedAMS for the sake of brevity, since FedAMS [1] was already shown to empirically beat FedADAM [2]. Nonetheless, we will make sure to add the comparison to FedADAM in the remaining plots for the potential camera-ready version of our paper, to aid better understanding.

Reviewer comment (Novelty):
Algorithm design: in my understanding, FedSPS is mainly an FL version of SPS [Loizou et al., 2021]. This extension is rather standard and the algorithmic novelty is not particularly strong.

Author response:
- (W1) **Algorithm:** As explained in Remark 2, we experimented with various other algorithm design choices for incorporating SPS in FedAvg, such as FedSPS-Global and FedSPS-Normalized which are more complicated than our proposed FedSPS but did not offer any empirical benefits---hence our choice.

Reviewer comment (Evaluation):
Experiments — From Figure 2(a), $\gamma_b=1$ performs the best. From 2(b), the effective stepsizes of $\gamma_b=1$ is very stable. To a large extent, I would say that this is almost a constant learning rate without adaptivity. In contrast, $\gamma_b=5$ really brings adaptive stepsizes because the y-axis jumps a lot through iterations. So it is not very clear to me how 'adaptivity' helps the FL training.

Author response:
- (W3) **Experiments:** Following are detailed answers to questions about experiments:
  - The purpose of Figure 2(b) was to show that adaptivity sets in from $\gamma_b = 1$, even if it is stable. Moreover, the purpose of Figure 2(a) was to show that all values of $\gamma_b$ leads to convergence of FedSPS, while that is not the case for FedAvg. So, our method is less sensitive to $\gamma_b$ than FedAvg is to $\gamma$.

Reviewer comment (Experiments):
Experiments — In Figure 4(b), why does FedAdam perform so poorly (almost diverging)? Non-iid MNIST is a standard setting and an easy task. In the original paper of FedAdam and FedAMS there are also MNIST experiments and their methods performed well. This result does not seem very plausible.

Author response:
- (W3) **Experiments:** Following are detailed answers to questions about experiments:
  - We are aware that in Figure 3(b), FedAMS shows a diverging behavior for non-i.i.d MNIST, but this is the plot obtained from our experiments. Note that the original paper on FedAMS [1] does not show any plot for non-i.i.d. data, and the original paper on FedADAM [2] has experiments for the EMNIST dataset and not MNIST. Nonetheless, we will verify this particular experiment regarding its correctness, for the potential camera-ready version.

Reviewer comment (Writing):
The proposed method is called adaptive FL, but it is different from the commonly noted adaptive methods (e.g., Adam, AMSGrad, etc.) which uses first and second order momentums. FedSPS is more like FedSGD with adaptive stepsizes. For better clarity on the contributions, I suggest that the title could follow [Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence, AISTATS 2021] and include 'Polyak Step-size' and 'SGD'.

Author response:
- (Q1) The term "adaptive methods" has a wide variety of connotations in the optimization literature: one of the earliest successful adaptive methods was AdaGrad, which is essentially just a stepsize. Adaptive methods can but do not necessarily have to involve gradient history and momentum. We chose our title to cater to the broader message that local adaptivity can be useful for Federated Learning.

Reviewer comment (Theory):
How does your analysis extend to the partial participation setting? I suggest adding a brief statement on this for clarity. For FedAdam, [Analysis of Error Feedback in Federated Non-convex Optimization with Biased Compression: Fast Convergence and Partial Participation, ICML 2023] might be a relevant but missing reference.

Author response:
- (Q2) Our proof is based on error feedback analysis, and we can use similar ideas from followup work [3, 4] considering partial participation to extend our convergence analysis to partial client participation setting. However, this seems non-trivial and might be a direction for future work. We shall add a brief statement regarding this, as well as the suggested reference in our potential camera-ready version.

Reviewer comment (Novelty):
The proposed algorithm seems to be a direct extension of Stochastic Polyak step to the federated learning setting. What is the major difficulty of this application?

Author response:
As explained in Remark 2, we experimented with various other algorithm design choices for incorporating SPS in FedAvg, such as FedSPS-Global and FedSPS-Normalized which are more complicated than our proposed FedSPS but did not offer any empirical benefits---hence our choice. Moreover, the primary theoretical challenge in analysis of FedSPS was extending the error-feedback framework (that originally works for equal stepsizes) to work for fully un-coordinated local stepsizes, and we did this for the first time in our work.

Reviewer comment (Theory):
The theoretical analysis to the heterogeneity is not convincing. $\sigma_f^2$ is used as a measure of client heterogeneity in the paper, however, it is just an upper-bound (Proposition 1) of some more classical measure of heterogeneity, which means the proposed measure is weaker. In fact, if $l^*$ is chosen to be 0 (as in the paper), this measure is irrelevant to the heterogeneity.

Author response:
It is indeed true that the notion of optimal objective difference $\sigma_f^2$ is a weaker notion of heterogeneity than the standard ones in Federated Learning (FL). However, one can note that such a quantity appears naturally in the analysis of SPS-type methods, and the purpose of our Proposition 1 was to show that $\sigma_f^2$ is comparable to the standard heterogeneity assumptions in FL.

Reviewer comment (Experiments):
Comparison with more baselines are desired. Although authors claim that "We design the first fully locally adaptive method for federated learning called FedSPS", there are already some local adaptive methods for FL, such as the Local-AMSGrad method cited by the authors. It is desirable to add some comparison with these methods.

Author response:
The existing algorithms for client-side adaptivity, such as Local-AdaAlter and Local-AMSGrad perform some form of aggregation of stepsizes during the communication round. Therefore, for these algorithms all the clients use the same stepsize for a particular round, and these methods are not fully locally adaptive as ours. Nonetheless, we agree with the reviewer that an empirical comparison with these methods would add value to the paper, and will include them in our potential camera-ready version.