Federated Hypergradient Descent
Anonymous Author(s)
Affiliation
Address email
Abstract
In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure.
We applya principled approach on a method for adaptive client learning rate, number oflocal steps, and batch size.
In our federated learning applications, our primarymotivations are minimizing communication budget as well as local computationalresources in the training pipeline.
Conventionally, hyperparameter tuning methods involve at least some degree of trial-and-error, which is known to be sampleinefficient.
In order to address our motivations, we propose FATHOM (Federated AuTomatic Hyperparameter OptiMization) as a one-shot online procedure.
We investigate the challenges and solutions of deriving analytical gradients withrespect to the hyperparameters of interest.
Our approach is inspired by the factthat all components involved in our training process are open-boxed, and thisfact can be exploited in our algorithm impactfully.
We show that FATHOM ismore communication efficient than Federated Averaging (FedAvg) with optimized,static valued hyperparameters, and is also more computationally efficient overall.
As a communication efficient, one-shot online procedure, FATHOM solves thebottleneck of costly communication and limited local computation, by eliminating a potentially wasteful tuning process, and by optimizing the hyperparamtersadaptively throughout the training procedure without trial-and-error.
We showour numerical results through extensive empirical experiments with the Federated
EMNIST-62 (FEMNIST) and Federated Stack Overflow (FSO) datasets, using
FedJAX as our baseline framework.
Introduction
Federated learning (FL) for on-device applications has its obvious social implications, due to itsinherent privacy-protection feature.
It opens up a broad range of opportunities to allow a massivenumber of devices to collaborate in developing a shared model by retaining private data on thedevices.
The ubiquity of machine learning (ML) on consumer data, coupled with the growth ofprivacy concerns, has pushed researchers and developers to look for new ways to protect and benefitend-users.
In order for FL to deliver its promise in deployed applications, there are still many openchallenges remained to be solved.
We are especially interested in the overall communication efficiencyof the FL pipeline for it to be realistically deployed in a unique communication environment overexpensive links.
To begin, consider a typical step in a machine learning (ML) pipeline: hyperparametertuning.
Whether it is in a centralized, distributed or federated setting, it is an essential step to achievean optimal operation for the training process.
At the heart of a ML training process is the optimizationalgorithm.
In particular, we are interested in using Federated Averaging (FedAvg) as our baseline federated optimization algorithm for our work.
This is because, despite all the recent innovations in
FL since its introduction in 2016 by McMahan et al.
[2016], FedAvg remains the de facto standardin federated optimization for both research and for practice, due to its simplicity and empiricaleffectiveness.
In order for FedAvg to operate effectively, it requires properly tuned hyperparametervalues.
Our work focuses specifically on hyperparameter optimization of: 1) client learning rate, 2) numberof local steps, as well as 3) batch size, for FedAvg.
We propose FATHOM (Federated AuTomatic
Hyperparameter OptiMization), which is an online algorithm that operates as a one-shot procedure.
Conventionally, in offline tuning, hyperparameter tuning is de-coupled from model training.
Whenhyperparameter tuning is done in an online-manner, the two steps are combined and done in a singleprocedure.
FATHOM is an online method, and when integrated with FedAvg, it minimizes the lossfunction by applying gradients with respect to both the parameters and the hyperparameters in everycommunication round.
A performance gain from FATHOM in comparison to FedAvg with alreadyoptimally tuned, static hyperparameters, is a step closer to eliminating the tuning step.
However, weneed to determine if the procedure makes any undesirable trade-off for the gain in communicationefficiency.
As an example, from worst case bounds in Wang et al. [2021], Gorbunov et al. [2020] and Woodworth et al.
[2020], FedAvg can reach optimal ( O ( T − 1 / 3 ) ) asymptotic performance when thenumber of local steps is sufficiently large, where the price may be increased local computation, whichis not always a desirable trade-off.
We will explore how FATHOM can address the above concerns.
These benefits are shown empirically in our numerical results in our FEMNIST and FSO experiments,justify its utility in a FL pipeline over any offline approach involving static valued FedAvg, such asthe ones used in Holly et al.
In the rest of this paper, we will go through a few notable recent state-of-the-art works on this topic,and make justifications for our new approach.
Then we will derive a few key steps for our algorithm,followed by a theoretical convergence bound for adaptive learning rate and number of local stepsin the non-convex regime.
Lastly, we present numerical results on our empirical experiments withneural networks on the FEMNIST and FSO datasets.
Our contributions are as follows:
• We derive gradients with respect to client learning rate and number of local steps for FedAvg,for an online optimization procedure.
We propose FATHOM, a practical one-shot procedurefor joint-optimization of hyperparameters and model parameters, for the federated learningsetting.• We derive a new convergence upper-bound with a relaxed condition, to highlight the benefits
from the extra degree-of-freedom that FATHOM delivers for performance gains.
• We present empirical results that show state-of-the-art performance.
To our knowledge,we are the first to show gain from an online hyperparameter optimization procedure over awell-tuned equivalent procedure with fixed hyperparameter values.
Related Work and Justifications for FATHOM
We explore the question whether the FATHOM approach is justified over the more recent, state-ofthe-art methods that are designed for the same goal: a single-shot online hyperparamter optimizationprocedure for FL, but approached from different principles.
A few notable works have producedmethods belonging in this category of algorithms.
Khodak et al.
[2021] draws inspiration fromweight-sharing in Neural Architectural Search (Pham et al.
[2018], Cai et al.
[2019]), and proposed FedEx, which is an online hyperparameter tuning algorithm that uses exponentiated gradients toupdate hyperparameters.
On the other hand, Mostafa [2019]’s RMAH and Guo et al. [2022]’s Auto-FedRL both use REINFORCE (Williams [1992]) in their reinforcement learning (RL) agentsto update hyperparameters in an online manner, by using relative loss as their trial rewards.
Onebasic assumption among these methods is that at least some of the gradients with respect to thehyperparameters are non-differentiable and unavailable directly.
A generalized technique is used toestimate these quantities by Monte-Carlo sampling, followed by an evaluation step with held-out data,and finally backpropagation by using the re-parametrization trick (Kingma and Welling [2013]).
Onekey benefit with this technique, which is common to the above cited methods, is its generalizabilityfor a wide range of different hyperparameters.
On the other hand, we identify a few areas with these methods that we would like to improve on.
One, information about the internals of the procedurecan and should be exploited.
Two, communication overhead becomes a concern, since sufficient Monte-Carlo sampling is required for the re-parametrization trick to converge (Kingma and Welling[2013]).
From initial observations of their empirical results, while these methods are successful inhyperparameter tuning and reaching target model accuracy as shown in these works, these goals areachieved in unspecified numbers of total communication rounds from works based on RL approachessuch as Mostafa [2019] and Guo et al. Similarly, performance numbers of FedEX from Khodak et al.
[2021] in their online-tuning experiments on the FEMNIST dataset are subpar in theirglobal model test error after 2000 rounds (roughly 20% test error), in comparison to the result after1500 rounds from the same experiment in Reddi et al.
[2020] conducted with tuned hyperparametersfrom brute-force grid search (roughly 15% test error).
The above observations justify exploring our problem differently from previous approaches.
Weexplore a method that exploits our open-box knowledge of the training process, and a methodthat does not require sufficient trials at potential expense of communication budget.
Inspired bythe hypergradient descent techniques developed by Baydin et al.
[2017] and Amid et al.
[2022]for centralized optimization learning rate, we develop FATHOM by directing deriving analyticalgradients with respect to the hyperparameters of interest.
Our work is based on the motivation thatwe assume direct access to gradient information with respect to the hyperparameters.
The result is asample efficient method, FATHOM, which offers both improvements in communication efficiencyand reduced local computation in a single-shot online optimization procedure.
Meanwhile, FATHOMis not as flexibly applicable in optimizing a wide range of hyperparameters, since, as a open-boxapproach, each gradient needs to be derived separately.
We believe this approach is a performanceadvantage, at the expense of its flexibility.
There are other notable works that are related to ours and are worth mentioning.
Charles and Koneˇcný[2020] and Li et al.
[2019] proved that reducing the client learning rate during training is necessary toreach the true objective.
Yet, a line of interesting works (Dai et al.
[2020] and Holly et al.
[2021])applies Bayesian Optimization on federated hyperparamter tuning, by treating it as a closed-boxoptimization problem.
However, they do not consider adaptive hyperparameters.
Yet, another work(Wang and Joshi [2018]) shares similarity to our approach of optimally adapting the number of localsteps, with their adaptive communication strategy, AdaComm, in the distributed setting.
However,their main interest is reducing wall-clock time.
Methodology
In this section we formalize the problem of hyperparameter optimization for FL.
We also review FedAvg, a de facto standard of federated optimization methods for research baseline and in practice.
We also present our method for online-tuning of its hyperparameters, specifically client learningrate, number of local steps, and batch size.
We call our method FATHOM (Federated AuTomatic
Hyperparameter OptiMization).
3.1 Problem Definition
In this paper, we consider the empirical risk minimization (ERM) across all the client data, as anunconstrained optimization problem:
[Equation]
where f i : R d → R is the loss function for data stored in local client index i with d being thedimension of the parameters x , m is number of clients, and f ∗ =
f ( x ∗ ) where x ∗ is a stationarysolution to the ERM problem in eq(1).
To facilitate some of the discussions that follow, it helps to define a few assumptions:
Assumption 1.
(Unbiased Local Gradient Estimator) Let g i ( x ) be the unbiased, local gradientestimator of ∇ f i ( x ) , i.e., E [ g i ( x )]
= ∇ f i ( x ) , ∀ x , and i ∈ [ m ] . Assumption 2. (Convexity with respect to η L and K
3.2 Federated Optimization and Tuning of Hyperparameters
Federated Averaging (FedAvg)
We describe the operations of FedAvg from McMahan et al.[2016], as follows.
At any round t , each of the m clients takes a total of K i local SGD steps, where
K i =
⌊ Eν i /B ⌋ , and where ν i is the number of data samples from client index i , B is batch size,with epoch number E
= 1 being a common baseline.
In this version of FedAvg, heterogeneous datasize is accommodated across clients, and the number of local steps can be manipulated via E and
B as hyperparameters.
Each local SGD step updates the local model parameters of each client i asfollows: x it,k
+
= x
it,k − η L g i ( x it,k ) , where η L is the local learning rate and k ∈ [ K ] is the local stepindex.
To conclude each round, these clients return the local parameters x it,K i to the server whereit updates its global model, with x t +1 =
(cid:80) i ν i x it,K /ν where ν = (cid:80) i ν i .
To facilitate some of thediscussions that follow, we define the following quantities:
[Equation]
where ∆ it ≜ −
[Equation]
i
[Equation]
Offline Hyperparameter Tuning Offline tuning is best to be summarized as follows.
We firstdefine A
= { a ∈ R | a ≥ 0 } with η L ∈ A , and B = { b ∈ I | b ≥ 1 } with K ∈ B .
We also define
C =
A × B , and c = ( η L , K ) , where c ∈ C .
Offline tuning would have the following objective:min c ∈ C f valid ( x, c ) s.t. x = argmin z ∈R d f train ( z, c ) .
With abuse of notation, we use f valid for theobjective function calculated from a validation dataset which is usually held-out before the procedure,and f train for the objective from training data which usually is just local client data.
We also usenotations such as f ( x, c ) to remind us that its solution, x ∗ , depends on the optimization algorithm.
With these clarifications, a few notable offline tuning methods are as follows.
Global grid-searchfrom Holly et al.
[2021] is an example of offline tuning, for example, that iterates over the entiresearch grid defined as C , completing an optimization process for each grid point and evaluatingthe result with a held-out validation set.
Global Bayesian Optimization from Holly et al.
[2021] isanother similar example of offline tuning that follows the same template and objective.
Instead ofbrute-force grid-search, c is sampled from a distribution D C over C , i.e. c ∼ D C , that updates afterevery iteration.
Online Hyperparameter Optimization We are interested in an online procedure that combineshyperparameter optimization and model parameter optimization, with the following objective:
[Equation]
This formulation is the objective of our method, FATHOM, which we will discuss shortly in detail.
Ithas the advantage of joint optimization in a one-shot procedure.
It does not assume the availability ofa validation dataset.
However, when a validation dataset is required in the joint optimization process,such as methods proposed in Khodak et al.
[2021] and Mostafa [2019], then the objective becomes :
[Equation]
3.3 Our Method: FATHOM
In this subsection we will introduce our method, FATHOM (Federated AuTomatic Hyperparameter
OptiMization), and discuss it in detail.
Recall from our joint objective, eq(3), that both the modelparameters, x , and hyperparameters of the optimization algorithm, c , are optimized jointly to minimizeour objective function.
An alternative view is to treat c as part of the parameters being optimized in aclassic formulation, i.e. min
y f ( y ) with y =
( x, c ) .
As previously mentioned, our method is inspiredby hypergradient descent from Baydin et al.
[2017] and by exponentiated gradient from Amid et al.[2022], both proposed for centralized learning rate optimization.
We will present how FATHOMexploits our knowledge of analytical gradients to update client learning rate, number of local steps, aswell as batch size, for an online, one-shot optimization procedure.
3.3.1 Hypergradient for Client Learning Rate
In this section, we derive the hypergradient for client learning rate in a similar fashion as Baydin et al.[2017], with the difference being that they are mainly concerned with the centralized optimization problem, and that we are concerned with the distributed setting where clients take local steps.
Wederive the following hypergradient of the objective function as defined in eq(1), taken with respect tothe learning rate η L,t − 1 such that it can be updated to obtain η L,t :
[Equation]
where ∆ t is the update step for the global model as defined in eq(2), and H t is the normalized updatesimilar to Amid et al.
We also make the approximation: ∇ f ( x t ) = (cid:80) mi =1 ν i
[Equation]
[-k =
 g-]∇ f i ( x t {+) ≈(cid:80) mi =1 ν i ν (cid:80) K − 1 k =
+}
g i ( x i,kt ) , to reduce communication and local memory requirements, since ∆ t isalready part of the original FedAvg algorithm and is readily available.
The resulting hypergradient isa scalar, as expected, and can be used efficiently as part of the update rule for η L , which we will seein Section 3.3.4.
The implementation is communication efficient, since in each round, each clientneeds one extra scalar to send back to the server, and likewise the server needs to broadcast one extrascalar back to the clients.
3.3.2 Hypergradient for Number of Local Steps
Since the number of local steps is an integer, i.e. K = { k ∈ I | k ≥ 1 } , this means f ( x t ) does notexist for non-integer values of K .
We formulate a subgradient as a surrogate of the hypergradient∂f ( x t ) /∂K , as follows.
We will call this a hyper-subgradient.
Theorem 1.
When a piecewise function L t is defined for every value of K 0 [-
-]∈ [ K ] on l , such that0 .
≤ l < 1 . 0 , we claim the following is a subgradient of f ( x t ) at K t
= K 0 :
[Equation]
where l represents the marginal fraction of local steps beyond K 0 .
We leave the proof in the Appendixsection beginning in eq(20).
The result from Theorem 1 is not sufficiently communication-efficient for implementing an updaterule for K .
This is because it would require the quantity g i ( x i,K t − 1 t −) to be communicated fromeach client i to the server.
To save communication, let us reuse what the server has in memory:
[Equation]
x i
[Equation]
[Equation]
If we let:
[Equation]
where eq(9) is the normalized update as in Amid et al.
We claim that eq(8) is a positivelybiased version of eq(6), which has its practical importance due to the fact that the last term in eq(6)from Theorem 1 results in zero-mean, noisy gradients, when the local functions are nearing their localsolutions, when in fact, this is the area where more local work is not needed.
Thus, a positive bias isdesirable to drive the number of local steps down.
This result is also useful from a communicationefficiency perspective in its implementation, because the server has all the components to calculatethis quantity, and would not require additional communication.
3.3.3 Regularization for Number of Local Steps
One of the goals for FATHOM is savings in local computation.
To avoid excessive number of localsteps, we further develop a regularization term for local computation against excessive K , which is aproxy for the hypergradient of the local client functions at the end of each round : ∂f i ( x i,Kt ) /∂K .
Theorem 2.
When a piecewise function J t is defined for every value of K 0 [-
-]∈ [ K ] on l , such that0 .
≤ l < 1 . 0 , we claim the following is a subgradient of (cid:80) mi
{+=
+}
f i ( x i,K t t ) at K t = K 0 :
[Equation] [Equation]
where l represents the marginal fraction of local steps beyond K 0 .
We leave the proof in the Appendixsection beginning in eq(24).
In our algorithm, we use the normalized update based on the following biased proxy, since eq(10)tends to be noisy from g i ( x i,K t t ) .
[Equation]
where G t is the normalized update.
The proxy yields a bias towards smaller number of local steps,which is desirable for reducing local computation.
We use this biased proxy against using a moretypical regularization such as L2 on K to avoid tuning an extra hyperparameter.
3.3.4 Normalized Exponentiated Gradient Updates
For the update rules of the hyperparameters η L (client learning rate) and K (number of client localsteps), we use the normalized exponentiated gradient descent method (EGN) with no momentum,rather than a conventional linear update method such as the additive update of hypergradient descentproposed in Baydin et al.
It is reasonable to use exponentiated gradient (EG) methods forupdates of hyperparameters that are strictly positive in value.
EG methods also enjoy significantlyfaster convergence properties when only a small subset of the dimensions are relevant, according to
Amid et al.
EG methods have been proposed in previous works for a variety of applications (Khodak et al.
Amid et al.
[2022], Li et al.
[2020]), and analyzed in depth (Ghai et al.
[2019]), where its convergencehas been studied and validated (Li and Cevher [2018]).
Recently, Amid et al.
[2022] showed that EGNis the same as the multiplicative update for hypergradient descent proposed in Baydin et al.
[2017],when the approximation exp ( · ) ≈ 1 + · is made.
From our observations, we believe that momentumis not needed for the effectiveness of EGN in our application, as validated in our numerical results.
We also opted-out of adding further complexity such as extra weights and activation functions tomodel the relationships between η L,t and K t , because it would require more samples to optimize andbecause FATHOM is a one-shot procedure.
Furthermore, due to the non-stationary nature of thesevalues, we opt for a simpler scheme for faster performance.
Hence, for the update rule of client learning rate, η L , we have:
[Equation]
(13) where H t is as defined in eq(5).
For local steps K , we observe that it is related to batch size B asfollows.
To accommodate heterogeneity of local dataset sizes among clients, we have number of localdata samples from client i to be ν i .
Then we have K i =
⌊ ν i E/B ⌋ , where E is number of epochs,with E = 1 meaning the entire local dataset for each client to be processed once per round.
We deriveupdate rules for E and B globally, without having to make any changes to our theoretical analysis toaccommodate the heterogeneity of local dataset sizes:
[Equation]
14) and
[Equation]
(15) where N t and G t are defined in eq(9) and eq(12), respectively.
Sample client set S out m client
Set ∆ for do
For client i parallel an unbiased stochastic gradient
For each client i , calculate ϕ i min( g i where ∆ i t − t
For each client i update in parallel its local solution: x i k +1 t
− end
Server calcualtes ν = (cid:80) i
S t ν
i , where ν i is the size of client i dataset.
Server calculates ∆ t
(cid:80) S i (
Server updates global model x t [-+
-]+1 = x
t − ∆ t
Server calculates H t = N t = − ∆ t ∥
∆ t ∥ · ∆ t − 1 ,sm ∥
∆ t − 1 , ∥ modified from eq( and eq(
Server calculates G t ; see eq(
Server updates client learning rate η L,t +1 , epochs, E t +1 , and batch size B t + for the next round; see eq , and eq
Server updates ∆ t,sm =
(1 − α ) ∆ t + α ∆ t − 1 ,sm for the next round end
3.3.5 Client Sampling
We present our method, FATHOM, as shown in Algorithm 1. One practical factor we have notconsidered in our discussions is partial client sampling. For our implementation to handle thestochastic nature of client sampling, the metric ∆ t − 1 for calculating H t in eq(5) and N t in eq(9) ismodified by a smoothing function for noise filtering, i.e. ∆ t,sm = α ∆ t − 1 ,sm +(1 − α )∆ t , which is asingle-pole infinite impulse response filter (Oppenheim and Schafer [2009]Oppenheimer et al.
[2009])with no bias compensation.
We use the notation "sm" for smoothed, and after many experiments, wedecide to use α = 0 .
5 for all of our numerical results.  Theoretical Convergence A standard approach to theoretical analysis of an online convex optimization method such as ours(see Assumption 2), is through analyzing the regret bound (Zinkevich [2003], Khodak et al.
Kingma and Ba [2014], and Mokhtari et al.
Nonetheless, this approach does not tell us theimpact on communication efficiency by the online updates introduced from FATHOM.
Therefore, wetake an alternative approach by extending the guarantees of FedAvg performance (Wang et al.
Reddi et al. [2020], Gorbunov et al.
[2020], Yang et al.
[2021], Li et al.
[2019], etc) to include bothadaptive learning rate and adaptive number of local steps.
We highlight the degree-of-freedom aspectof our result offered by the new relaxed conditions in eq(17), for not imposing any constraint, e.g. adecay function such as that done by Li et al.
[2019] on learning rate, over the course of the process.
We assume the special case in our analysis to have full client participation.
We prove that adaptivelearning rate and adaptive number of local steps does not impact asymptotic convergence, despitethe given relaxed conditions.
Furthermore, the relaxed conditions allow strategies that emphasizedifferent performance metrics in various stages of the optimization process, such as the classicacceleration in earlier rounds followed by reduced local computation towards the asymptote.
4.1 Assumptions
Assumption 3.
(L-Lipschitz Continuous Gradient)
There exists a constant L > 0 , such that∥∇ f i ( x ) − ∇ f i ( y ) ∥
≤ L ∥ x − y ∥ , ∀ x , y ∈ R d , and i ∈ [ m ] .
Assumption 4.
(Bounded Local Variance)
There exist a constant σ L > 0 , such that the variance ofeach local gradient estimator is bounded by E ∥∇ f i ( x ) − g i ( x ) ∥ 2 ≤ σ 2 L , ∀ x , and i ∈ [ m ] .
Assumption 5.
(Bounded Second Moment)
There exists a constant G > 0 , such that E t ∥∇ f i ( x t ) ∥ ≤ G , i ∈ [ m ] , ∀ x t .
4.2 Convergence Results
Theorem 3.
Under Assumptions 1-5 and with full client participation, when FATHOM as shownin Algorithm 1 is used to find a solution x ∗ to the unconstrained problem defined in eq(1), thesequence of outputs { x t } satisfies the following upper-bound, where, with slight abuse of notation,
[Equation]
with the following conditions: η L = min (cid:32)(cid:113)
[Equation]
and η L,t ≤ 1 /L for all t , whereη L ≜ 1 T
T (cid:88) t = η L,t and
K
[Equation]
T
T (cid:88) t =
K t (17) and where
[Equation]
We leave the proof in the Appendix beginning in eq(29).
The values of β 0 , β 1 , β 2 , β 3 , and β 4 are dependent on the relative changes over the adaptive processof these components, according to Chebyshev’s Sum Inequalities (Hardy et al.
A specialcase is when these quantities equal to 1 when both η L,t and K t are constant, which recovers thestandard upperbound for FedAvg from eq(16).
Empirical Evaluation and Numerical Results
We present an empirical evaluation of FATHOM proposed in Section 3 and outlined in Algorithm1.
We conduct extensive simulations of federated learning in character recognition on the federated
EMNIST-62 dataset (FEMNIST) (Cohen et al.
[2017]) with a CNN, and in natural language next-wordprediction on the federated Stack Overflow dataset (FSO) (TensorFlow-Federated-Authors [2019])with a RNN.
For completeness, we defer most of the details of the experiment setup in Appendix Section C.1.
Our choice of datasets, tasks and models, are exactly the same as the "EMNIST CR"task and the "SO NWP" task from Reddi et al.
See Figure 1 and Table 1 and their captionsfor details of the experiment results.
The underlying principle behind these experiments is evaluating the robustness of FATHOM versus
FedAvg under various initial settings, to mirror realistic usage scenarios where the optimal hyperparameter values are unknown.
For FATHOM, we start with the same initial hyperparameter valuesas FedAvg.
The test accuracy progress with respect to communication rounds is shown in Figure 1from these experiments.
We also pick test accuracy targets for the two tasks.
For FEMNIST CR weuse 86% and for FSO NWP we use 23%.
Table 1 shows a table of resource utilization metrics withrespect to reaching these targets in our experiments, highlighting the communication efficiency aswell as reduction in local computation from FATHOM in comparison to FedAvg.
To our knowledge,we are the first to show gain from an online hyperparameter optimization procedure over a well-tunedequivalent procedure with fixed hyperparameter values.
The federated learning simulation framework on which we build our algorithms for our experimentsis FedJAX
(Ro et al.
[2021]) which is under the Apache License.
The server that runs the experimentsis equipped with Nvidia Tesla V100 SXM2 GPUs.
within 1500 rounds for FSO NWP and 2000 rounds for FEMNIST CR, in any of our trials.
LR_0 is initial client learning rate, BS_0 is initial batch size, and NCPR is number of clients per round. All experiments use baseline initial values except where indicated. For clarification, M is used in place for "million", and K for "thousand"
[Table]
Conclusion and Future Work
In this work, we propose FATHOM for adaptive hyperparameters in federated optimization, specifically for FedAvg.
We analyze theoretically and evaluate empirically its potential benefits in convergence behavior as measured in test accuracy, and in reduction of local computations, by automaticallyadapting the three main hyperparameters of FedAvg: client learning rate, and number of local stepsvia epochs and batch size.
Server learning rate online optimization is an area that has not beenaddressed by our method and therefore is a limitation of FATHOM.
However, this area will beconsidered for our future work, perhaps by using FATHOM in conjunction with methods such asthose from Reddi et al. [2020] that specifically address this topic.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] The code is MIT licensed.
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Please refer to Sections and
(c) Did you discuss any potential negative societal impacts of your work?
[No] Notspecifically, but it is alluded to how FL applications have social implications in theintroductory section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please refer toAssumptions 1, 2, 3, 4, and
(b) Did you include complete proofs of all theoretical results?
[Yes] Yes, in the supplemental material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Yes, in the supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] Yes, in the supplemental material.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Yes, see Table
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes, it is mentioned in Section
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, it is mentioned
in Section
(b) Did you mention the license of the assets?
[Yes] Yes, it is mentioned in Section
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes, in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/