Federated Hypergradient Descent
Anonymous Author(s)
Afﬁliation
Address email
Abstract
In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure.
We applya principled approach on a method for adaptive client learning rate, number oflocal steps, and batch size.
In our federated learning applications, our primarymotivations are minimizing communication budget as well as local computationalresources in the training pipeline.
Conventionally, hyperparameter tuning methods involve at least some degree of trial-and-error, which is known to be sampleinefﬁcient.
In order to address our motivations, we propose FATHOM (Federated
AuTomatic Hyperparameter OptiMization) as a one-shot online procedure.
Weinvestigate the challenges and solutions of deriving analytical gradients with respectto the hyperparameters of interest.
Our approach is inspired by the fact that wehave full knowledge of all components involved in our training process, and thisfact can be exploited in our algorithm impactfully.
We show that FATHOM ismore communication efﬁcient than Federated Averaging (FedAvg) with optimized,static valued hyperparameters, and is also more computationally efﬁcient overall.
As a communication efﬁcient, one-shot online procedure, FATHOM solves thebottleneck of costly communication and limited local computation, by eliminating a potentially wasteful tuning process, and by optimizing the hyperparamtersadaptively throughout the training procedure without trial-and-error.
We showour numerical results through extensive empirical experiments with the Federated
EMNIST-62 (FEMNIST) and Federated Stack Overﬂow (FSO) datasets, using
FedJAX as our baseline framework.
Introduction
Federated learning (FL) for on-device applications has its obvious social implications, due to itsinherent privacy-protection feature.
It opens up a broad range of opportunities to allow a massivenumber of devices to collaborate in developing a shared model by retaining private data on thedevices.
The ubiquity of machine learning (ML) on consumer data, coupled with the growth of privacy concerns, has pushed researchers and developers to look for new ways to protect and beneﬁt end-users
In order for FL to deliver its promise in deployed applications, there are still many open challenges remained to be solved.
We are especially interested in the overall communication efﬁciency of the FL pipeline for it to be realistically deployed in a unique communication environment over expensive links.
To begin, consider a typical step in a machine learning (ML) pipeline: hyperparameter tuning.
Whether it is in a centralized, distributed or federated setting, it is an essential step to achieve an optimal operation for the training process.
At the heart of an ML training process is the optimization algorithm.
In particular, we are interested in using Federated Averaging (FedAvg) as our baseline federated optimization algorithm for our work.
This is because, despite all the recent innovations in
FL since its introduction in 2016 by McMahan et al.
[2016], FedAvg remains the de facto standard infederated optimization for both research and practice, due to its simplicity and empirical effectiveness.
In order for FedAvg to operate effectively, it requires properly tuned hyperparameter values.
Our work focuses speciﬁcally on hyperparameter optimization (HPO) of: 1) client learning rate,2) number of local steps, as well as 3) batch size, for FedAvg.
We propose FATHOM (Federated
AuTomatic Hyperparameter OptiMization), which is an online algorithm that operates as a one-shotprocedure.
In the rest of this paper, we will go through a few notable recent state-of-the-art works onthis topic, and make justiﬁcations for our new approach.
Then we will derive a few key steps for ouralgorithm, followed by a theoretical convergence bound for adaptive learning rate and number of localsteps in the non-convex regime.
Lastly, we present numerical results on our empirical experimentswith neural networks on the FEMNIST and FSO datasets.
Our contributions are as follows:
• We derive gradients with respect to client learning rate and number of local steps for FedAvg,
for an online optimization procedure.
We propose FATHOM, a practical one-shot procedure
for joint-optimization of hyperparameters and model parameters, for FedAvg.
• We derive a new convergence upper-bound with a relaxed condition (see Section 4 and
remark 2), to highlight the beneﬁts from the extra degree-of-freedom that FATHOM delivers
for performance gains.
• We present empirical results that show state-of-the-art performance.
To our knowledge,
we are the ﬁrst to show gain from an online HPO procedure over a well-tuned equivalent
procedure with ﬁxed hyperparameter values.
Related Work and Justiﬁcations for FATHOM
We explore the question whether the FATHOM approach is justiﬁed over the more recent, state-ofthe-art methods that are designed for the same goal: a single-shot online hyperparamter optimizationprocedure for FL.
Zhou et al.
[2022] proposed Federated Loss SuRface Aggregation (FLoRA), ageneral single-shot HPO for FL, which works by treating HPO as a black-box problem and byperforming loss surface aggregation for training the global model.
Khodak et al.
[2021] drawsinspiration from weight-sharing in Neural Architectural Search (Pham et al.
[2018], Cai et al.
[2019]),and proposed FedEx, which is an online hyperparameter tuning algorithm that uses exponentiatedgradients to update hyperparameters.
On the other hand, Mostafa [2019]’s RMAH and Guo et al.[2022]’s Auto-FedRL both use REINFORCE (Williams [1992]) in their agents to update hyperparameters in an online manner, by using relative loss as their trial rewards.
One basic assumption amongthese methods, is that at least some of the gradients with respect to the hyperparameters are unavailable directly.
Generalized techniques are used to update these quantities, involving Monte-Carlosampling and evaluation with held-out data.
One key beneﬁt with techniques such as these is theirgeneralizability for a wide range of different hyperparameters.
On the other hand, we identify a fewareas with these methods that we would like to improve on.
One, information about the internals ofthe procedure can and should be exploited.
Two, communication overhead becomes a concern, sincesufﬁcient Monte-Carlo sampling is required for some of these techniques to converge, an examplebeing the re-parametrization trick (Kingma and Welling [2013]) which is used for FedEx, RMAH and
Auto-FedRL.
From initial observations of their empirical results, while these methods are successfulin hyperparameter tuning and reaching target model accuracy as shown in these works, these goals areachieved in unspeciﬁed numbers of total communication rounds from works based on RL approachessuch as Mostafa [2019] and Guo et al.
The above observations justify exploring our problem differently from previous approaches.
Ourmethod exploits full knowledge of the training process, and it does not require sufﬁcient trialsat potential expense of communication budget.
Inspired by the hypergradient descent techniquesdeveloped by Baydin et al.
[2017] and Amid et al.
[2022] for centralized optimization learning rate,we develop FATHOM by directing deriving analytical gradients with respect to the hyperparametersof interest.
The result is a sample efﬁcient method which offers both improvements in communicationefﬁciency and reduced local computation in a single-shot online optimization procedure.
Meanwhile,
FATHOM is not as ﬂexibly applicable in optimizing a wide range of hyperparameters, since each gradient needs to be derived separately to take advantage of our full knowledge of the training process.
We believe this approach is a performance advantage, at the expense of its ﬂexibility.
There are other notable relevant works.
Charles and Koneˇcn ý [2020] and Li et al.
[2019] proved thatreducing the client learning rate during training is necessary to reach the true objective.
Yet, a line ofinteresting works, such as Dai et al.
[2020] and Holly et al.
[2021]) applies Bayesian Optimization(BO) on federated hyperparamter tuning, by treating it as a closed-box optimization problem.
Daiet al.
[2021] further updates their use of BO in FL by incorporating differential privacy.
However,these BO-based works do not consider adaptive hyperparameters.
Yet, another work (Wang and Joshi[2018]) shares similarity to our approach of optimally adapting the number of local steps, with theiradaptive communication strategy, AdaComm, in the distributed setting.
However, their main interestis reducing wall-clock time.
Lastly, around the same time of this writing, Wang et al.
[2022] publishestheir benchmark suite for FL HPO, called FedHPO-B, which would be valuable to our future work.
Methodology
In this section we formalize the problem of hyperparameter optimization (HPO) for FL.
We ﬁrstreview FedAvg, a de facto standard of federated optimization methods for research baseline andpractice.
Then, we present our method for online-tuning of its hyperparameters, speciﬁcally clientlearning rate, number of local steps, and batch size.
We call our method FATHOM
(Federated
AuTomatic Hyperparameter OptiMization).
3.1 Problem Deﬁnition
In this paper, we consider the empirical risk minimization (ERM) across all the client data, as anunconstrained optimization problem:
[Equation]
where f i : R d !
R is the loss function for data stored in local client index i with d being thedimension of the parameters x , m is number of clients, and f ⇤ =
f ( x ⇤ )
where x ⇤ is a stationarysolution to the ERM problem in eq(1).
To facilitate some of the discussions that follow, it helps to deﬁne assumptions here as we dothroughout the rest of this paper:
Assumption 1.
(Unbiased Local Gradient Estimator) Let g i ( x ) be the unbiased, local gradientestimator of r
f i ( x ) , i.e., E [ g i ( x )] =
r f i ( x ) , 8 x , and i 2 [ m ] .
3.2 Federated Optimization and Tuning of Hyperparameters
Federated Averaging (FedAvg)
We describe the operations of FedAvg from McMahan et al.[2016], as follows.
At any round t , each of the m clients takes a total of K i local SGD steps, where
K i =
b E ⌫ i /B c , and where ⌫ i is the number of data samples from client index i , B is batch size,with epoch number E
= 1 being a common baseline.
In this version of FedAvg, heterogeneous datasize is accommodated across clients, and the number of local steps can be manipulated via E and
B as hyperparameters.
Each local SGD step updates the local model parameters of each client i asfollows: x it,k
+
= x
it,k  ⌘ L g i ( x it,k ) , where ⌘ L is the local learning rate and k 2 [ K ] is the local stepindex.
To conclude each round, these clients return the local parameters x it,K i to the server whereit updates its global model, with x t +1 =
P i ⌫ i x it,K / ⌫ where ⌫ =
P i ⌫ i .
To facilitate some of thediscussions that follow, we deﬁne the following quantities:
[Equation]
i
[Equation]
Ofﬂine Hyperparameter Tuning Ofﬂine tuning is best to be summarized as follows.
We ﬁrstdeﬁne U
= { u 2 R | u  0 } with ⌘ L 2 U , and V = { v 2 I | v  1 } with K 2 V .
We also deﬁne 129 C
=
U ⇥ V , and c = ( ⌘ L , K ) , where c 2 C .
Ofﬂine tuning would have the following objective: min c 2 C f valid ( x, c ) s.t. x = argmin
z 2 R d f train ( z, c ) .
With abuse of notation, we use f valid for theobjective function calculated from a validation dataset which is usually held-out before the procedure,and f train for the objective from training data which usually is just local client data.
A few notableofﬂine tuning methods are as follows.
Global grid-search from Holly et al.
[2021] is an exampleof ofﬂine tuning that iterates over the entire search grid deﬁned as C , completing an optimizationprocess for each grid point and evaluating the result with a held-out validation set.
Global Bayesian
Optimization from Holly et al.
[2021] is another similar example of ofﬂine tuning that follows thesame template and objective.
Instead of brute-force grid-search, c is sampled from a distribution D Cover C , i.e. c ⇠ D C , that updates after every iteration.
Online Hyperparameter Optimization We are interested in an online procedure that combineshyperparameter optimization and model parameter optimization, with the following objective:
[Equation]
This formulation is the objective of our method, FATHOM, which we will discuss shortly in detail.
Ithas the advantage of joint optimization in a one-shot procedure.
Furthermore, it does not assume theavailability of a validation dataset.
3.3 Our Method: FATHOM
In this section we will introduce our method, FATHOM (Federated AuTomatic Hyperparameter
OptiMization).
Recall from our joint objective, eq(3), that both the model parameters, x , andhyperparameters of the optimization algorithm, c , are optimized jointly to minimize our objectivefunction.
An alternative view is to treat c as part of the parameters being optimized in a classicformulation,
i.e. min y f ( y ) with y =
( x, c ) .
As previously mentioned, our method is inspired byhypergradient descent from Baydin et al.
[2017] and by exponentiated gradient from Amid et al.[2022], both proposed for centralized learning rate optimization.
We will present how FATHOMexploits our knowledge of analytical gradients to update client learning rate, number of local steps, aswell as batch size, for an online, one-shot optimization procedure.
Assumption 2.
(Convexity w.r.t.
⌘ L and K )
We assume E t ( f ( x t )) is convex w.r.t.
⌘ L and K , eventhough we assume non-convexity w.r.t.
x t ).
Speciﬁcally, convexity w.r.t.
K follows the deﬁnition in
Murota [1998], to accommodate the integer space where K is deﬁned.
Remark 1.
Assumption 2 is necessary to guarantee the existence of subgradients derived in Theorems1 and 2, and it will be assumed for this work.
In problems dealing with deep neural networks, it isreasonable to not assume convexity w.r.t.
hyperparameters.
However, from our empirical results, weclaim that the proposed algorithm is still able to operate as desired under this condition.
3.3.1 Hypergradient for Client Learning Rate
In this section, we derive the hypergradient for client learning rate in a similar fashion as Baydin et al.[2017], with the difference being that they are mainly concerned with the centralized optimizationproblem, and that we are concerned with the distributed setting where clients take local steps.
Wederive the following hypergradient of the objective function as deﬁned in eq(1), taken with respect tothe learning rate ⌘ L,t  1 such that it can be updated to obtain ⌘ L,t :
[Equation]
where  t is the update step for the global parameters x t as deﬁned in eq(2), leading to @
t @⌘ L,
t =  t ⌘ L,t =
[Equation]
k =
g i ( x i,kt ) .
We also make the approximation x t +1  x t =
t ⇡  ⌘ L,t r f ( x t ) .
Wecan then write the normalized update, H t , similar to Amid et al.
[2022], as follows:
[Equation]
The resulting hypergradient is a scalar, as expected, and can be used efﬁciently as part of the updaterule for ⌘ L , which we will see in Section 3.3.4.
The implementation is communication efﬁcient, sincein each round, each client needs one extra scalar to send back to the server, and likewise the serverneeds to broadcast one extra scalar back to the clients.
It is also computationally efﬁcient since itavoids calculating the full local gradient r f ( x t ) .
3.3.2 Hypergradient for Number of Local Steps
Since the number of local steps is an integer, i.e. K = { k 2 I | k  1 } , this means f ( x t ) does not 177 exist for non-integer values of K .
We formulate a subgradient as a surrogate of the hypergradient@ f ( x t ) / @
K , as follows.
We will call this a hyper-subgradient.
Theorem 1.
When a piecewise function L t is deﬁned for every value of K 0
[ K ] on l , such that0 .
 l < 1 . 0 , we claim, under Assumption 2, that the following is a subgradient of f ( x t ) at
K t = K 0 :
[Equation]
where l represents the marginal fraction of local steps beyond K 0 .
We leave the proof (with anillustration in Figure 2) in the Appendix section beginning in eq(20).
The result from Theorem 1 is not sufﬁciently communication-efﬁcient for implementing an updaterule for K .
This is because it would require the quantity g i ( x i,
K t  1 t ) to be communicated fromeach client i to the server.
To save communication, let us reuse what the server has in memory: t =
[Equation]
P
[Equation]
.
If we let:
[Equation]
where eq(9) is the normalized update as in Amid et al.
We claim that eq(8) is a positivelybiased version of eq(6), which has its practical importance due to the fact that the last term in eq(6)from Theorem 1 results in zero-mean, noisy gradients, when the local functions are nearing their localsolutions, when in fact, this is the area where more local work is not needed.
Thus, a positive bias isdesirable to drive the number of local steps down.
This result is also useful from a communicationefﬁciency perspective in its implementation, because the server has all the components to calculatethis quantity, and would not require additional communication.
3.3.3 Regularization for Number of Local Steps
One of the goals for FATHOM is savings in local computation.
To avoid excessive number of localsteps, we further develop a regularization term for local computation against excessive K , which is aproxy for the hypergradient of the local client functions at the end of each round : @
f i ( x i,Kt ) / @
K .
Theorem 2.
When a piecewise function J t is deﬁned for every value of K 0
[ K ] on l , such that0 .
 l < 1 . 0 , we claim, under Assumption 2, that the following is a subgradient of
P m i =
1 f i ( x i,K t t )at K t = K 0 :
[Equation] [Equation]
where l represents the marginal fraction of local steps beyond K 0 .
We leave the proof in the Appendixsection beginning in eq(24).
In our algorithm, we use the normalized update based on the following biased proxy, since eq(10)tends to be noisy from g i ( x i,K t t ) .
[Equation]
where G t is the normalized update.
The proxy yields a bias towards smaller number of local steps,which is desirable for reducing local computation.
We use this biased proxy against using a moretypical regularization such as L2 for the number of local steps, based on initial empirical results forbetter performance..
3.3.4 Normalized Exponentiated Gradient Updates
For the update rules of the hyperparameters ⌘ L (client learning rate) and K (number of client localsteps), we use the normalized exponentiated gradient descent method (EGN) with no momentum,rather than a conventional linear update method such as the additive update of hypergradient descentproposed in Baydin et al.
It is reasonable to use exponentiated gradient (EG) methods forupdates of hyperparameters that are strictly positive in value.
EG methods also enjoy signiﬁcantlyfaster convergence properties when only a small subset of the dimensions are relevant, according to
Amid et al.
EG methods have been proposed in previous works for a variety of applications (Khodak et al.
Amid et al.
[2022], Li et al.
[2020]), and analyzed in depth (Ghai et al.
[2019]), where its convergencehas been studied and validated (Li and Cevher [2018]).
Recently, Amid et al.
[2022] showed that EGNis the same as the multiplicative update for hypergradient descent proposed in Baydin et al.
[2017],when the approximation exp ( · ) ⇡ 1 + · is made.
From our observations, we believe that momentumis not needed for the effectiveness of EGN in our application, as validated in our numerical results.
We also opted-out of adding further complexity such as extra weights and activation functions tomodel the relationships between ⌘ L,t and K t , because it would require more samples to optimize andbecause FATHOM is a one-shot procedure.
Furthermore, due to the non-stationary nature of thesevalues, we opt for a simpler scheme for faster performance.
Hence, for the update rule of client learning rate, ⌘ L , we have:
[Equation] [Equation]
where H t is as deﬁned in eq(5).
For number of local steps, we observe that it is related to batchsize in round t , B t , as follows.
To accommodate heterogeneity of local dataset sizes among clients,we have number of local data samples from client i to be ⌫ i .
The number of local steps for client iis K i =
b ⌫ i E t /B t c , where E t is number of epochs, with E t = 1 meaning the entire local datasetfor each client to be processed once per round.
We derive update rules for E t and B t globally tooptimize the number of local steps, without having to make any changes to our theoretical analysis toaccommodate the heterogeneity of local dataset sizes:
[Equation]
and
[Equation]
15) where N t and G t are deﬁned in eq(9) and eq(12), respectively.
These update rules accomplish the goalof updating the number of local steps via E t /B t with E t +1 B t +1 =
E t B t exp
[Equation]
t
[Equation]
Typically, with  B   E , (  B   E ) G t becomes a tunable regularization term as discussed at theend of Section 3.3.3.
3.3.5 Client Sampling
We present our method, FATHOM, as shown in Algorithm 1.
One practical factor we have notconsidered in our discussions is partial client sampling.
For our implementation to handle thestochastic nature of client sampling, the metric  t  1 for calculating H t in eq(5) and N t in eq(9) ismodiﬁed by a smoothing function for noise ﬁltering, i.e.  t,sm =
↵  t  1 ,sm +(1  ↵ )  t , which is asingle-pole inﬁnite impulse response ﬁlter (Oppenheim and Schafer [2009]Oppenheimer et al.
[2009])with no bias compensation.
We use the notation "sm" for smoothed, and after many experiments, wedecide to use ↵ = 0 .
5 for all of our numerical results.
Sample client set
For each client i initialize: t = x t and K t i b ⌫ i E
/B t c .
Set  i =
and  i =
+ 1 .
for k
1 do
For each client i , compute in parallel an unbiased stochastic gradient g i
( i
For each client i , calculate  i =
min(  i , g i ( x i kt ) ·  i ) where  i =
x i,kt  x t
For each client i , update in parallel its local solution: x i,k +1 t
= x i,kt  ⌘ L,t g i ( x i kt ) end
Server calcualtes ⌫ =
P i 2 S t ⌫ i , where ⌫ i is the size of client i dataset.
Server calculates  t =
P i 2 S t  i ( ⌫ i / ⌫ ) ; see eq(2)
Server updates global model x t +
= x t   t
Server calculates H t = N t =
t k  t k ·  t  1 ,sm k
t  1 ,sm k , modiﬁed from eq(5) and eq(9)
Server calculates G t ; see eq(
Server updates client learning rate ⌘ L,t +1 , epochs, E t +1 , and batch size B t +1 for the next round; see eq(13), eq(14), and eq(15).
Server updates  t,sm =
(1  ↵ )  t + ↵  t  1 ,sm for the next round end
Theoretical Convergence
A standard approach to theoretical analysis of an online optimization method such as ours, is throughanalyzing the regret bound (Zinkevich [2003], Khodak et al.
[2019], Kingma and Ba [2014], and
Mokhtari et al.
Nonetheless, this approach does not tell us the impact on communicationefﬁciency by the online updates introduced from FATHOM.
Therefore, we take an alternativeapproach by extending the guarantees of FedAvg performance (Wang et al.
[2021], Reddi et al.[2020], Gorbunov et al.
[2020], Yang et al.
[2021], Li et al.
[2019], etc) to include both adaptivelearning rate and adaptive number of local steps.
We assume the special case in our analysis to havefull client participation.
We prove that adaptive learning rate and adaptive number of local steps doesnot impact asymptotic convergence, despite the given relaxed conditions.
4.1 Assumptions
Assumption 3.
(L-Lipschitz Continuous Gradient for Parameters x t )
There exists a constant L > 0 ,such that kr f i ( x )  r f i ( y ) k  L k x  y k , 8 x , y 2 R d , and i 2 [ m ] , where x and y are the 263 parameters in eq(1.
Assumption 4.
(Bounded Local Variance)
There exist a constant  L > 0 , such that the variance ofeach local gradient estimator is bounded by E kr f i ( x )  g i ( x ) k 2   2 L , 8 x , and i 2 [ m ] .
Assumption 5.
(Bounded Second Moment)
There exists a constant G > 0 , such that E t kr f i ( x t ) k  267 G , i 2 [ m ] , 8 x t .
4.2 Convergence Results
Theorem 3.
Under Assumptions 1-5 and with full client participation, when FATHOM as shownin Algorithm 1 is used to ﬁnd a solution x ⇤ to the unconstrained problem deﬁned in eq(1), thesequence of outputs { x t } satisﬁes the following upper-bound, where, with slight abuse of notation,
[Equation]
[Equation]
with the following conditions: ⌘ L = min
[Equation]
and ⌘ L,t  1 /L for all t , where
[Equation]
, 1 T
[Equation]
and
K ,
[Equation]
T
[Equation]
17) and where
[Equation]
We leave the proof in the Appendix beginning in eq(29).
The values of  0 ,  1 ,  2 ,  3 , and  4 are dependent on the relative changes over the adaptive processof these components, according to Chebyshev’s Sum Inequalities (Hardy et al.
A specialcase is when these quantities equal to 1 when both ⌘ L,t and K t are constant, which recovers thestandard upperbound for FedAvg from eq(16).
Remark 2.
The deﬁnitions in eq(17) combined with the conditions for ⌘ L above is called the relaxedconditions in this paper for the hyperparameters ⌘ L,t and K t .
The values of ⌘ L,t and K t are adaptiveduring the optimization process between rounds t = 1 and
t = T , as long as the above conditions aresatisﬁed for the guarantee in eq(31) to hold.
This relaxation presents opportunities for a scheme suchas FATHOM to exploit for performance gain.
For example, suppose T approaches 1 for a prolongedtraining session.
Then ⌘ L would necessarily be sufﬁciently small for E fathom to be bounded byeq(16).
However, for early rounds i.e. small t values, ⌘ L,t  T ⌘ L can be reasonably large and stillcan satisfy eq(17), for the beneﬁt of accelerated learning and convergence progress early on.
Similarstrategy can be used for number of local steps to minimize local computations towards later rounds.
In any case, these strategies are mere guidelines meant to remain within the worst case guarantee.
However, Theorem 3 offers the ﬂexibility otherwise not available.
We will now show the empiricalperformance gained by taking advantage of this ﬂexibility.
Empirical Evaluation and Numerical Results
We present an empirical evaluation of FATHOM proposed in Section 3 and outlined in Algorithm1.
We conduct extensive simulations of federated learning in character recognition on the federated
EMNIST-62 dataset (FEMNIST) (Cohen et al.
[2017]) with a CNN, and in natural language next-wordprediction on the federated Stack Overﬂow dataset (FSO) (TensorFlow-Federated-Authors [2019])with a RNN.
We defer most of the details of the experiment setup in Appendix Section C.1.
Ourchoice of datasets, tasks and models, are exactly the same as the "EMNIST CR" task and
the "SO
NWP" task from Reddi et al.
See Figure 1 and Table 1 and their captions for details of theexperiment results.
Our evaluation lacks comparison with a few one-shot FL HPO methods discussedearlier in the paper because of a lack of standardized benchmark (until FedHPO- B Wang et al.
[2022]was published concurrently as this work) to be fair and comprehensive.
The underlying principle behind these experiments is evaluating the robustness of FATHOM versus
FedAvg under various initial settings, to mirror realistic usage scenarios where the optimal hyperparameter values are unknown.
For FATHOM, we start with the same initial hyperparameter valuesas FedAvg.
The test accuracy progress with respect to communication rounds is shown in Figure 1from these experiments.
We also pick test accuracy targets for the two tasks.
For FEMNIST CR weuse 86% and for FSO NWP we use 23%.
Table 1 shows a table of resource utilization metrics withrespect to reaching these targets in our experiments, highlighting the communication efﬁciency aswell as reduction in local computation from FATHOM in comparison to FedAvg.
To our knowledge,we are the ﬁrst to show gain from an online HPO procedure over a well-tuned equivalent procedurewith ﬁxed hyperparameter values.
The federated learning simulation framework on which we build our algorithms for our experimentsis FedJAX
(Ro et al.
[2021]) which is under the Apache License.
The server that runs the experimentsis equipped with Nvidia
[Table] [Table]
[Table]
Conclusion and Future Work
In this work, we propose FATHOM for adaptive hyperparameters in federated optimization, speciﬁcally for FedAvg.
We analyze theoretically and evaluate empirically its potential beneﬁts in convergence behavior as measured in test accuracy, and in reduction of local computations, by automaticallyadapting the three main hyperparameters of FedAvg: client learning rate, and number of local stepsvia epochs and batch size.
An example of future efforts to extend this work is using a standardizedbenchmark such as Wang et al.
[2022] for performance comparison against other FL HPO methods.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing [Figure] the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] The code is MIT licensed.
Please do not modify the questions and only use the provided macros for your answers.
Note that theChecklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Please refer to Sections 2 and 6(c)
Did you discuss any potential negative societal impacts of your work?
[No] Notspeciﬁcally, but it is alluded to how FL applications have social implications in theintroductory section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please refer to
Assumptions 1, 2, 3, 4, and
(b) Did you include complete proofs of all theoretical results?
[Yes] Yes, in the supple-
mental material.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Yes, in thesupplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Yes, in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Yes, see Table
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes, it is mentioned in Section 54.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, it is mentioned
in Section
(b) Did you mention the license of the assets?
[Yes] Yes, it is mentioned in Section
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes, in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]