A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study a large-scale multi-agent minimax optimization problem,which models many interesting applications in statistical learning and game theory,including Generative Adversarial Networks (GANs).
The overall objective is asum of agents’ private local objective functions.
We first analyze an importantspecial case, empirical minimax problem, where the overall objective approximatesa true population minimax risk by statistical samples.
We provide generalizationbounds for learning with this objective through Rademacher complexity analysis.
Then, we focus on the federated setting, where agents can perform local computation and communicate with a central server.
Most existing federated minimaxalgorithms either require communication per iteration or lack performance guarantees with the exception of Local Stochastic Gradient Descent Ascent (SGDA),a multiple-local-update descent ascent algorithm which guarantees convergenceunder a diminishing stepsize.
By analyzing Local SGDA under the ideal conditionof no gradient noise, we show that generally it cannot guarantee exact convergencewith constant stepsizes and thus suffers from slow rates of convergence.
To tacklethis issue, we propose FedGDA-GT, an improved Federated (Fed) Gradient Descent
Ascent (GDA) method based on Gradient Tracking (GT).
When local objectives are
Lipschitz smooth and strongly-convex-strongly-concave, we prove that FedGDAGT converges linearly with a constant stepsize to global ϵ -approximation solutionwith O (log(1 /ϵ )) rounds of communication, which matches the time complexityof centralized GDA method.
Finally, we numerically show that FedGDA-GToutperforms Local SGDA.
Introduction
In recent years, minimax learning theory has achieved significant success in attaching relevance tomany modern machine learning and statistical learning frameworks, including Generative Adversarial
Networks (GANs) [1–3], reinforcement learning [4], adversarial training [5,6], robust estimationand optimization [7–11] and domain adaptation [12,13].
Generally, a minimax learning problem ismodeled as a game between two players with opposite goal, i.e., one minimizes the objective whilethe other maximizes it.
Most current studies in the field of machine learning are targeted at understanding the minimaxproblem from the view of the speed of convergence and the accuracy of fixed points.
In the centralizedsetting, gradient descent ascent (GDA), which is an extension of gradient descent (GD), stands out forits simple implementation.
Specifically, at each iteration, the "min" player conducts gradient descentover its decision variable while the "max" player performs gradient ascent in contrast.
Due to thehuge volumn of data, stochastic gradient descent ascent (SGDA) is preferred in machine learningsettings.
Theoretical guarantees are well-established for GDA and SGDA in [14,15].
However, in practical scenarios, concerns on computation efficiency and data privacy trigger the developmentof federated learning over a server-client topology and distributed learning over a general graph.
These often require communication with the server or neighbors at each iteration [13,16,17] andinapplicable to scenarios where communication is expensive.
In this work, we focus solving a minimax problem in Federated Learning (FL) setting with oneserver and multiple client/agents.
In FL, the server hands over computation burden to agents, whichperform training algorithms on their local data.
The local trained models are then reported to theserver for aggregation.
This process is repeated with periodic communication.
Much existingliterature in FL, however, focuses on minimization optimization [18–22].
The limited literature onfederated minimax problems either lacks theoretical guarantees or requires frequent communication[13,23,24], with the exception of Local Stochastic Gradient Descent Ascent (SGDA) [25].
In Local
SGDA, each agent (or client) performs multiple steps of stochastic gradient descent ascent beforecommunication to the server, which then aggregates local models by averaging.
Under carefulselection of diminishing learning rates, [25, 26] show that Local SGDA converges to the globaloptimal solution sub-linearly.
However, as we show here, when we try to improve the speed ofconvergence and reduce communication overhead by introducing a constant stepsize to Local SGDA,it fails to converge to the exact optimal solution, even when full gradients are used.
Thus Local
SGDA can either be fast(with little communication)-but-inaccurate or accurate-but-slow(with muchcommunication).
To address this tradeoff between model accuracy and communication efficiency, wedevelop FedGDA-GT, Federated Gradient Descent Ascent based on Gradient Tracking (FedGDA-GT),and show that it can achieve fast linear convergence while preserving accuracy.
In addition to solving the minimax problem, we also study the generalization performance ofdistributed minimax problems in statistical learning, which measures the influence of sampling onthe trained model.
In most existing works, generalization analysis is established in the contextof empirical risk minimization (ERM) [27, 29].
It is well-known that for generic loss functions,the learning error for centralized ERM is on the order of O (1 / √ N ) with N denoting the totalnumber of training samples.
Recently, several works derive generalization bounds on centralizedminimax learning problems with the same order [30–32].
For generalization analysis in distributedminimax learning, learning bounds are only provided for specific scenarios, e.g., agnostic federatedlearning [13] and multiple-source domain adaptation [12].
In this paper, we provide generalizationbounds for distributed empirical minimax learning with the same order as results of centralized cases,generalizing the results in [13].
1.1 Related work
Generalization of minimax learning Recently, generalization properties of minimax learningproblems have been widely studied in different scenarios, including GANs and adversarial training.
For GANs, [33] analyzes the generalization performance of GANs when discriminators have restrictedapproximability.
[34] evaluates generalization bounds under different metrics.
In contrast, [35]suggests a dilemma about GANs’ generalization properties.
In the context of adversarial training,generalization performances are studied through Rademacher complexity analysis [36,39], functiontransformation [37], margin-based [38] approaches.
[40] studies the generalization bounds of convexconcave objective functions with Lipschitz continuity.
However, all these works are under centralizedsetting.
Recently, [13] provides generalization analysis under agnostic federated learning, where theobjective is optimized for any target distribution formed by a mixture of agents’ distributions.
Ourwork extends their generalization analysis to general distributed minimax learning problems.
Centralized minimax learning Historically, minimax problems have gained attraction of researchers since several decades ago.
An early instantiation is bilinear minimax problem, whichbecomes a milestone in game theory together with von Neumann’s theorem [41].
A simple algorithmis then proposed to solve this bilinear problem efficiently [42].
[43] generalizes von Neumann’stheorem to convex-concave games, which triggers an explosion in algorithmic research [44–46].
GDA, as one widely used algorithm, is notable for its simple implementation.
It is well-knownthat GDA can achieve an ϵ -approximation solution with O (log(1 /ϵ ))
iterations for strongly-convexstrongly-concave games and with O ( ϵ − 2 ) iterations for convex-concave games under diminishingstepsizes [14].
Very recently, nonconvex-nonconcave minimax problems appear to be a main focus inoptimization and machine learning, due to the emergence of GANs.
Several related works are listedtherein [
Distributed and federated minimax learning A few recent studies are devoted to distributedminimax problems due to the increasing volume of data and concerns on privacy and security.
Algorithm design and convergence behaviors are extensively studied for minimax problems in thecontext of distributed optimization, where communication is required at each iteration [52–55].
In the federated setting, [56] proposes a multiple-local-update algorithm to deal with distributionshift issue.
[57] studies federated adversarial training under nonconvex-PL objectives.
FedGANis proposed in [58] to train GANs in a communication-efficient way.
However, these works aretargeted at some specific scenarios.
Very recently, aiming to solve the general federated minimaxproblems, [25] proposes Local SGDA by allowing each agent performing multiple steps of GDAbefore communication.
The authors also prove sub-linear convergence for Local SGDA underdiminishing stepsizes.
Their convergence guarantees is then improved by [26] to match the results ofcentralized SGDA [15].
However, we note that all these algorithms require diminishing learning ratesto obtain exact solutions, which suffer from relatively slow convergence speed, but our algorithmallows constant stepsizes and hence linear convergence can be achieved.
1.2 Our Contributions.
We summarize our main contributions as follows:(1)
Analyzing the generalization properties ofempirical minimax learning in distributed settings through Rademacher complexity analysis [see
Section 3], (2) In federated setting, characterizing the behavior of fixed points of Local SGDA, whichreveals the impact of objective heterogeneity and multiple local updates on the model accuracy [see
Section 4.1], (3) Resolving the tradeoff between model accuracy and communication efficiency bydeveloping a linear-rate federated minimax algorithm that guarantees exact convergence [see Section4.2], (4) Providing numerical results which suggest communication efficiency of our algorithmcompared to Local SGDA and centralized GDA [see Section 5].
Notations.
In this paper, we let ∥ · ∥ denote l 2 -norm and | · | denote the cardinality of a set ora collection, or absolute value of a scalar.
Vectors are column vectors by default and z = ( x, y )forms the concatenated vector with z =
[ x T , y T ] T .
Vectors and scalars for agent i are denoted usingsubscript
i , e.g., f i ( x, y ) .
Superscripts, e.g., t , denote the indices of iterations.
We let the gradientof f ( x, y ) by ∇ f ( x, y ) =
( ∇ x f ( x, y ) , ∇ y f ( x, y )) , where ∇ x f ( x, y ) and ∇ y f ( x, y ) denote thegradients with respect to x and y , respectively.
Problem Setup
In this paper, we consider the general constrained minimax distributed optimization problem collectively solved by m agents:
[Equation]
where X , Y are some compact feasible sets contained in R p and R q , x ∈
X is a p -dimension vector,y ∈ Y is a q -dimension vector and f i ( · , · ) is the local objective function of agent i , ∀ i = 1 . . .
, m .
Solving (1) is equivalent to finding a minimax point of f ( x, y ) , defined as follows:
Definition 1.
The point ( x ∗ , y ∗ ) is said to be a minimax point of f ( x, y ) if
[Equation]
The first-order necessary condition for minimax points is given by the following lemma.
Lemma 1.
[59] Assume f is differentiable.
Then, any minimax point ( x ∗ , y ∗ ) in the interior of X × Y satisfies
[Equation]
Generalization bounds on minimax learning problems
In this section, we consider minimax statistical learning, an important application of the minimaxframework.
In a minimax statistical learning task, each agent is allocated with some local dataset
S i = { ξ i,j } n i j =1 , where ξ i,j denotes the j th sample of agent i and n i denotes the number of localsamples.
S =
(cid:83) mi
=1 S i is the dataset with all samples.
Moreover, we assume n i = n, ∀ i = 1 , . .
. , mand N = mn is the total number of samples.
Then, each local objective function f i ( x, y ) is definedby using local data, i.e.,
[Equation]
where l ( x, y ; ξ i,j ) is the loss function measured at point ξ i,j and (2) is called local empirical minimaxrisk.
Suppose that for agent i each data sample ξ i,j is independently drawn from some underlyingdistribution P i , denoted by ξ ∼ P i .
We can further define the local population minimax risk asfollows:
[Equation]
and similarly, the global population minimax risk is defined by
[Equation]
where P is the underlying distribution of the whole dataset S .
Then, the minimax problem based onthe population minimax risk is given by
[Equation]
Sample applications of our formulation (1)-(5) can be found in Appendix A.
While we are interested in calculating the population minimax risk, in practice we can only solve(1), which is a sampled version of (5), since the true distribution P is unknown.
In this sense, howcould we expect the optimal solution of (1) performs successfully on (4)?
To answer this question,we provide the generalization bound which measures the performance of the model trained on theempirical minimax risk f ( x, y ) .
Our generalization bound is based on the notion of Rademachercomplexity [29] defined by
[Equation]
where σ = { σ i,j } , ∀ i = 1 , . . .
, m, ∀ j = 1 , . .
. , n , is a collection of Rademacher variables takingvalues from {− 1 , 1 } uniformly.
Basically, the Rademacher complexity (6) captures the capability of
X to fit random sign noise, i.e., σ .
Note that σ i,j l ( x, y ; ξ i,j ) measures how well the loss l correlateswith noise σ on sample space S .
By taking the supremum, it means what the best extent the feasibleset X on average can correlate with random noise σ .
Thus, if X is richer, then the Rademachercomplexity is bigger.
Moreover, the minimax Rademacher complexity is defined on Y by
[Equation]
Given ϵ > 0 , we further define the ϵ -minimum cover of Y in l 2 distance by
[Equation]
where C ( Y , ϵ )
= { B ϵ ( y ) :
y ∈ Y} is a collection of open balls B ϵ ( y ) centered at y with radius ϵsuch that for any y ∈ Y there exists some B ϵ ( y ′ )
∈ C
( Y , ϵ ) with y ∈ B ϵ ( y ′ ) .
Note that (cid:12)(cid:12)
Y ϵ (cid:12)(cid:12) <
∞ ,since Y is compact and thus every open cover of Y has a finite subcover.
Let Y Uϵ be the union over
Y ϵ .
Then, we have the following generalization bound for the distributed minimax learning problem:
Theorem 1.
Suppose | l ( x, y ; ξ ) − l ( x, y ′ ; ξ ) | ≤ L y ∥
y − y ′
∥ and | l ( x, y ; ξ ) | ≤ M i ( y ) , ∀ x ∈ X ,∀ y,
y ′ ∈
Y U ϵ and ∀ ξ ∼ P
i , i = 1 . . .
, m with some positive scalar L y and real-valued function
M i ( y ) > 0 .
Then, given any ϵ > 0 and δ > 0 , with probability at least 1 − δ for any ( x, y ) ∈ X × Y ,
[Equation]
Theorem 1 generally states that given any x, y feasible, it is highly possible that the distance betweenthe global population minimax risk R ( x, y ) and the global empirical minimax risk f ( x, y ) can bebounded by Rademacher complexity and a term related to the number of agents and local samplesize.
Moreover, we allow the upper bound of l ( · , y ) can depend on different choices of y and agentssince different local distributions always have different effects on the value of l ( · , y ) .
Based on Theorem 1, we further derive the high-probability bound on max y ∈Y R ( x, y ) .
Corollary 1.
Under the same conditions of Theorem 1, with probability at least 1 − δ for any x ∈ X ,the following inequality holds for any ϵ > 0 , δ > 0 :
[Equation]
where Q ( x ) = max y ∈Y R ( x, y ) , g ( x ) = max y ∈Y f ( x, y ) are the worst-case population and empirical risks, respectively.
We can further bound R ( X , Y ) when the loss function l ( x, y ; ξ ) takes finite values for any ( x, y ) ∈
X × Y and any data samples.
Lemma 2.
Suppose for any y ∈ Y and i ∈ { 1 , . . .
, m } , | l ( · , y ; · ) | is bounded by M i ( y ) and takesfinite number of values.
Further, assume that the VC-dimension of X is d .
Then, the followinginequalities hold:
[Equation]
Corollary 1 measures the error between the worst-case population and empirical risks, that ismax y ∈Y R ( x, y ) − max y ∈Y f
( x, y ) .
A smaller error indicates a better performance of the model,which is trained empirically, generalized on the underlying distribution P .
Combining Lemma 2, weknow that this worst-case error can be bounded by some decreasing function with respect to samplesize.
Thus, in order to get a better generalization performance, one effective way is to draw morelocal samples for each agent.
It it worth noting that the generalization bound is related to the term (cid:80) mi =1 M
i
( y ) as shown in(8)-(10), which essentially measures the effect of parametrized hypothesis set Y by means of differentlocal data distributions on each agent.
This also captures the heterogeneity of agents.
In fact, the bounds (8)-(10) we proposed are generalized versions of those in [13], which essentiallyinclude them as special cases by selecting suitable M i ( y ) .
Specifically, noting that in [13], the globalpopulation risk takes the form of R ( x, y ) = (cid:80) mi =
y i R i ( x ) with y i ≥
and (cid:80) mi =
y i = 1 , then bychoosing M i ( y ) =
my i M with some M > 0 we recover the same result.
Moreover, comparedto [30,31], where only centralized minimax learning problems are considered, our bounds have thesame order of complexity O (1 / √ N ) with N denoting the total sample size by taking the uniformbound on l for all agents.
FedGDA-GT: A linear-rate algorithm for federated minimax learning
We now focus on solving (1) in the federated setting, where the m agents are connected to a centralserver.
The agents’ communication with the server is more expensive than local computation.
Theexisting methods either require lots of communication or could only converge to an inexact solution.
This suggests a tradeoff between model accuracy and communication efficiency.
Motivated bythis phenomenon, we aim at developing a communication-efficient minimax algorithm with linearconvergence that preserves model accuracy and low communication overhead simultaneously.
We adopt the following standard assumptions on the problem.
Assumption 1 ( µ -strongly-convex-strongly-concave) .
For any i = 1 , . . .
, m , f i ( x, y ) is twice continuously differentiable and is µ -strongly-convex-strongly-concave with some µ > 0 for any ( x, y ) ∈
R p × R q , i.e., for any given y ,
[Equation]
for any given x ,
[Equation]
Assumption 2 ( L -smoothness) .
There exists some L > 0 such that for any i = 1 , . . .
, m ,∥∇
[Equation]
We note that although each f i ( x, y ) may have different µ i and L i , we can set µ = min i =1 ,...,m µ iand L = max i =1 ...,m L i to ensure Assumptions 1 and 2 hold.
4.1 Analysis on Local SGDA
We first study Local Stochastic Gradient Descent Ascent (SGDA) proposed in [25], which is the onlyknown method with convergence guarantees and can utilize multiple local computation steps beforecommunicating with the central server for solving general federated minimax problems.
In particular,in each iteration of Local SGDA, each agent updates its local model ( x i , y i ) for K times by usinglocal stochastic gradients before communication, and then sends its local model to the server, whichthen computes the average of local models and sends it back.
In [25], the authors prove that under Assumptions 1-2, with bounded variance assumption on all localgradients and vanishing learning rate, after T rounds of communication, the convergence result of
Local SGDA is
[Equation]
(11) for some scalar a .
This shows the slow sublinear convergence rate of this method, which translatesto large amount of communication.
We next consider a ideal deterministic version of Local SGDA(Algorithm 1), where local stochastic gradients are replaced with full gradients.
Input: ( x 0 , y 0 ) as initialization of the server 1: for t = 0 , 1 , . .
, T do 2: x t +1 i,
= x t , y t +1 i, 0 =
y t , ∀ i = 1 , . . .
, m 3: for k = 0 , 1 , . . .
, K − 1 do (in parallel for all agents) 4: x t
+1 i,k +
= x
t +
i,k − η x ∇ x f i ( x t +1 i,k , y t
+1 i,k ) 5: y t
+1 i,k +1 =
y t
+1 i,k + η y ∇
y f i ( x t +1 i,k , y t
+1 i,k ) 6: end for 7: x t +1 = 1 m (cid:80) mi =1 x t
+
i,K , y t +1 =
1 m (cid:80) mi =1 y t +1 i,K 8: end for
Output: ( x T , y T ) given by the server
In order to rewrite Local SGDA in a concise form, we define the operator for gradient descent evaluated at ( x, y ) for agent i as
[Equation]
Given some k ≥ 1 , let D ki define the composition of k D 1 i operators.
Moreover, D
i ( x, y ) =
x isthe identity operator on the first argument.
Let A 1 i ( x, y )
=
y + η y ∇ y f i ( x, y ) be the gradient ascentoperator with A ki ( x, y ) and A 0 i ( x, y ) =
y defined similarly.
Then, given some initial point ( x 0 , y 0 ) ,
Algorithm 1 is rewritten as the recursion of the following for t = 0 , 1 , . . .
, T :
[Equation]
[Equation]
The following result shows that Local SGDA generally cannot guarantee the convergence to theoptimal solution of (1) with fixed stepsizes even with full gradients.
Proposition 1.
Suppose f i is differentiable, ∀ i = 1 , . . .
, m .
For any K ≥ 1 , let { ( x t , y t ) } be thesequence generated by (12) (or equivalently by Algorithm 1).
If { ( x t , y t ) } converges to a fixed point( x ∗ , y ∗ ) , then
[Equation]
Note that when K = 1 , Local SGDA reduces to GDA and the fixed point ( x ∗ , y ∗ ) satisfies
[Equation]
which meets the optimality condition (by Lemma 1) of the minimax point of f ( x, y ) .
For any other
K , the fixed point characterization is different from the minimax point.
An illustrative example isprovided in Appendix D.
We mention that while having multiple local gradient steps in the inner loop allows the agents toreduce the frequency of communication, it also pulls the agents towards their local minimax points asopposed to the global one.
Motivated by this observation, we next propose a modification involving agradient correction term, also known as gradient tracking (GT), to make sure the agents are movingtowards the global minimax point.
4.2 FedGDA-GT and convergence guarantees
In this section, we introduce our algorithm FedGDA-GT, formally described in Algorithm 2, that canreach the optimal solution of (1) with fixed stepsizes, full gradients and the following assumption.
Assumption 3.
The feasible sets X and Y are compact and convex.
Moreover, there is at least oneminimax point of f ( x, y ) lying in the interior of X × Y .
Input
, K − 1 do (in parallel for all agents) 7: x t
+1 i,k +
= x
t +
i,k
[Equation]
η ( ∇ x f i ( x t +
i,k , y t +1 i,k
[Equation]
x f i ( x t , y t )
[Equation]
x f ( x t , y t )) 8: y t +
[Equation]
,k +
[Equation]
=
y t +1 i,k
[Equation]
η ( ∇ y f i ( x t +
i,k , y t +1 i,k
[Equation]
− ∇ y f i ( x t , y t )
+ ∇ y f
( x t , y t )) 9: end for 10:
All agents send ( x t +1 i,K , y t
+1 i,K ) to the server to compute ( x t +1 , y t +1 ) by 11:
[Equation]
12: end for
Output: ( x T , y T ) given by the server
Proj X ( · ) and Proj Y ( · ) denote the projection operators, i.e.,
[Equation]
and Proj
[Equation]
arg
[Equation]
−
[Equation]
∥ , which are well defined by the previous assumption and are used to guarantee the output of Algorithm2 is feasible.
The main differences between FedGDA-GT and Local-SGDA are the correction termsof −∇ x f i ( x t , y t ) + ∇ x f ( x t , y t ) and −∇ y f i ( x t , y t )
+ ∇ y f
( x t , y t ) , which track the differencesbetween local and global gradients.
By including the gradient correction terms, when ( x t +1 i,k , y t +1 i,k )is not too far from ( x t , y t ) , we can expect ∇ x f i ( x t +1 i,k , y t
+1 i,k )
≈
∇ x f i ( x t , y t ) and similarly for ygradients.
Hence the updates reduce to simply taking global gradient descent and ascent steps, whichcoincide with central GDA updates and thus would have the correct fixed points.
We next show theconvergence guarantee of FedGDA-GT.
We first establish the uniqueness of minimax point of f ( x, y ) by the following lemma.
Lemma 3.
Under Assumptions 1 and 3, ( x ∗ , y ∗ ) is the unique minimax point of f ( x, y )
if and only if
[Equation]
Next, we state the main convergence result of our algorithm.
Theorem 2.
Suppose Assumptions 1, 2, 3 are satisfied.
Let { ( x t , y t ) } ∞ t =0 be a sequence generated by Algorithm 2.
Then there exists a scalar η 0 > 0 , such that for any stepsize η ∈ (0 , η 0 )
[Equation]
where ( x ∗ , y ∗ ) is the unique minimax point of f ( x, y ) and ρ ( η ) is some scalar in (0 , 1) .
Theorem 2 guarantees linear convergence of FedGDA-GT (Algorithm 2) to the correct optimalsolution of (1) under suitable choices of stepsize η .
We note that linear convergence here is withrespect to the outer-loop (indexed by t ), as the inner-loop (indexed by k ) can be implementedcheaply without any communication.
The fast convergence speed and exact convergence of thisresult eliminate the tradeoff between model accuracy and communication efficiency.
Furthermore,no restriction on heterogeneity level of local objectives is placed.
For the homogeneous setting, weshow in Appendix E.2 that the convergence rate of FedGDA-GT can be improved at least K times,compared to heterogeneous setting.
Experiments
In this section, we numerically measure the performance of FedGDA-GT compared to Local SGDAwith full gradients on a personal laptop.
We consider first perform experiments on quadratic objectivefunctions with x and y uncoupled.
Then, we test our algorithm on the robust linear regressionproblem.
In both cases, FedGDA-GT performs much better than Local SGDA with heterogeneouslocal objectives.
5.1 Uncoupled quadratic objective functions
We first consider the following local objective functions:
[Equation]
where x, y ∈ R d and A i ∈ R n i × d with n i representing the number of samples of agent i .
Wegenerate A
i , b i as follows:
For each agent, every entry of A i , denoted by [ A i ] kl , is generated by Gaussian distribution
N (0 , (0 . 5 i ) − 2 ) .
To construct b i , we generate a random reference point θ i ∈ R d , where θ i ∼
N ( µ i , I d × d ) .
Each element of µ i is drawn from N ( α, 1) with α ∼ N (0 , 100) .
Then b i =
A i θ i + ϵ iwith
ϵ
i ∼ N (0 , 0 .
25 I n i × n i ) .
We set the dimension of model as d = 50 and number of samples asn i = 500 and train the models with m = 20 agents by Algorithm 1 and Algorithm 2, respectively.
Inorder to compare them, the learning rate is 10 − 4 for both algorithms and we choose Local SGDA with
K = 1 , which is equivalent to a centralized GDA, as the baseline.
Figure 1 shows the trajectories of
[Figure]
280 Algorithms 1 and 2 under objective functions constructed by (13), respectively.
Different numbers oflocal updates are selected (with K = 20 and K = 50 ).
In this heterogeneous setting, we can see that
FedGDA-GT achieves linear convergence, converging to a more accurate solution with significantlyfewer rounds of communication, compared with Local SGDA and centralized GDA.
Moreover, ournumerical results suggest that Local SGDA may converge to a non-optimal point (optimality gap over10 4 ), which conforms with our Theorem 1.
5.2 Robust linear regression
Next, we consider the problem of robust linear regression, which is widely studied in estimation withgross error [28,39].
As the same formulation in [25], each agent’s loss function is defined by
[Equation]
where ( a i,j , b i,j ) is the j th data sample of agent i , n i is local sample size.
Specifically, in (14),x ∈ R d represents the model of linear regression and
y ∈ R d represents the gross noise aimed atcontaminating each sample.
We assume that there is an upper bound on the noise, i.e., ∥ y ∥ ≤ 1 .
By solving min x ∈ R d max ∥ y ∥≤ 1 1 m (cid:80) mi =
f
i ( x, y ) , we obtain a global robust model of the linearregression problem even under the worst contamination of gross noise.
To measure the convergenceof algorithms, we use the robust loss, i.e., given a model ˆ x , the corresponding robust loss [25,26] isdefined by ˜ f
(ˆ x ) = max ∥
y ∥≤ 1 (cid:80) mi =
f
i
(ˆ x, y ) .
We generate local models and data as follows: the local model x ∗ i is generated by a multivariatenormal distribution.
The output for agent i is given by y i,j = ( x ∗ i ) T
a i,j + ϵ j with ϵ j ∼ N (0 , 1) .
Each input point a i,j is with dimension d and drawn from a Gaussian distribution a i,j ∼ N ( µ i , K i )
where µ i ∼ N ( c i , I d × d ) and K i = i − 1 .
3 I d × d .
Each element of c i is drawn from N (0 , α 2 ) .
Bychoosing different α , we control the heterogeneity of local data and hence f i ( x, y ) .
In this experiment, we compare Algorithms 1 and 2 under different heterogeneity levels, i.e., α = 1 ,α = 5 and α = 20 .
To make a fair comparison, for each case, we choose the same constant η for both
Local SGDA and FedGDA-GT.
As shown in Figure 2, when local agents are more heterogeneous,
FedGDA-GT performs better than Local SGDA, which lies not only in faster convergence but alsosmaller robust loss.
Specifically, when α = 1 , two algorithms almost have the same performance.
Toexplain this phenomenon, let us recall FedGDA-GT again.
Smaller α essentially means more similarlocal objectives.
In particular, α = 0 corresponds to i.i.d.
cases.
In this sense, the local updates of
FedGDA-GT become the same as that in Local SGDA, which indicates similar performance as shownin Figure 2(a).
[Figure] [Figure]
Conclusion
In this paper, we first study the generalization properties of distributed minimax learning problems.
We establish generalization error bounds without strong assumptions on local distributions andloss functions by using Rademacher complexity.
The bounds match existing results of centralizedminimax learning problems.
Then in federated setting, we characterize the fixed-point behavior of arecent algorithm Local SGDA to show that it presents a tradeoff between communication efficiencyand model accuracy and cannot achieve linear convergence under constant learning rates.
To resolvethis issue, we propose FedGDA-GT that guarantees exact linear convergence and reaches ϵ -optimalitywith
O (log(1 /ϵ )) time, which is the same as centralized GDA method.
Finally, we compare FedGDAGT with two state-of-the-art algorithms, Local SGDA and GDA, through numerical experiments andshow that FedGDA-GT outperforms in efficiency and/or accuracy.