F ED A VG C ONVERGES TO Z ERO T RAINING L OSS L INEARLY FOR O VERPARAMETERIZED M ULTI -L AYER N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated Learning (FL) is a distributed learning paradigm that allows multiple clients to learn a joint model by utilizing privately held data at each client.
Significant research efforts have been devoted to develop advanced algorithms that deal with the situation where the data at individual clients have heterogeneous distributions.
In this work
, we show that data heterogeneity can be dealt from a different perspective.
That is by utilizing a certain overparameterized multi layer neural network at each client even the vanilla FedAvg (a.k.a the Local SGD) algorithm can accurately optimize the training problem: When each client has a neural network with one wide layer of size N (where N is the number of total training samples) followed by layers of smaller widths
FedAvg converges linearly to a solution that achieves (almost) zero training loss without requiring any assumptions on the clients’ data distributions.
To our knowledge, this is the first work that demonstrates such resilience to data heterogeneity for FedAvg when trained on multi-layer neural networks
Our experiments also confirm that neural networks of large size can achieve better and more stable performance for FL problems
I NTRODUCTION
In Federated Learning (FL), multiple clients collaborate with the help of a server to learn a joint model McMahan et al.
The privacy guarantees of FL has made it a popular distributed learning paradigm, as each client holds a private data set and aims to learn a global model without leaking its data to other nodes or the server.
The performance of FL algorithms is known to degrade when training data at individual nodes originates from different distributions, referred to as the heterogeneous data setting Yu et al. (2019a);
Woodworth et al.
(2020a).
In the past few years, a substantial research effort has been devoted towards developing a large number of algorithms that can better deal with data heterogeneity, Karimireddy et al.
(2020b)
; Zhang et al.
; Li et al.
; Acar et al.
Khanduri et al.
However, in practice it has been observed by a number of recent works, that in spite of the data heterogeneity, the simple vanilla FedAvg algorithm (a.k.a.
the Local SGD) still offers competitive performance in comparison to the state-of-the-art.
For example, see Table 2 in Karimireddy et al.
(2020a),
Table 1 in Reddi et al.
(2020), and Tablein Yang et al.
(2021) for performance comparison of FedAvg on popular FL tasks.
Motivated by these observations, we ask: Is it possible to handle the the data heterogeneity issue from a different perspective, without modifying the vanilla FedAvg algorithm?
To answer this question, in this work we show that FedAvg can indeed perform very well regardless of the heterogeneity conditions, if the models to be learned are nice enough.
Specifically, FedAvg finds solutions that achieve almost zero training loss (or almost global optimal solution) very quickly (i.e., linearly), when the FL model to be trained is certain overparameterized multi-layer neural network.
To the best of our knowledge, this is the first result that shows (linear) convergence of FedAvg in the overparameterized regime for training multilayer neural networks.
The major contributions of our work are listed below.
• Under certain assumptions on the neural network architecture, we prove some key properties of the clients’ (stochastic) gradients during the training phase (Lemmas 1 and 2).
These results allow us to establish convergence of FedAvg for training overparameterized neural networks without imposing restrictive heterogeneity assumptions on the gradients of the local loss functions.
• We design a special initialization strategy for training the network using FedAvg.
The initialization is designed such that the singular values of the model parameters and the outputs of the first layer of local and aggregated model parameters stay positive definite during the training.
This property combined with overparameterization enables FedAvg to converge linearly to a (near) optimal solution.
• We conduct experiments on CIFAR-10 and MNIST datasets in both i
i.d. and heterogeneous data settings to compare the performance of FedAvg on various network architectures of different sizes.
To our knowledge, this is the first work that shows the linear convergence of FedAvg (both SGD and
GD versions) to the optimal solution when training a overparameterized multi-layer
neural networks
Related Work:
Federated Learning (FL).
FL algorithms were first proposed in McMahan et al.
(2017), where within each communication round the clients utilize their private data to update the model parameters using multiple SGD steps.
Earlier works analyzed the performance of FedAvg for the case of homogeneous data setting Zhou and Cong (2018); Stich (2018); Lin et al.
Woodworth et al.
(2020b)
; Wang and Joshi (2021), i.e., when the local data at each client follows the same underlying distribution.
Motivated by practical applications, recent works have analyzed
FedAvg for heterogeneous client data distributions Yu et al.
(2019b;a); Haddadpour and Mahdavi (2019);
Woodworth et al.
(2020a)
and it was observed that the performance of FedAvg degrades as the data heterogeneity increases.
To address the data heterogeneity issue among clients, many works have focused on developing sophisticated algorithms Karimireddy et al.
(2020b)
; Zhang et al.
(2021); Acar et al.
; Li et al.
Khanduri et al.
; Karimireddy et al.
(2020a)
; Das et al.
Overparameterized Neural Networks.
The surprising performance of overparameterized neural networks 1 has raised significant research interest in the ML community to analyze the phenomenon of overparameterization Belkin et al.
Consequently, many works have analyzed the performance of centralized (stochastic) gradient descent (S)GD on overparameterized neural network architectures under different settings Jacot et al.
; Li and Liang (2018); Arora et al.
Du et al.
(2018; 2019); Allen-Zhu et al.
; Zou and Gu (2019); Nguyen and Mondelli (2020);
Nguyen (2021).
However, there are only a handful of works that have attempted to analyze the performance of overparameterized neural networks in the distributed setting Li et al.
; Huang et al.
Deng and Mahdavi (2021).
The works most closely related to our work are Huang et al.
(2021) and
Deng and Mahdavi (2021).
Huang et al.
(2021) analyzed the performance of FedAvg on a single hidden-layer neural network for the case when each client utilizes GD for the local updates.
The authors established linear convergence of FedAvg using the NTK parameterization and showed that it suffices to design the neural network of width Ω( N 4 ) to achieve this performance (where N is the number of training samples).
Similarly, Deng and Mahdavi (2021) analyzed the performance of
FedAvg on a ReLU neural network but when each client utilizes SGD (or GD) for the local updates.
The authors proved convergence of FedAvg under the standard parameterization while requiring the very large network width of Ω( N 18 ) .
Note that since individual clients can be devices with limited computational capabilities, in realistic settings it is undesirable to have networks of such large widths.
In contrast to both these works, we focus on the more practical setting of a multi-layer neural network Nguyen and Mondelli (2020) and establish linear convergence of FedAvg even for the case when each client utilizes SGD for the local updates.
Importantly, we show that with proper initialization, it only requires a network of width N at each client, which is much smaller compared to the unrealistic requirements of Huang et al.
; Deng and Mahdavi (2021).
P ROBLEM S ETUP
In this section, we define the multi-layer neural network and formalize the problem we aim to solve.
We consider a distributed system of K clients with each client having access to a privately held data set.
We assume that each client k ∈ { 1 , . . .
, K } has N k training samples denoted as { ( X k , Y k ) } , with X k ∈ R N k ×
d in and Y k ∈ R N k × d out .
Note that each row of X k and Y k represents the feature vector and its corresponding label, and d in and d out denote the feature (input) and label (output) dimensions, respectively.
We further denote N = (cid:80)
Kk =1 N k as the total samples across all clients.
Suppose each client trains a fully-connected neural network with L layers, and with activation function σ : R → R .
We denote the vectorized parameters at each node k ∈ { 1 , . . .
, K } as θ k = [vec ( W 1 ,k ) , . .
, vec ( W L,k )]
∈ R D , where W l,k ∈ R n l − 1 × n l represents the weight matrix of each layer l ∈ { 1 , . . .
, L } and n l represents the width of each layer.
Note that each layer inputs a (feature) vector of dimension n l − 1 and outputs a (feature) vector of dimension n l .
For simplicity, define n 0 = d in and n L = d out as the input and the output dimensions of the neural network.
We define F l,k as the local output of each layer l at client k , then using the above notations, we have
[Equation]
We further define the vectorized output of each layer and the labels at each client as f l,
k = vec ( F l,k ) ∈
R N k n l and y k = vec ( Y k ) ∈ R N k n L .
Similar to the above setup, we also define the notations to describe a single network, with the full data ( X, Y ) with X ∈ R N × d in and Y ∈ R N × d out as input.
This “centralized" network will be useful later to perform the analysis.
Then given parameter θ = [vec ( W 1 ) , . . .
, vec ( W L )] , the output at each layer of the network is defined as
[Equation]
Next, we define the local and global loss functions.
First, each client k ∈ { 1 , . . .
, K } has a local loss function given by: Φ k ( θ ) :
=
12 N k ∥ f L,k ( θ ) − y k ∥ 22 , where ∥ · ∥ 2 denotes the standard ℓ 2 -norm.
Then the global loss function is the sum of weighted local loss functions, given by:
[Equation]
Additionally, define the gradient of (3) as g :=
[vec( ∇ W 1 Φ( θ )) , . .
, vec( ∇ W L Φ( θ ))] , which is the stacked gradient of the loss w.r.t.
the 1 st to L th layer’s parameters; define the gradient of the losses at each client k ∈
[ K ] as: g k :=
[ g 1 ,k , . . .
, g L,K ] with g l,k := vec( ∇ W l,k Φ( θ )) for all l ∈
[ L ] .
Next, we define the optimality criteria to solve (3) using an overparameterized neural network.
Definition 1 ( ϵ -optimal solution) .
Consider an overparameterized problem min θ Φ( θ ) , where there exist θ ∗ such that Φ( θ ∗ ) = 0 .
A solution θ is called an ϵ -optimal solution if it satisfies Φ( θ ) ≤ ϵ .
Moreover, if θ is a random variable, then we use E [Φ( θ )] ≤ ϵ to denote an ϵ -optimal solution, where the expectation is taken w.r.t.
the randomness of x .
T HE F ED A VG A LGORITHM
A classical algorithm to solve problem (3) is the FedAvg McMahan et al.
In FedAvg, each client performs multiple local updates before sharing their updated parameters with the server.
We refer the algorithm as FedAvg-SGD (resp.
FedAvg-GD)
if the clients employ SGD (resp.
GD) for the local updates.
The detailed steps to implement FedAvg-SGD are listed in Algorithm 1.
We execute the algorithm for a total of T communication rounds, within each communication round every client performs r local updates.
In each communication round t the server aggregates the local parameters and constructs ¯ θ rt from each client’s local parameters θ rt + r k and shares it with the clients.
The clients use the aggregated parameter, ¯ θ rt + r k , as the initial parameter value for computing the next round of local updates.
For each v ∈ { 0 , 1 , . . .
, r − 1 } , to update the local parameters the clients compute the (unbiased) stochastic gradient using m -samples drawn form their private data set ( X k , Y k ) .
We denote the random sample drawn at v th local step in the t th communication round as ( ˜ X rt + v k , ˜ Y rt + v k ) .
Using the stochastic gradient estimate, the clients update their parameters locally by employing the
SGD step.
After r local SGD steps, each client shares its updated parameters with the server and gets back the aggregated parameters before starting the next round of updates.
Note that if we choose the batch size m = N k , for all k ∈ { 1 , . .
, K } , FedAvg-SGD becomes FedAvg-GD.
Algorithm
Initialize: Parameters θ 0 k
= θ 0 , Step-size η , # of communication rounds, local updates T , r for t = 0 , 1 , . .
, T − 1 do for each client k ∈ { 1 , . . .
, K } do
Set θ rtk = ¯ θ rt for v = 0 , 1 , . . .
, r − 1 do
Sample mini-batch of size m , ( ˜ X rt + v k , ˜ Y rt
+ v k )
Compute stochastic gradient ˜ g rt + v k using (5)
Update: θ rt + v +1 k
= θ rt + v k − η ˜
g
rt + v k .
Aggregation : ¯ θ r ( t +1) =
K (cid:80) k
=
N k N θ rt + r k
Return: Parameters, ¯ θ rT
Algorithm 1 summarizes the above description.
For each communication round t ∈ { 0 , 1 , . . .
, T − 1 } and local step v ∈ { 0 , 1 , . . .
, r − 1 } , we define the vector θ rt + v k :=
[vec( W rt + v 1 ,k ) , . .
, vec( W rt + v L,k )] .
For FedAvgSGD, define ˜ F rt + v l,k and ˜ f
rt + v l,k as the output and the vectorized output of each hidden layer l , respectively, when the input to the client k ’s local network is the stochastic (mini-batch) samples ( ˜ X rt
+ v k , ˜ Y rt + v k ) .
Using the notation ˜
y rt
+ v k = vec( ˜ Y rt + v k ) as the vectorized labels of the stochastic samples at each local step, we define the mini-batch stochastic loss as:
[Equation]
and the stochastic gradient as ˜ g rt + v k :=
[˜ g rt + v 1 ,k , . . .
, ˜ g rt + v L,k ] , where ˜ g rt + v l,k is the stochastic gradient w.r.t.
the l th layer of the network evaluated at the k th client:
[Equation]
For each communication round, let us define the aggregated parameters as:
[Equation]
For FedAvg-GD, we denote g rt + v k :
[Equation] [Equation]
rt
[Equation]
L
[Equation]
] as the full gradient of k th client’s loss function, where similar to (5) g rt + v 1 ,k defines the gradient of the loss function w.r.t.
the l th layer’s parameters.
Throughout, we make the following standard assumption Ghadimi and Lan (2013).
Assumption 1.
The stochastic gradients at each client are unbiased, i.e., we have E [˜ g rt + v k ]
= g rt + v k ∀ k ∈
[ K ] .
Next, we analyze the performance of the FedAvg for an overparameterized neural network.
C ONVERGENCE A NALYSIS
We present the convergence guarantees of FedAvg when training an overparameterized neural network.
We first present a set of assumptions on the network architecture, and activation functions.
Assumption 2.
The width of each hidden layer satisfies: n 1 ≥ N, n 2 ≥ n 3 ≥ . .
≥ n L ≥ 1 .
Assumption 3.
The activation function σ ( · ) in (1) satisfies the following:
[Equation]
[ γ, 1];
) | σ ( x ) | ≤ | x | ; ∀ x ∈ R ; 3) σ ′ is β -Lipschitz, with γ ∈ (0 , 1) and β > 0 .
Remark 1.
Assumptions 2 and 3 play an important role in our analysis.
They help ensure that the local and global loss functions and their (stochastic) gradients are well behaved.
Note that Assumption 2 only requires the first layer to be wide while the rest of the layers can be of constant width.
Assumption 2 is required to establish a PL like property for the global and local loss functions Nguyen and Hein (2018); Nguyen and Mondelli (2020).
Assumption 3 is also standard in the analysis of overparameterized neural networks.
Similar assumptions on the smoothness of the activation functions have been made in the past Jacot et al.
Du et al.
; Nguyen and
Mondelli (2020); Huang and Yau (2020) and are utilized to manage the behavior of the gradients of the loss functions.
Importantly, note that as demonstrated in Nguyen and Mondelli (2020) activation functions satisfying Assumption 3 can be utilized to uniformly approximate the ReLU function to arbitrary accuracy.
Remark 2.
We do not impose any assumptions on the distribution of individual clients’ local data sets.
In contrast, a majority of works on FL impose restrictive assumptions on the gradients (and/or the Hessians) of each client’s local loss functions to guarantee algorithm convergence Yu et al.
(2019b)
; Li et al.
; Yu et al.
(2019a)
; Karimireddy et al.
(2020a).
Below, we list two most popular heterogeneity assumptions (from Yu et al. (2019a) and Koloskova et al.
(2020), respectively):
[Equation]
Both conditions impose strong restrictions on the gradients of the local clients, and they do not hold for even simple quadratic loss Khaled et al.
; Zhang et al.
We will see shortly that, our results will indicate that as long as the neural network is large enough, then the local (stochastic) gradients will be well-behaved, thereby eliminating the need to impose any additional assumptions on the data distributions.
In the following, we show the convergence guarantees achieved by FedAvg.
Our analysis roughly follows the four steps presented below:
[Step 1] We first show a key result , that the ratio of the local stochastic gradients and the local full
gradients
stays
bounded
(Lemma 1).
This result
is
crucial for the
FedAvg
SGD
analysis, as it allows us to work with the
full
local
gradients
directly , and it helps to bound the gradient drift across local updates within each communication round.
[Step 2] Using the result of Step
, we bound the summation of (stochastic
gradients and the gradient drift during the local updates within each communication round (Lemma
This result ensures
that
irrespective of the data
heterogeneity, the
gradients
size will not change
too much from their
initial values at the beginning of each round
[Step
We then
show that
adopted
network
architecture allows
us to derive bounds on the size of the gradients and ensure
the loss
function to be PL during the each communication round (Lemma 3).
Utilizing this and the results derived in Steps 1 and 2, we show that the expected loss (3) converges linearly to zero (Proposition 1).
[Step 4]
Finally, we find a special initialization strategy so that all the conditions imposed on the network properties are satisfied during the entire training process.
Next, let us begin with Step 1.
We need the following definition.
Definition 2.
Given parameter θ rt + v k , we define the following quantity for each k ∈
[ K ] , t ∈ { 0 , 1 , . .
, T − 1 } and v ∈ { 0 , 1 , . . .
, r − 1 } : ρ ( θ rt + v k
[Equation]
Clearly, ρ ( θ rt + v k ) measures the ratio of the norm of stochastic and full gradients of the local loss functions.
In the following, we show that if the model parameters at each client satisfy certain conditions, then ρ ( θ rt + v k ) is uniformly bounded.
Define σ max ( · ) and σ min ( · ) as the largest and smallest singular value of a matrix, respectively.
Lemma 1.
Let Assumptions 2 and 3 hold.
Suppose in any iteration rt + v, v ∈ { 0 , 1 , · · · , r − 1 } , for θ rt + v k =
[vec( W rt + v 1 ,k ) , . .
, vec( W rt + v L,k )] , there exists constant ¯Λ l , Λ l , Λ F > 0 such that the singular values of W rt + v l,k and F rt + v 1 ,k satisfy
[Equation]
) where λ i → j := j (cid:81) l
= i λ l for given layer-wise parameter λ l , then: ρ ( θ rt + v k
[Equation]
As discussed earlier in Step 1, this lemma is crucial to our analysis as it allows us to work with full gradients of individual clients.
Before proceeding to Step 2, we need the following definitions:
[Equation]
Here ¯ g rt + v and ¯˜ g rt + v are the weighted averages of the full and stochastic gradients, respectively.
Next, in Step 2 (Lemma 2) we first bound the size of the sum of ¯˜ g rt + v over the local updates within each communication round.
Then we bound the change in ¯ g rt + v from v = 0 to any v ∈ { 0 , 1 , . . .
, r − 1 } .
Note that this quantity measures the drift in the averaged gradients from the start of each communication round.
Lemma 2.
For FedAvg-SGD, given step size η > 0 , v ∈ { 0 , 1 , . . .
, r − 1 } and q ∈ { 0 , 1 , . .
. , v − 1 } .
Suppose there exists constants ¯Λ l , ρ , and A > 0 such that the following conditions hold:
[Equation]
Then we have
[Equation]
Further, for all k ∈ [ K ] , ∃ Q k > 0 , such that we have
[Equation]
Next, we show Step 3, that the averaged parameter ¯ θ rt defined in (6), after t th communication round, will have good performance.
Towards this end, we define the full gradient given parameter ¯ θ rt as
[Equation]
Lemma 3.
Let Assumptions 2 and 3 hold.
At each communication round rt , suppose there exists constant ¯Ω l , Ω l , Ω F , such that
[Equation]
where ¯ θ rt and ¯ W lrt are defined in (6) .
Then we have
[Equation]
Remark 3.
Note that (14) is a PL-type inequality Karimi et al.
and requires the special structure of the network that satisfies Assumption 2 Nguyen and Hein (2018)
; Nguyen and Mondelli (2020).
Also, (15) can be proven using Assumption 3.
Now, we utilize the results of Steps 1 - 2 and Lemma 3 to derive the convergence of FedAvg.
Proposition 1.
Use Algorithm 1 to minimize (3) .
Suppose Assumptions 1, 2 and 3 are satisfied, and for each iteration rt + v, v ∈ { 0 , 1 , · · · , r − 1 } , θ rt + v k satisfies the conditions in Lemmas 1 and 2; and for each communication round rt , ¯ θ rt satisfies conditions in Lemma 3, then ∃ η > 0 such that
[Equation]
Remark 4.
Proposition 1 above shows that, if the conditions in Lemmas 1, 2 and 3 are satisfied, i.e., we have well-behaved gradients (Lemmas 1 and 2) and PL condition (Lemma 3), we achieve linear convergence of expected loss function for solving (3) with FedAvg-SGD.
We outline the major steps in the proof of Proposition 1.
Proof Sketch.
Consider the t th communication round, and suppose the singular values of the parameters satisfy (13), then it is easy to show that Φ( ¯ θ rt ) is Lipschitz smooth with some constant Q > 0 .
Then using the Lipschitz smoothness of Φ(¯ θ rt ) , we get
[Equation]
Taking expectation on both sides and conditioning on ¯ θ rt and the past, we get the following
[Equation]
Now we bound each term in (17) using Lemmas 2 and 3.
We first use the upper and lower bounds in Lemma 3 to bound the gradient norm.
First, to bound the second term on the right hand side (rhs) of (17) we use the PL-inequality in (14) of Lemma
[Equation]
[Equation]
We bound gradient norm in the third term using the upper bound of gradient in (15) of Lemma
[Equation]
[Equation]
Additionally, we use (11) in Lemma 2 to bound the gradient drift in the third term, we get ∥
[Equation]
Next, using (10) in Lemma 2 to bound fourth term on the rhs
[Equation]
the sum of stochastic gradient as
[Equation]
Finally, plugging the bounds for each term in (17)
[Equation]
using the definition of loss function Φ(¯ θ rt )
= 12 N ∥ f L (¯ θ rt ) − y ∥
22 along with the choice of step-size η < γ
[Equation]
, we get
[Equation] [Equation]
Using the above inequality recursively
[Equation]
we get the statement of Proposition 1.
Now Step 3 is complete and we move on to define the initialization strategy of Step 4.
It is important to note
that Proposition 1 utilized Lemmas 1 – 3, all of which impose some conditions on the singular values of the model parameters and the outputs of the first layer at each client during the entire training phase.
Next, we define the initialization strategy that ensures that the conditions of Lemmas 1 – 3 are satisfied almost surely.
Next, we go to Step 4, and discuss the initialization strategy.
Define λ
l := σ min (cid:0) W 0 l (cid:1) and
[Equation]
We also define the largest and smallest singular values of the output of the first layer at initialization for each client as α 0 ,k :=
σ min (cid:16) σ (cid:16) X k W 01 ,k (cid:17)(cid:17) .
Similarly, for the centralized setting when all the clients share the same parameter and full data, we define α 0 :
= σ min (cid:0) σ (cid:0) XW 01 (cid:1)(cid:1) .
Initialization Strategy:
Given any ϵ <
Φ( θ 0 ) , we initialize the model weights such that for some constants M 1 , M 2 , M 3 > 0 , the following are satisfied
[Equation]
To satisfy the required initialization, we follow the initialization strategy of Nguyen and Mondelli (2020).
First, randomly initialize (cid:2) W 01 (cid:3) ij ∼ N (0 , 1 /d 2in ) .
Broadcast [ W 01 ] ij to each client and collect F 1 ,k , which is the output of the first layer of each client, as well as the norm of local data ∥
X k ∥ F .
With F 1 ,k , α 0 and α 0 ,k can be computed.
For (23), since we have n 1 > N , α 0 and α 0 ,k are strictly positive.
Then it is easy to verify that given ϵ > 0 , (23) and the second relation in (24) will be satisfied if we choose large enough ¯
λ 1 → L min l ∈
[ L ] ¯ λ l .
This can be realized by choosing arbitrarily large ¯ λ l , l ∈ { 3 , · · · , L } .
In order to satisfy the first relation in (24), we need to make ¯ λ l and λ l close to each other.
Intuitively, one way is to construct (cid:0) W 0 l (cid:1)
L l =3 such that λ l = ¯
λ l = ζ > 1 , where ζ can be chosen to be any large number such that (23) and the second relation in (24) are satisfied.
We also need to upper bound Φ( θ 0 ) .
This can be done by choosing small W 02 .
Randomly initialize
W 02 such that (cid:2)
W 02 (cid:3) ij ∼ N (0 , κ ) .
We can set κ to be arbitrarily small, then Φ( θ 0 ) is bounded by 2 N ∥ y ∥ 22 with high probability (see (10) in Nguyen and Mondelli (2020)).
Note that the desired error ϵ is another key constant in the initialization.
When we expect the error to be small, (23) and the second relation in (24) will be more strict.
But this is not an issue since we can choose a larger ζ such that the initial conditions are satisfied.
The detailed initialization strategy that ensures that the conditions of Lemmas 1, 2 and 3 are satisfied is given in the Appendix B.2.
Next, let us state our main result, which indicates the linear convergence of local SGD to any ϵoptimal solution (see Definition 1).
The proof is attached in Appendix B.3.
Theorem 1.
Using FedAvg-SGD to minimize (3) with Algorithm 1.
Suppose Assumptions 1, 2 and 3 are satisfied, then there exists an initialization strategy such that for any ϵ <
Φ( θ 0 ) , there exists step-size η > 0 such that we have (where µ ′ :
= r 2 N γ
( L − 2) (cid:0) 12 (cid:1) 2( L − 1) λ 23 → L α 20 , and ηµ ′ < 1 )
[Equation]
Theorem 1 shows that, for any ϵ > 0 , we can always find an initialization, such that FedAvgSG achieves an ϵ accuracy within O (cid:0) log( 1 ϵ ) (cid:1) rounds of communication.
Notice that there is no heterogeneity assumption on the data (see Remark 2), and no assumption on the Lipschitz gradient of the loss function.
Remark 5.
We comment on the key novelties of this work compared to Nguyen and Mondelli (2020).
(1) Our work requires a careful analysis to deal with multiple local updates at each client.
Note that in contrast to Nguyen and Mondelli (2020), for our algorithm there is no guarantee that the overall objective will always decrease during local updates.
In fact, our analysis demonstrates that the overall objective can increase after each local iteration, we show that this increase will be compensated by the descent in the objective value between each communication round.
(2) Our algorithm and analysis can deal with the stochastic gradients for conducting local updates, while Nguyen and Mondelli (2020) only considered gradient descent in a centralized setting.
A key step in our analysis is to characterize the relationship between the stochastic and full gradient updates, which is illustrated in Lemma 1.
Remark 6.
We comment on the choice of parameters and the convergence rate.
As will be shown in Appendix B.3, by utilizing our initialization strategy, we can choose η = c/µ ′ for some constant c ∈ (0 , 1) (independent of ϵ ).
This implies that µ ′ η
= c < 1 , which further implies that we have (1 − µ ′ η ) < 1 in Theorem 1, ensuring linear convergence of FedAvg-SGD.
Finally, we present the convergence guarantees for the case when FedAvg-GD is utilized.
Corollary 1.
Using FedAvg-GD to minimize (3) with Algorithm 1.
Suppose Assumptions 2 andare satisfied, then there exists an initialization strategy and step-size η > 0 , such that we have
[Equation]
Remark 7.
Corollary 1 implies that FedAvg-GD achieves linear convergence when optimizing (3) .
We note that the result of Corollary 1 is much stronger compared to Theorem 1 as the initialization for FedAvg-GD is independent of ϵ compared to the one for FedAvg-SGD (shown in Appendix B.2).
N UMERICAL E XPERIMENTS
In this section, we analyze the effect of increasing the network sizes on popular image classification tasks with MNIST, Fashion MNIST and CIFAR-10 data sets.
We compare the performance of
FedAvg in both homogeneous (i.i.d.) and heterogeneous (non-i.i.d.)
data settings.
Through our experiments we establish that larger sized networks uniformly outperform smaller networks under different settings.
Next, we discuss the data and the model setting for our experiments.
Data set: For MNIST, Fashion MNIST and CIFAR-10 data sets, we split the data set among K = 100 clients.
For the homogeneous (i.i.d.) setting, we randomly distribute the complete data set with 60 , 000 samples to each client.
To model the heterogeneous (non i.i.d.) setting, we split the clients into two sets.
One set of clients receive randomly drawn samples while the second set of clients receive data from only two out of ten labels McMahan et al.
For our experiments on MNIST and Fashion MNIST data, 70% of the users receive non-i.i.d samples, while for CIFAR-10 data, the fraction is 20% .
Results and Discussion
For each setting, we compare the training loss and testing accuracy of FedAvg on smaller and larger sized networks.
To analyze the effect of network sizes on the stability of FedAvg, we also plot the performance of FedAvg averaged over 10 iterations for non-i.i.d.
client data setting for all the network architectures.
From our experiments, we make a few observations.
First, we observe from Figures 1 and 2 that in all the cases, the i.i.d setting has more stable performance (lower variance) than non-i.i.d setting.
Second, we note that the larger network uniformly outperforms the smaller network under all the settings.
Third, we note from the box plots in Figures 1 and 2 that the performance of the larger networks have lower variance, hence more stable performance compared with what can be achieved by the smaller networks.
Finally, we compare the random initialization with special initialization strategy which satisfies (23), (24).
We can conclude from Figure 3 that these two initialization are similar in test performance.
A PPENDIX A R ELATED W ORK
Overparameterized Neural Networks.
The surprising performance of overparameterized neural networks has raised significant research interest in the ML community to analyze the phenomenon of overparameterization Belkin et al.
Consequently, a number of works have analyzed the performance of centralized (stochastic) gradient descent (S)GD on overparameterized neural network architectures under different settings Jacot et al.
; Li and Liang (2018); Arora et al.
Du et al.
(2018; 2019); Allen-Zhu et al.
; Zou and Gu (2019); Nguyen and Mondelli (2020);
Nguyen (2021).
The authors in Jacot et al.
(2018), showed that an infinite width neural network when trained using gradient descent (GD) behaves like a kernel method with the kernel defined as neural tangent kernel (NTK).
Using this NTK parameterization Li and Liang (2018) showed that deep neural networks trained using GD require Ω( N 4 ) width to find the global optimal.
This result was later improved to Ω( N 3 ) in Huang and Yau (2020).
The authors in Du et al.
and
Du et al.
(2019) also analyze the performance of GD on overparameterized neural networks under different settings.
Under standard parameterization, the work Allen-Zhu et al.
(2019) studied the convergence of SGD and showed that network width of Ω( N 24 ) suffices to guarantee linear convergence.
Recently, Nguyen and Mondelli (2020) and Nguyen (2021) have improved the dependence on the width and have shown that GD requires only Ω( N ) width to achieve linear convergence.
All the works mentioned above focus on the centralized setting, and therefore, do not deal with data heterogeneity problem.
B P ROOF OF M AIN R ESULT B. P ROOF OF L EMMAS
We define some additional notations before we state some lemmas which are needed in the proof.
Let ⊗ denote the Kronecker product, and denote Σ l :=
[Equation]
] ∈
R Nn l × Nn l , Σ l
[Equation]
k
[Equation]
(cid:104) vec (cid:16)
[Equation]
F l −
[Equation]
k
[Equation]
l
[Equation]
k (cid:17)(cid:17)(cid:105) ∈
R mn l × mn l .
Define f rt + v L,k
[Equation]
Lemma 4.
Nguyen and Mondelli (2020) Suppose Assumptions 2 and Assumption 3 are satisfied
[Equation]
Then for l ∈ [ L ] the following holds:
g l,k =
[Equation]
∂f L
[Equation]
k
[Equation]
∥ g 2 ,k ∥
[Equation]
∥ F l,k
∥ F ≤ ∥ X k ∥ F
[Equation]
(cid:13)
[Equation]
∥
g k ∥ 2 ≤
[Equation]
∥ X k ∥
[Equation]
Furthermore, given with θ ak and θ bk , if ¯Λ l ≥ max (cid:16) σ max (cid:16) W al,k (cid:17) , σ max (cid:16) W bl,k (cid:17)(cid:17) for some scalars ¯Λ l .
Let R =
L (cid:81) p =1 max
[Equation]
1 , ¯Λ p (cid:1) .
Then, for l ∈ [ L ] ,
[Equation]
The above Lemma follows Lemma 4.1 Nguyen and Mondelli (2020): (26) gives the expression of the vectorized gradient; (27) provides the vectorized Jacobian matrix of the output of the network; (28) gives a lower bound on the norm of the gradient, which holds under Assumption 2.
(29) provides an upper bound on the norm of output of each layer while (30) gives an upper bound on the norm of gradient of each layer; (32) derives the Lipschitz constant of the networks and (33) provides the Lipschitz constant for the Jacobian of each layer.
Similar results can be derived in centralized optimization problem, so we do not include the results here.
Lemma 5.
(Nguyen and Mondelli, 2020, Lemma 4.3) Let f : R n → R be a C 2 function.
Let x, y ∈ R n be given, and assume
that ∥∇ f ( z ) − ∇ f ( x ) ∥ 2 ≤ C ∥ z − x ∥ 2 for every z = x + t ( y − x ) with t ∈ [0 , 1] .
Then,
[Equation]
Lemma 6.
For constant C, µ, ρ , if η → 0 , we have
[Equation]
Furthermore, given ϵ < Φ( θ 0 ) , let T =
(cid:22)
[Equation]
, then there exists constant ξ , such that
[Equation]
where ξ ≥ e 3 ρ 2 µ is a constant dependent on ρ and µ .
Proof.
Take logarithm on both sides, we get
[Equation]
Now let η → 0 , by L’Hôpital’s rule, take derivative over η , we have
[Equation]
Next, if we can show the function of η , which is √ 1 + 3 ρCη
[Equation]
1 1 − µCη , has a limit when η → min( 1 ρC , 1 µC ) , then by the continuity, it has an upper bound in (cid:16) 0 , min( 1 ρC , 1 µC ) (cid:17) , denote it as ξ .
It is easy to derive that
[Equation]
Then by the continuity of the function, (cid:0) √ 1 +
[Equation]
is bounded by some constant ξ .
Then we can derive
[Equation]
then we have there exists some constant ξ ≥ e 3 ρ 2 µ , such that
[Equation]
Lemma 7.
Let Assumption 2 and Assumption 3 hold.
For θ k , suppose there exists constant ¯Λ l , Λ l , Λ F such that
and
[Equation]
then we have
[Equation]
Proof.
By definition, we have
[Equation] [Equation]
Since by (31) and (28) in Lemma
[Equation]
we have
[Equation]
where X k is the sampled data at θ k .
So we can derive
[Equation]
where the last inequality is because ∥ ˜
X k ∥ F ≤ ∥ X ∥ F and ∥ ˜ f L,k ( θ ) − ˜ y k
∥ 2 ≤ ∥ f L,k ( θ ) − y k ∥ 2 .
Lemma 8.
For the FedAvg-SGD algorithm
[Equation]
given step size η > 0 , v ∈ { 0 , 1 , . . .
, r − 1 } and q ∈ { 0 , 1 , . .
. , v − 1 } .
Suppose the following conditions hold:
[Equation] [Equation]
[Equation]
then we have
[Equation]
Further, there exists constant Q k , such that ∀ k ∈
[ K ] we have
[Equation] [Equation]
and
[Equation]
Proof.
First, let us show (49).
[Equation]
So we can derive (49).
Next, we show (50).
Let us denote Jf rt + q L,k := (cid:20) ∂f rt + q L,k ∂ vec( W 1 ,k ) , . .
. , ∂f rt +
[Equation]
vec
[Equation]
W
[Equation]
k ) (cid:21) .
By triangle inequality, we have (cid:13)(cid:13)(cid:13) g rt + q +1 k − g rt + q k (cid:13)(cid:13)(cid:13) 2 = (cid:13)(cid:13)(cid:13)
Jf rt + q +1 L,k (cid:16)
f rt + q +1 L,k − y k (cid:17) − Jf rt + q L,k (cid:16) f rt + q L,k − y k (cid:17)(cid:13)(cid:13)(cid:13) ≤
(cid:13)(cid:13)(cid:13) f rt + q +1 L,k − f rt + q L,k (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)
Jf rt + q +1 L,k (cid:13)(cid:13)(cid:13)
+ (cid:13)(cid:13)(cid:13)
Jf rt + q +1 L,k − Jf rt + q L,k (cid:13)(cid:13)(cid:13)
(cid:13)(cid:13)(cid:13)
f rt + q L,k − y k (cid:13)(cid:13)(cid:13)(59)
Now we find the bound for each term in (59).
Since max (cid:16) σ max (cid:16) W rt + q +1 l,k (cid:17) , σ max (cid:16) W rt + q l,k (cid:17)(cid:17) ≤ ¯Λ l , by (32) in Lemma 4, we get (cid:13)(cid:13)(cid:13) f rt + q +1 L,k − f rt + q L,k (cid:13)(cid:13)(cid:13) 2 ≤ √ L ∥
X k ∥ F ¯Λ 1 → L min l ∈ [ L ] ¯Λ l (cid:13)(cid:13)(cid:13) θ rt + q +1 k − θ rt + q k (cid:13)(cid:13)(cid:13)
[Equation]
Further
[Equation]
by (27) we have
[Equation]
Using (33) in Lemma 4, we have
[Equation]
) where R ′ =
(cid:81)
Lp =1 max (cid:0) 1 , ¯Λ l (cid:1) .
So plug the above bounds into (59).
Set Lipschitz constant
[Equation]
k =
[Equation]
then we derive
[Equation]
Now (50) is proved.
Last, we prove (51).
We have
[Equation]
[Table]
≤
[Table]
K
[Table]
k
[Table]
N k
N (cid:13)(cid:13)(cid:13)
[Table]
− g
[Table]
k (cid:13)(cid:13)(cid:13)
[Table]
≤
[Table]
q =
K
[Table]
k =
N k
N Q k (cid:13)(cid:13)(cid:13) θ
[Table]
− θ
[Table]
(cid:13)(cid:13)(cid:13)
[Table]
= v
[Table]
q =
[Table]
K
[Table]
k =
[Table]
N k
N Q k · η (cid:13)(cid:13)
˜
g
[Table]
(cid:13)(cid:13)
[Table]
k =
[Table]
Q k
N L ∥
X k
[Table]
F ¯Λ
[Table]
→
[Table]
min l ∈ [ L ] ¯Λ l (cid:13)(cid:13) f
[Table]
− y k (cid:13)(cid:13)
[Table] [Table]
ηL
N ¯Λ
[Table]
L min l ∈
[ L ] ¯Λ l
K
[Table]
k =
Q k ∥
X k ∥ F v −
[Table]
q =
A q 2 (cid:13) (cid:13) f rt + q L − y k
(cid:13)(cid:13)
2≤ ηL
N ¯Λ 1 → L min l ∈ [ L ] ¯Λ l
A v +12 − 1 √
A −
K (cid:88) k
=
Q k ∥
X k ∥ F (cid:13)(cid:13)
f rt + q L − y k
(cid:13)(cid:13) 2(
[Equation]
where (i) uses triangle inequality; (ii) uses the Lipschitz gradient assumption in condition 2; (iii) comes from (31) in Lemma 4; (iv) uses condition 3; (v) is from Cauchy-Schwartz inequality.
B. I NITIALIZATION S TRATEGY
Detailed Initialization for FedAvg-SGD: Denote
[Equation]
Suppose given any small ϵ such that ϵ <
Φ( θ 0 )
[Equation]
the initialized weights satisfies the following conditions:
[Equation]
where ξ ≥ e 3 ρ 2 µ is some constant dependent on ρ and µ .
Now we provide a detailed way to realize the above initialization condition.
To satisfy the required initialization, we follow the initialization strategy of Nguyen and Mondelli (2020).
First, randomly initialize (cid:2) W 01 (cid:3) ij ∼ N (0 , 1 /d 2in ) .
Broadcast [ W 01 ] ij to each client and collect F 1 ,k , which is the output of the first layer of each client, as well as the norm of local data ∥
X k ∥ F and norm of local label ∥
y k ∥ 2 .
With F 1 ,k , α 0 and α 0 ,k can be computed.
For (23), since we have n 1 > N , α 0 and α 0 ,k are strictly positive with probability 1 .
Then it is easy to verify that given ϵ > 0 , (23) and the second relation in (24) will be satisfied if we choose large enough ¯
λ 1 → L min l ∈
[ L ] ¯ λ l .
This can be realized by choosing arbitrarily large λ l , l ∈ { 3 , · · · , L } .
However, notice that by Lemma 6, the constant ξ , which is defined in (39), is only dependent on ρ and µ and ξ ≥ e 3 ρ 2 µ .
So if we can fix ρ and µ as some constants, ξ is a bounded constant.
Notice in (66) and (67), for l ∈ { 3 , · · · , L } , if we can make ¯ λ l and λ l close to each other, then ρ and µ are also close, so 3 ρ 2 µ is not large.
This is equivalent to the first relation in (24) in main text.
In order to satisfy the above conditions, one way is to construct (cid:0) W 0 l (cid:1)
L l =3 in such way that λ l = ¯
λ l = ζ > 1 , where ζ can be chosen to be any large number such that (23) and the second relation in (24) are satisfied.
Specifically, we can utilize the following construction: Initialize W 0 l such that its top block is a scaled identity matrix and rest of entries are zero
[Equation]
We also need to upper bound Φ( θ 0 ) .
This can be done by choosing small W 02 .
Randomly initialize
W 02 such that (cid:2)
W 02 (cid:3) ij ∼ N (0 , κ ) .
We can set κ to be arbitrarily small, similar to (10) in Nguyen and Mondelli (2020),we can find a bound for Φ( θ 0 ) with high probability:
[Equation]
Then the loss function at initialization can be bounded by constant (cid:112) 2 N Φ( θ 0 ) ≤ 2 ∥ y ∥ 2 .
Initialization for FedAvg-GD: The initialized weight matrices satisfy the following conditions:
[Equation]
The initialization strategy is similar to FedAvg-SGD, so we omit the discussion here
[Equation]
B. P ROOF OF T HEOREM
Theorem 1.
Using FedAvg-SGD to minimize (3) with Algorithm 1.
Suppose Assumptions 1, 2 and 3 are satisfied, then there exists an initialization strategy such that for any ϵ <
Φ( θ 0 ) , there exists step-size η > 0 such that we have
[Equation]
where µ ′ = rN γ 2( L − 2) (cid:0) 12 (cid:1) 2( L − 1) λ
→ L α 20 .
Proof.
First, we provide a structure of our proof.
We will show the following recursively at each communication round: 1)
The averaged weights are bounded at each communication round; 2)
The divergence of loss function (3) is bounded at each communication round; 3)
The expected loss function (3) decreases linearly at each communication round.
Further, we will show that in each local epoch within a fixed communication round, we have: 1) The weights of each client are bounded; 2)
The divergence of loss function Φ k of each client is bounded.
Now let us set T
= (cid:106) log ( Φ( θ 0 ) /ϵ )
log (
[Equation]
+
[Equation]
(cid:107) .
If we can show (75) holds for t = 0 , . .
, T , then it is easy to show that
[Equation]
We prove Theorem 1 by induction.
Define
[Equation]
We show that ∀ t ≤ T , we have
[Equation]
where ¯ λ l is defined in (22) and λ l is the smallest eigen value of the weight matrix, C, µ, ρ defined in
B.2 and µC = µ ′ .
The above recursive equation describes the weight matrix and loss function in each communication round.
To prove (77)
[Equation]
we decompose the recursive equation into two steps, as follows
Step1: For a fixed t and v ∈ [ r − 1] , given
[Equation]
we aim to show  
[Equation]
θ rt + q k (cid:1) ≤ (1 +
[Equation]
η ) q
[Equation]
( θ rtk ) , q
[Equation]
, v } , k ∈
[ K ] .
, (79) where C ′ =
max k (cid:16) 1 N k (cid:0) 74 (cid:1) 2( L − 1) ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l (cid:17) .
Step 2: Given (78) and (79), we show 
[Equation]
Now we show Step 1 first.
(1) We first show
[Equation]
q ∈ { 0 , . .
. , v } , l ∈ [ L ] , k ∈
[ K ] .
We have
[Equation]
where (i) is because the norm of concentated gradient is no smaller than norm of one-layer gradient; (ii) results from Lemma (1); (iii) comes from (31) in Lemma 4; (iv) is because of the induction assumption.
Let η < 1 ρC ′ , we have
[Equation]
where (i) uses η < 1 ρC ′ ; (ii) is because ∥ f rt L,k − y k
∥ 2 ≤ ∥ f rtL − y ∥ 2 ; the last inequality holds if we choose small enough η .
To be more specific, we can choose
[Equation]
By Weyl’s inequality
[Equation]
we have
[Equation]
We next show that
[Equation]
It is sufficient to show σ min (cid:16) F rt + v 1 ,k (cid:17) ⩾ 14 α 0 ,k , k ∈ [ K ] .
[Equation]
where (i) results from the Lipschitz gradient of σ in Assumption 3 and (ii) comes from (86).
If we choose small enough η , which satisfies
[Equation]
then we have
[Equation] [Equation]
(3) Next, we show that
[Equation] [Equation]
Since we have already shown in (81) that σ max ( W rt + v l,k ) ≤ 74 ¯ λ l , σ min ( W rt + v l,k ) ≥ 14 ¯ λ l
and we have shown in (89) that σ min ( F rt + v) ≥ 14 α 0 ,k .
By lemma 1, we have
[Equation]
(4) Next, we prove
[Equation]
We show
[Equation]
First, we need to show Φ k has Lipschitz gradient within [ θ rt + v − 1 , θ rt + v ] .
This is similar to the proof of (50) in Lemma 8.
So we don’t include the details here.
It is easy to show that, for θ rt + v − 1 ,s k := θ rt + v − 1 k
+ s
[Equation]
− θ
[Equation]
) , there is max (cid:16) σ max (cid:16) W rt + v − 1 ,s l,k (cid:17) , σ max (cid:16) W rt + v − 1 l
[Equation]
k (cid:17)(cid:17) ≤ 74 ¯ λ l .
So similarly we can derive the Lipschitz constant
[Equation]
N k ∥
X k ∥ F (1 + Lβ ∥
X k ∥ F R ′ )
R ′ (cid:13) (cid:13) f
0 L,k − y k (cid:13) (cid:13) 2 , (99) such that ∀ s ∈ [0 , 1] , (cid:13)(cid:13)(cid:13) g rt + v − 1 ,s k − g rt + v − 1 k (cid:13)(cid:13)(cid:13) 2 ≤ Q k ∥ θ rt + v − 1 ,s k − θ rt + v − 1 k ∥ 2 .
With Lipschitz gradient within [ θ rt + v − 1 k , θ rt + v k ] , by Lemma 5, we have Φ k (cid:0)
θ rt + v k (cid:1) ≤ Φ k (cid:0) θ rt + v − 1 k (cid:1) + ⟨∇ Φ k (cid:0) θ rt + v − 1 k (cid:1) , θ rt
+ v k − θ rt + v k (cid:11) + Q k 2 (cid:13)(cid:13)
θ rt + v − 1 k − θ rt + v − 1 k
(cid:13)(cid:13) 2(101)
=
Φ k (cid:0) θ rt + v − 1 k (cid:1) + (cid:10) g rt + v − 1 k , − η ˜
g rt + v − 1 k (cid:11) + Q k 2 (cid:13)(cid:13)
η ˜ g rt + v − 1 k
(cid:13)(cid:13) 2(102) ≤ Φ k (cid:0)
θ rt + v − 1 k (cid:1) + η (cid:13)(cid:13)
g rt + v − 1 k
(cid:13)(cid:13) 2 (cid:13)(cid:13) ˜
g rt + v − 1 k
(cid:13)(cid:13) 2 + Q k 2 η 2 (cid:13)(cid:13)
˜
g rt + v
[Equation]
Let η < 1 Q k ρ , we have the above inequality
[Equation]
where the third inequality comes from (31) in Lemma 4.
Recall C ′
[Equation]
= max
[Equation]
we have
[Equation]
Now Step 1 is proved.
Next we show Step 2.
(1) Show
[Equation]
Define ˜ ∇ W l ,k Φ k ( θ rtk ) be the stochastic gradient over layer l of each client.
Denote ¯˜ g l,rt + v :=
[Equation]
) We have
[Equation]
By Step 1, we know for v ∈ { 0 , 1 , . . .
, r − 1 } , we have ρ rt + v ≤ ρ.
So by definition of ρ rt + v , we have ∥
˜
g ru + v ∥ 2 ≤ ρ ∥ g ru + v ∥ 2 .
Then it is easy to verify that the assumptions in Lemma 8 are satisfied, where ¯Λ l = 74 ¯ λ l , Q k is defined in (63) and A = 1+3 ρC ′ η .
Then by Lemma 8, if η < 1 ρC ′ , we have η t
[Equation] [Equation]
Using the definition of P =
L ∥
X ∥ F N
[Equation]
74 (cid:1)
L − 1 (2 r − 1) , we have
[Equation]
where the last inequality comes from the induction assumption.
Now let S = √ 1 + 3 ρCη , if we choose η < 1 ρC , we get
[Equation]
By Lemma
[Equation]
we have S T ≤ ξ Φ ( θ 0 ) ϵ .
Additionally, S ≤ 2 , therefore, we have
[Equation]
1 , l ∈ { 1 , 2 } .
, where the last inequality is from (68).
So by Weyl’s ineuality
[Equation]
we have 
[Equation] [Equation]
121) (2) Show σ
[Equation]
Similarly, we have
[Equation] [Equation]
[Equation]
where (i) is because σ is 1 − Lipschitz; (ii) comes from (120), and (iii) is because (69) in B.2.
So similarly by Weyl’s inequality, we have σ min (cid:16) F
[Equation] [Equation]
(3) Show
[Equation]
Similarly, we have
[Equation]
133) where (i) is because σ is 1 − Lipschitz; (ii) comes from (120), and (iii) is because (69) in B.2.
So similarly by Weyl’s inequality, we have σ min (cid:16) F r
[Equation]
+
[Equation]
1 ,k
[Equation]
(3) Show
[Equation]
Since we have already shown in (81) that σ max (cid:16) W r ( t +1) l,k (cid:17) ≤ 32 ¯ λ l , σ min (cid:16) W r ( t +1) l,k (cid:17) ≥
12 ¯ λ l , and we have shown in (89) that σ min ( F r ( t +1)) ≥ 12 α 0 .
By Lemma 1, we have
[Equation]
(135) (4) Show
[Equation]
First, similar to the proof of (50), we can derive
[Equation] [Equation]
where R = L (cid:81)
[Equation]
max
[Equation]
such that ∀ ¯ θ rt,s =
¯ θ rt + s (¯ θ r ( t +1) − ¯ θ rt ) , s ∈ [0 , 1] , we have
[Equation]
Then by Lemma 5 we have
[Equation]
By (49) in Lemma 8, if η < 1 ρC , we have A = 2 , and we have
[Equation]
Then we have
[Equation]
∥
f rtL − y ∥(142)
( i ) ≤ Φ (cid:0) ¯ θ rt (cid:1) + ηρPL ∥
X ∥ F
N (cid:18) (cid:19)
L − 1 ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l ∥ f rtL − y ∥
22 + Q 2 η 2 ρ 2 P 2 ¯ λ 21 → L min l ∈
[ L ] ¯ λ 2 l ∥ f rtL − y ∥ 22 , (143) where (i) comes from (31) in Lemma 4.
Let η < L ∥ X
∥ F ( 32 ) L − 1 QρPN =
( 67 ) L − 1 Qρ (2 r − 1) , we get Φ (cid:16) ¯ θ r ( t +1) (cid:17) ≤ Φ (cid:0) ¯ θ rt (cid:1) + ηρPL ∥
X ∥ F
N (cid:18) (cid:19)
L − 1 ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l ∥ f rtL − y ∥
22 + Q 2 η 2 ρ 2 P 2 ¯ λ 21 → L min l ∈
[ L ] ¯ λ 2 l ∥ f rtL − y ∥(144) ≤ Φ (cid:0) ¯ θ rt (cid:1) + 3 2 · ηρPL ∥ X ∥ F N (cid:18) (cid:19)
L − 1 ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l ∥ f rtL − y ∥≤ Φ (cid:0) ¯ θ rt (cid:1)   1 + 3 ηρPL
∥ X ∥ F
N (cid:18) (cid:19)
L − 1 ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l  
Recall C = PL ∥
X ∥ F (cid:0) 32 (cid:1) L − 1 ¯ λ 21 → L min l ∈ [ L ] ¯ λ 2 l , then we have Φ (cid:16) ¯ θ r ( t +1) (cid:17) ≤ (1 +
[Equation]
(5) Show
[Equation]
By (138), we have
[Equation]
we get
[Equation]
Now it is easy to verify the assumptions in Lemma 1 are satisfied.
Let ¯Λ l = 74 ¯ λ l , Q defined in (136),
A = 1 + 3 ρC ′ η ≤ 2 , by (51) in Lemma 8:, we have
[Equation]
g
[Equation]
Plug (148) into (147), we get:
[Equation]
where (i) uses (31) to provide an upperbound for ∥ g rt ∥
[Equation]
(ii) uses (28) to provide a lower bound for ∥ g rt ∥ .
Let η < µ ′ 2 B
[Equation]
we have
[Equation]
Let
[Equation]
, we have
[Equation]
where µC = µ ′ .
Now we summarize the choice of η , it should be smaller than all the following quantities:
[Equation]
Remark 8.
To satisfy the initialization assumptions defined in (69) and (68) , we initialize the neural network coefficients such that we have ¯ λ 1 → L ∼ O (1 /ϵ ) .
Note from the definition of µ ′ in (152) that this implies µ ′ ∼ O
( λ 23 → L ) =
O (¯ λ 21 → L ) =
O (1 /ϵ 2 ) .
Also, note from the choice of step-size in (153) that, we have η ∼ O
( ϵ × 1 / ¯ λ 1 → L )
=
O ( ϵ 2 ) .
Note that this follows from the fact that η is smaller than each quantity defined in (153) above.
Thus, we have µ ′ η
= O (1)
and we can always choose η = c/µ ′ for some c ∈ (0 , 1) , which guarantees linear convergence of the objective in each communication round (see Theorem 1).
C E XPERIMENT S ETTING AND R ESULT C. M ODEL AND P ARAMETER S ETTINGS :
To analyze the performance of FedAvg-SGD on the MNIST data set, we use a single hidden-layer fully-connected neural network (MLP) with ReLU activation.
We set the hidden-layer size to be 32 (resp.
1 , 000 ) for the small (resp.
large) network.
We choose the mini-batch size m = 10 and choose the number of local steps to be r = 10 .
Using the above network, we also compare the random initialization with the special initialization strategy in (23) , (24) with MNIST and Fashion
MNIST dataset.
For the CIFAR-10 data set, we analyze the performance of FedAvg-SGD on two network architectures – convolutional neural network (CNN) and ResNet.
We design the smaller
CNN using two 5 × 5 convolutional layers followed by 2 × 2 max pooling, each has 6 andchannels, connected by 2 fully-connected layers with 120 and 84 hidden neurons.
For larger CNN, we use three 3 × 3 convolutional layers each with 128 channels followed by 2 × 2 max pooling.
The ReLU activation function is used after each hidden layer for small/large CNN.
For ResNet, we compare the performance on ResNet18 with ResNet50 architectures.
For both the CNN and ResNet, we use a mini-batch size of m = 32 and number of local steps to be r = 5 .
We randomly sampleclients in each epoch and perform FedAvg-SGD for more efficient training.
C.
E XPERIMENT R ESULT FOR MNIST