R ECYCLING M ODEL U PDATES IN F EDERATED L EARNING : A RE G RADIENT S UBSPACES L OW -R ANK ?
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we question the rationale behind propagating large numbers of parameters through a distributed system during federated learning.
We start by examining the rank characteristics of the subspace spanned by gradients (i.e., the gradientspace) in centralized model training, and observe that the gradient-space often consists of a few leading principal components accounting for an overwhelming majority ( 95 − 99% ) of the explained variance.
Motivated by this, we propose the "Look-back Gradient Multiplier" ( LBGM ) algorithm, which utilizes this low-rank property of the gradient-space in federated learning.
Operationally, LBGM recycles the gradients between model update rounds to signiﬁcantly reduce the number of parameters to be propagated through the system.
We analytically characterize the convergence behavior of LBGM , revealing the nature of the trade-off between communication savings and model performance.
Our subsequent experimental results demonstrate the improvement LBGM obtains on communication overhead compared to several federated learning baselines.
Additionally, we show that LBGM is a general plug-and-play algorithm that can be used standalone or stacked on top of existing sparsiﬁcation techniques for distributed model training.
I NTRODUCTION
Federated Learning (FL) (Koneˇcn`y et al., 2016) has emerged as a popular paradigm for distributing machine learning (ML) across a set of computing devices.
By having each device conduct local model updates, FL substitutes raw data transmissions with model parameter transmissions, promoting data privacy (Shokri & Shmatikov, 2015; Azam et al., 2021) and reducing communication requirements (Wang et al., 2020a; Hosseinalipour et al., 2020).
At the same time, overparameterized neural networks (NN) are becoming ubiquitous in the ML models trained by FL, e.g., in computer vision (Ren et al., 2015;
He et al., 2016; 2017; Liu et al., 2016; Huang et al., 2017) and natural language processing (Brown et al., 2020; Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2019;
Liu et al., 2019a), which reintroduces communication burden.
While NNs have parameters in the range of a few million (AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016)) to several billions (GPT-3 (Brown et al., 2020), GPT-2 (Radford et al., 2019), Turing NLG (tur, 2020)), prior works have demonstrated that a majority of these parameters are often irrelevant (Frankle & Carbin, 2019; Liu et al., 2019b; Han et al., 2015; Li et al., 2017; LeCun et al., 1990) in optimization and inference.
This presents an opportunity to reduce communication in FL by transmitting lighter representations of the model, conventionally achieved through compression/sparsiﬁcation techniques (Wang et al., 2018;
Seide et al., 2014;
Alistarh et al., 2017; Vogels et al., 2019).
In this work, we investigate the “overparameterization” of NN training in FL systems, through the lens of rank characteristics of the subspaces spanned by the gradients (i.e., the gradient-space) generated during stochastic gradient descent (SGD).
Our ﬁrst fundamental question is: Can we observe the effect of overparameterization in gradient descent-based optimization of NNs?
And if so: Can this property be used to reduce communication overhead in FL?
Our main hypothesis is that the subspaces spanned by gradients generated over SGD iterations are low-rank.
( H1)
This leads us to propose a technique that instead of propagating a million/billion-dimensional vector (i.e., gradient) over the system in each iteration of FL only requires propagating a single scalar in the majority of the iterations.
Our algorithm introduces a new class of techniques to the literature which
[Figure]
are built on reusing/recycling the previous gradients to represent the newly generated gradients at the devices in FL.
The main contributions of this work can be summarized as follows:
• We demonstrate the low-rank characteristics of the gradient-space generated over consecutive SGD model training iterations for overparameterized NNs.
We show that the principal components of the gradient-space (i e., principal gradient directions) can be approximated in terms of actual gradients generated during the model training process.
• Our insights lead us to develop the “Look-back Gradient Multiplier” ( LBGM ) algorithm to improve the model training vs. communication overhead tradeoff in FL. LBGM recycles periodicallytransmitted gradients to represent the newly-generated gradients at each device with a single scalar.
We further analytically investigate the convergence characteristics of this algorithm.
• Our experiments show the communication savings obtained via LBGM in FL both as a standalone solution and a plug-and-play method used with other gradient compression techniques, e.g., top-K. We further reveal that LBGM can be extended to distributed training, e
g., LBGM with SignSGD (Bernstein et al., 2018) substantially reduces communication overhead in multi-GPU systems.
A G RADIENT -S PACE O DYSSEY
We ﬁrst experimentally study the gradient-space spanned by gradients generated during SGD model training of overparameterized NNs.
Given a centralized ML training task (e.g., classiﬁcation, regression, segmentation), we exploit principal component analysis (PCA) (Pearson, 1901) to answer the following: How many principal components explain the 99% and 95% variance (termed N 99- PCA and N 95- PCA , respectively; together N - PCA ) of all the gradients generated during model training?
We start with 4 different NN architectures: (i) fully-connected neural network (FCN), (ii) convolutional neural network (CNN), (iii) ResNet18 (He et al., 2016), and (iv) VGG19 (Simonyan &
Zisserman, 2014); trained on 2 different datasets: CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015), which have classiﬁcation and regression learning tasks, respectively.
We then compute the N - PCA for each epoch by applying PCA on the set of gradients accumulated until that epoch; refer to Algorithm 2 in Appendix C for the pseudocode.
The results depicted in Fig.agree with our hypothesis ( H1 ): both N 99- PCA and N 95- PCA are signiﬁcantly lower than that the total number of gradients calculated during model training, e.g., Fig.
1 shows that the number of principal components (red and blue lines in top plots) are substantially lower (often as low as 10% of number of epochs, i.e., gradients generated) for both datasets.
In Appendix D.1 , we further ﬁnd that ( H1 ) holds for additional experiments using several datasets: CIFAR-100 (Krizhevsky et al., 2009),
MNIST (LeCun & Cortes, 2010), FMNIST (Xiao et al., 2017), CelebA (Liu et al., 2015), PascalVOC (Everingham et al., 2010), COCO (Lin et al., 2014); models: U-Net (Ronneberger et al., 2015), SVM (Cortes & Vapnik, 1995); and tasks: semantic segmentation and multi-regression.
It is also worth noting (especially on CIFAR-10) that variations in N - PCA across models are not necessarily related to the model performance (CNN performs almost as well as ResNet18 but has much lower N - PCA ;
The fact rank deﬁciency not consequence of model complexity gradient-space of state-of- -art large-scale ML models could be principal gradient directions
N-PCA and FL.
In an “ideal” FL (McMahan et al., 2017) framework, if both the server and the workers/devices have the PGDs (principal components of the gradient-space), then the newly generated gradients can be transmitted by sharing the projections of the generated gradients on the PGDs, i e., the PGD multipliers (PGM).
PGMs and the PGDs can together be used to reconstruct the generated gradients of the devices at the server.
This would dramatically reduce the communication cost by sharing scalar PGMs instead of the gradient vector.
However, this setting is impractical since: (i) it is infeasible to obtain the PGDs prior to the training, and (ii) calculating PCA/SVD is computationally intensive.
We thus look for an efﬁcient online approximation of the PGDs.
Overlap of Actual Gradients and N-PCA.
To approximate PGDs, we exploit an observation made in Fig.
1 (and further in Appendix D.1): N - PCA mostly remains constant over time, suggesting that the gradients change gradually during SGD iterations.
To further investigate this, in Fig.
2 (and further in Appendix D.2)
we plot the cosine similarity between PGDs and actual gradients as a heatmap.
‡ We observe that (i) the similarity of actual gradients with PGDs varies gradually over time, and (ii) actual gradients have a high cosine similarity with one or more PGDs.
This leads to our second hypothesis: PGDs can be approximated using a subset of the actual gradients generated over time .
( H2) “Look-back Gradients” as PGD Approximations.
Our above-mentioned observations suggest that there should be a signiﬁcant overlap among consecutive gradients generated during SGD iterations.
This is further veriﬁed in Fig.
3 (and in Appendix D.3), where we plot the pairwise cosine similarity of consecutive gradients generated during SGD iterations.
‡ For example, consider the boxes marked B1, B2, ... in layer 1 (L#1 in Fig. 3) for reference.
Gradients in each box can be used to approximate other gradients within the box.
Also, interestingly, the number of such boxes that can be drawn is correlated with the corresponding number of PGDs.
Based on ( H2 ), we next propose our Look-back Gradient
Multiplier ( LBGM ) algorithm that utilizes a subset of actual gradients, termed “look-back gradients”, in order to reuse/recycle gradients to be transmitted in a federated learning (FL) framework.
L OOK - BACK G RADIENT M ULTIPLIER M ETHODOLOGY
Federated Learning (FL) considers a system of K workers/devices indexed 1 , ..., K , as shown in
Fig.
Each worker k possesses a local dataset D k with n k = |D
k | datapoints. The goal of the system is to minimize the global loss function F ( · ) expressed through the following problem:
[Equation]
where M is the dimension of the model θ , ω k
= n k /N , N = P
Kk [Equation] n k , and F k ( θ ) =
P d ∈D k f k ( θ ; d ) /n
k is the local loss at worker k , with f k ( θ ; d ) denoting the loss function for data sample d given parameter θ .
FL tackles (1) via engaging the workers in local SGD model training on their own datasets.
The local models are periodically transferred to and aggregated at the main server after τ local updates, forming a global model that is used to synchronize the workers before starting the next round of local model training.
During each global aggregation round t , worker k updates its parameters θ ( t,b ) k as: θ ( t,b +1) k ← θ
[Equation]
− η g k ( θ ( t [Equation] [Equation] ) , where g k ( θ ( t,b ) k ) is the stochastic gradient at local step b , and η is the step size.
During a vanilla FL aggregation, the global model parameters are updated as θ ( t +1) ← θ ( t ) − η
P Kk =1 ω k g ( t ) k , where g ( t ) k
= P τ − 1 b =
g k ( θ ( t,b ) k ) is accumulated stochastic gradient (ASG) at worker k .
More generally, this aggregation may be conducted over a subset of workers at time t , e.g., in a large-scale wireless network where only a fraction of devices can be active in each round (Ruan et al., 2021).
We deﬁne ∇ F k ( θ ( t ) k ) =
P
[Equation] and
d ( t ) k = g ( t ) k /τ as the corresponding accumulated true gradient and normalized ASG, respectively.
Indexing and Notations.
In superscripts with parenthesis expressed as tuples, the ﬁrst element denotes the global aggregation round while the second element denotes the local update round, e.g., g ( t,b ) k is the gradient at worker k at global aggregation round t at local update b .
Superscripts without parenthesis denote the index for look-back gradients, e.g., ℓ in ρ ( t ) ,ℓ k deﬁned below.
LBGM Algorithm.
LBGM (see Fig. 4) consists of three main steps: (i) workers initialize and propagate their look-back gradients (LBGs) to the server; (ii) workers estimate their look-back coefﬁcients (LBCs), i.e., the scalar projection of subsequent aggregated gradients on their LBGs, and the lookback phase (LBP), i.e., the angle between the accumulated gradient and the LBG; and (iii) workers update their LBGs and propagate them to the server if the value of LBP passes a threshold, otherwise they only transmit the scalar LBC.
Thus, LBGM propagates only a subset of actual gradients generated at the devices to the server.
The intermediate global aggregation steps between two LBG propagation rounds only involve transfer of a single scalar , i.e., the LBC, from each worker, instead of the entire vector of accumulated gradient.
In LBGM , the local model training is conducted in the same way as vanilla FL, while model aggregations at the server are conducted via the following rule:
[Equation]
where e g ( t ) k is the approximation of worker k ’s accumulated stochastic gradient, given by Deﬁnition 1.
Deﬁnition 1.
(Gradient Approximation in LBGM )
Given the accumulated stochastic gradient g ( t ) k , and the LBG g ℓk , the gradient approximation e g ( t ) k recovered by the server is given by:
[Equation]
( t +1) ← θ ( t ) − η P Kk =1 ω k h s k · µ ( t ) k · g ℓk + (1 − s k ) · µ ( t ) k
i , where s k is an indicator function given by, s k = ( 1 , if µ ( t ) k is a scalar 0 , otherwise, i.e., if µ ( t ) k is a vector . Update server-copy of look-back gradients (LBGs): g ℓk ← (1 − s k ) µ ( t ) k
+ ( s k ) g ℓk , ∀ k . 18: end for where LBC ρ ( t ) ,ℓ k
= h g ( t ) k , g ℓk i (cid:14)
k g ℓk k 2 is the projection of the accumulated gradient g ( t ) k on the LBG g ℓk , and α ( t ) ,ℓ k denotes the angle between g ( t ) k and g ℓk (see Fig. 4(a)).
LBGM initializes the LBGs g ℓk with the ﬁrst actual gradients propagated by the devices at t = 1 .
For the subsequent aggregations, each worker k shares only its LBC ρ ( t ) ,ℓ k with the server if the value of
LBP error sin 2 ( α ( t ) ,ℓ k ) is below a threshold, i.e., sin 2 ( α ( t ) ,ℓ k ) ≤ δ threshold k , where δ threshold k ∈ [0 , 1] is a tunable parameter; otherwise it updates the LBG via transmitting the entire gradient to the server.
The details of LBGM are given in Algorithm 1.
We next conduct convergence analysis for LBGM .
Assumptions: It is presumed that the local loss functions are bounded below: min θ ∈ R M F k ( θ ) >
−∞ , ∀ k .
Let k · k denote the 2-norm.
In the analysis we use the following assumptions (Wang et al., 2020a;
Friedlander & Schmidt, 2012; Hosseinalipour et al., 2020; Li et al.
; Stich, 2019):
Smoothness of Local Loss Functions: Local loss function F k : R M → R , ∀ k , is β -smooth:
[Equation]
SGD Characteristics: Local gradients g k ( θ ) , ∀ k , estimated by SGD are unbiased estimators of the true gradients ∇ F k ( θ ) , and have a bounded variance σ 2 ≥ 0 ; mathematically:
[Equation]
and E
[Equation]
Bounded Dissimilarity of Local Loss Functions:
For any realization of weights { ω k ≥ 0 } Kk =1 , where P Kk [Equation] ω k = 1 , there exist non-negative constants Υ 2 ≥ 1 and Γ 2 ≥ {+
+}
such that
[Equation]
Theorem 1.
(General Convergence Characteristic of LBGM )
Assume A1 , A2 , A3 , and that ηβ ≤ min (cid:8) 1 / (2 τ ) , 1 / (cid:0) τ p 2(1 + 4Υ 2 ) (cid:1)(cid:9) .
If the threshold value in step 7 of Algorithm 1 satisﬁes the condition δ threshold k
[Equation]
( α ( t ) ,ℓ k
[Equation]
∀ k ), where ∆ 2 ≥ 0 is a constant, then after T rounds of global aggregations, the performance of
LBGM is characterized by the following upper bound:
[Equation]
Proof.
The proof is provided in Appendix A.
The condition on δ threshold k and LBP error sin 2 ( α ( t ) ,ℓ k ) in the above theorem implies that to have a ﬁxed bound in (3), for a ﬁxed ∆ 2 , a larger gradient norm k d ( t )
k k 2 is associated with a tighter condition on LBP error sin 2 ( α ( t ) ,ℓ k ) and δ threshold k .
This is intuitive because a larger gradient norm corresponds to a larger estimation error when the gradient is recovered at the server for a given LBP (see Fig. 4).
In practice, since the gradient norm k d ( t ) k k 2 does not grow to inﬁnity during model training, the condition on LBP error in Theorem 1, i.e., sin 2 ( α ( t ) ,ℓ k ) ≤ ∆ 2 / k d ( t ) k k 2 , can always be satisﬁed for any ∆ 2 ≥ 0 , since transmitting actual gradients of worker k makes α ( t ) ,ℓ k = 0 .
Given the general convergence behavior in Theorem 1, we next obtain speciﬁc choice of step size and an upper bound on ∆ 2 for which LBGM approaches a stationary point of the global loss function (1).
Corollary 1.
(Convergence of LBGM to a Stationary Point)
Assuming the conditions of Theorem 1, if ∆ 2 introduced in Theorem 1 satisﬁes ∆ 2 ≤ η , where η = 1 / √ τT , then LBGM converges to a stationary point of the global loss function, with the convergence bound characterized below:
[Equation]
Proof.
The proof is provided in Appendix B.
Considering the deﬁnition of ∆ 2 in Theorem 1 and the condition imposed on it in Corollary 1, the
LBP error (i.e., sin 2 ( α ( t ) ,ℓ k ) ) should satisfy k d ( t ) k k 2 sin 2 ( α ( t ) ,ℓ k ) ≤ η = 1 / √ τT for LBGM to reach a stationary point of the global loss.
Since k d ( t ) k k 2 is bounded during the model training, this condition on LBP error can always be satisﬁed by tuning the frequency of the full (actual) gradient transmission, i.e. updating the LBG as in lines 10&11 of Algorithm 1.
Speciﬁcally, since the correlation across consecutive gradients is high and it drops as the gradients are sampled from distant epochs (see
Fig. 3), a low value of LBP error can be obtained via more frequent LBG transmissions to the server, which makes the generated gradients similar to the stored LBG at the server.
Main Takeaways of Theorem 1, Corollary 1, and Algorithm 1:
Recovering Vanilla-FL Bound:
In (3), if the LBGs are always propagated by all the devices, we have α ( t ) ,ℓ k = 0 , ∀ k , and thus ∆ 2 = 0 satisﬁes the condition on the LBP error.
Then, (3) recovers the bound for vanilla
FL (Wang et al., 2020a; Stich, 2019; Wang & Joshi, 2018).
Recovering Centralized-SGD Bound:
In (3), if the LBGs are always propagated by all the devices, i.e., ∆ 2 = 0 , the local dataset sizes are equal, i.e., w k = 1 /K, ∀ k , and τ = 1 , then (3) recovers the bound for centralized-SGD, e.g., see Friedlander & Schmidt (2012).
Unifying Algorithm 1 and Theorem 1: The value of ∆ 2 in (3) is determined by the value of the LBP error sin 2 ( α ( t ) ,ℓ k ) , which is also reﬂected in step 7 of Algorithm 1.
This suggests that the performance improves when the allowable threshold on sin 2 ( α ( t ) ,ℓ k ) is decreased (i.e., smaller ∆ 2 ), which is the motivation behind introducing the tunable threshold δ threshold k in our algorithm.
Effect of LPB Error on Convergence: As the value of sin 2 ( α ( t ) ,ℓ k ) increases, it can be seen that the term in (3) containing ∆ 2 will start diverging (it can become the same order as the gradient k d ( t ) k k 2 ).
In particular, the condition in Corollary 1 on ∆ 2 avoids such scenario, under which the convergence to a stationary point is achieved.
Performance vs. Communication Overhead Trade-off: Considering step 7 of Algorithm 1, increasing the tolerable threshold on the LBP error increases the chance of transmitting a scalar (i.e., LBC) instead of the entire gradient to the server from each worker, leading to communication savings.
However, as seen in Theorem 1 and the condition on ∆ 2 in Corollary 1, the threshold on LBP error cannot be increased arbitrarily since the LBGM may show diverging behavior.
[Figure] [Figure]
E XPERIMENTS
We run experiments using several NN architectures and datasets.
Speciﬁcally, we train: S1: CNN on
FMNIST, MNIST, CelebA and CIFAR-10, S2: FCN on FMNIST and MNIST, and S3: ResNet18 on
CelebA, CIFAR-10 and CIFAR-100 for both iid and non-iid data distributions.
We present results of
S1 (on non-iid data) in this section and defer the rest (including S1 on iid data) to Appendix E.
Properties Studied.
We speciﬁcally focus on four properties of LBGM : P1: beneﬁts of gradient recycling by LBGM as a standalone algorithm , P2: conﬁrming the insights from Theorem 1 about effect of δ threshold k on LBGM , P3: practical capabilities of LBGM as a general plug-and-play algorithm that can be stacked on top of other gradient compression techniques to further reduce communication overhead in FL training by exploiting the low-rank property of the gradient subspace, and ﬁnally P4: generalizability of LBGM to other distributed learning frameworks, e.g., multi-processor ML system over multiple GPUs within a single machine, or even multiple machines linked together.
Baselines.
For P1 , we compare standalone LBGM with vanilla FL.
For P3 , we stack LBGM on top of existing gradient compression techniques: top-K and ATOMO (Wang et al., 2018) – among the state-of-the-art techniques in the domains of sparsiﬁcation and low-rank approximation based gradient compression, respectively.
For P4 , we stack LBGM on top of SignSGD (Bernstein et al., 2018), which is one of the state-of-the-art solutions in gradient compression for distributed learning.
Implementation Details.
We consider a FL system consisting of 100 workers.
We consider both the independently and identically distributed (iid) and non-iid data distributions among the workers.
Under the iid setting, each worker has training data with all the labels, while under the non-iid setting the workers have training data only from a subset of all labels (e.g., each worker has training data from only 3 of the 10 classes in MNIST/FMNIST).
The workers train identical networks with a mini-batch size ranging from 128 to 512 based on the choice of dataset.
We implement LBGM with uniform δ threshold k across workers.
We also use error feedback (Karimireddy et al., 2019) as standard only if top-K sparsiﬁcation is used in the training.
The FL system is simulated using PyTorch (Paszke et al., 2019) and PySyft (Ryffel et al., 2018) and trained on a 48GB Tesla-P100 GPU with 128GB RAM.
All of our code, and hyperparameters are available at https://github.com/PairML/LBGM .
Complexity.
Compared to other gradient compression techniques, the processing overhead introduced by LBGM is negligible.
Considering Algorithm 1, the calculation of LBCs/LBP errors involves inner products and division of scalars, while reconstruction of LBG-based gradient approximations at the server is no more expensive than the global aggregation step: since the global aggregation step requires averaging of local model parameters, it can be combined with gradient reconstruction.
This also holds for LBGM as a plug-and-play algorithm, as top-K and ATOMO (Wang et al., 2018) introduce considerable computation overhead.
In particular, LBGM has O ( M ) complexity, where M is the dimension of the NN parameter, which is inexpensive to plug on top of top-K
( O ( M log M ) ),
ATOMO (Wang et al., 2018)
( O ( M 2 ) ), and SignSGD (Bernstein et al., 2018)
( O ( M ) ) methods.
[Figure]
LBGM as a Standalone Algorithm.
We evaluate the effect of gradient recycling by LBGM in FL.
We choose δ threshold k = 0 .
2 , ∀ k , and observe that LBGM is able to save communication overhead in the order of 10 7 ﬂoating point parameters per worker.
Communication beneﬁts are depicted in Fig.
Refer to Appendix E.1 for similar results on other datasets and NN architectures.
Effect of δ threshold k
We analyze the behavior of LBGM under different δ threshold k values.
As pointed out in takeaway 3 in Sec. 3, decreasing the allowable threshold on LBP error, sin 2 ( α ( t ) ,ℓ k ) improves the model performance.
The effect of changing the threshold is controlled by changing δ threshold k values in Algorithm 1.
A drop in model performance can be observed as we increase δ threshold k in Fig.
This performance drop is accompanied by an increase in communication savings as discussed in takeaway 5 in Sec.
3, i.e., while a higher threshold value requires less frequent updates of LBGs, it converges slower or to a suboptimal solution.
Refer to Appendix E.2 for additional results.
LBGM as a Plug-and-Play Algorithm.
Next we show that state-of-the-art results in gradient compression can beneﬁt from using LBGM .
For plug-and-play setup, the LBGM follows the same steps as in Algorithm 1, with the slight modiﬁcation that the output of gradient compression techniques, e.g., top-K, ATOMO, are used in place of accumulated gradient g ( t ) k and LBGs g ℓk , ∀ k .
We consider top-K sparsiﬁcation, and ATOMO approximation as the base gradient compression techniques and plug LBGM to operate on their outputs.
Interestingly, LBGM adds on top of existing communication beneﬁts of both top-K and ATOMO in the order of 10 6 and 10 5 ﬂoating point model parameters per worker, respectively, as shown in Fig.
7: while top-K and ATOMO compress gradients through approximation, they do not alter the underlying low-rank characteristics of the gradient-space.
Our algorithm LBGM exploits this property to obtain substantial communication savings on top of that of these algorithms.
Refer to Appendix E.3 for additional experiments.
[Figure]
Generalizability of LBGM to Distributed
Training.
Although throughout the paper, we focus on FL, LBGM can be applied to more general distributed gradient computation settings, e.g., multi-core, multi-GPU ML systems.
Thus, most of our experiments above can be easily extended to more general distributed settings.
While non-iid distribution is not a consideration in multi-core distributed systems because of possibility of data transfer, there is a research interest to minimize parameter exchange among systems to reduce communication latency.
In the domain of distributed training, SignSGD (Bernstein et al., 2018) is one of the state-of-the-art solution that reduces the communication requirements by several order of magnitude by converting ﬂoating-point parameter communication to binary bit communication.
We apply LBGM as a plug-and-play addition on top of SignSGD and ﬁnd that LBGM can further reduce the overall bits transferred by SignSGD, by an order of 10 7 bits in Fig. 8.
Refer to Appendix E.4 for additional experiments [Figure]
R ELATED W ORK
Eigenvalues of Hessian.
While to the best of our knowledge there are no prior works directly studying the gradient subspace, there exist several works (Sagun et al., 2016; 2017; Gur-Ari et al., 2018) that analyze the eigenvalues of the Hessian to quantify the overparameterization of NNs.
(Sagun et al., 2016; 2017) divide the eigenspace of Hessian into two parts: bulk (with eigenvalues concentrated around zero) and edges (with eigenvalues scattered away from zero), and show that increasing the complexity of network only affects the bulk component.
(Gur-Ari et al., 2018) aims to show that the cardinality of the edge space is equal to the number of classes in the classiﬁcation task.
In parallel, (Ghorbani et al., 2019) studies the Hessian of larger networks and argues that the existence of large isolated eigenvalues in eigenspace of the Hessian is correlated with the slow convergence.
Gradient Compression.
Gradient compression techniques proposed in literature can be broadly categorized into (i) sparsiﬁcation, (ii) quantization, and (iii) low-rank approximation.
Sparsiﬁcation compresses gradients by promoting sparsity in transmitted gradients.
Speciﬁcally, (Wangni et al., 2018) proposes probabilistic sparsiﬁcation by optimizing the trade-off between sparsity and gradient variance.
(Sattler et al., 2019) extends top-K sparsiﬁcation using ternarization for downstream communication.
Quantization methods on the other hand aim to ﬁnd approximate representations of gradients that promote tensor compression.
(Seide et al., 2014; Alistarh et al., 2017) propose approximation using quantization with error feedback to substantially boost communication compression.
These works were originally proposed for inter-GPU communication speedups and not FL.
Finally, low-rank approximation techniques (Wang et al., 2018; Vogels et al., 2019) decompose large gradients matrices as an outer product of smaller matrices to reduce communication cost.
Communication Delay.
Several works propose methods to reduce the gradient communication frequency.
(Hosseinalipour et al., 2020; Lin et al., 2021) use device-to-device local network communication, while SloMo (Wang et al., 2020b) uses momentum to delay global aggregation step.
Model Compression.
Compressing model would naturally reduce communication overhead.
(Li et al., 2020) extends the lottery ticket hypothesis to federated learning to boost communication saving.
Model distillation (Ba & Caruana, 2014; Hinton et al., 2015) trains a smaller student network to mimic a larger teacher network, and model pruning (LeCun et al., 1990; Hinton et al., 2015) systematically removes model parameters since NNs tend to be dramatically overparameterized in practice and neurons can be removed without affecting the accuracy.
Squeezenets (Iandola et al., 2016) and Mobilenets (Howard et al., 2017) are designed with orders of magnitude smaller size than conventional architectures.
(Denil et al., 2013) represents model parameters with low rank matrices.
Unlike prior works that study the low-rank property in the context of gradient compression during a single gradient transfer step (via exploiting low-rank approximation of matrices), our work explores the low rank property by studying the relationship among gradients generated at successive gradient transfer steps during the distributed training process (i.e., low-rank property of subspace spanned by these gradients).
To the best our knowledge, we are the ﬁrst to study this effect of “overparameterization” on optimization of NN via the rank-characteristics of the ﬁrst order optimization (gradient) landscape.
We propose the LBGM algorithm based on the low-rank property of gradient-space, which is itself orthogonal to other related works in the ﬁeld of gradient compression, as none of these techniques exploit the rank-property of the gradient space.
Thus, most of the prior techniques would further beneﬁt by using LBGM algorithm during FL training as we show in our experiments for top-K,
ATOMO, and SignSGD.
Our work gives a novel insight into designing a class of techniques based on “Look-back Gradients” that can be used in general distributed machine learning systems.
D ISCUSSION & C ONCLUSIONS
In this paper, we gave a fresh perspective into the effect “overparamterization” on the optimization of NNs by analyzing the gradient subspace.
We drew insights from our preliminary experiments to support our two hypotheses: “the subspace spanned by gradients generated during SGD is low-rank” and “principal gradients can be approximated using a subset of actual gradients” .
We then proposed
LBGM algorithm that uses the low-rank property of gradient-space to recycle the previously generated gradients at the devices to represent the newly generated gradients.
LBGM reduces communication overhead in FL by several orders of magnitude by replacing the transmission of vectors of million/billion parameters with a single scalar at each device.
We theoretically characterized the convergence behavior of LBGM algorithm and experimentally substantiate our claims.
Furthermore, we showed that our algorithm LBGM can be extended to further reduce latency of communication in large distributed training systems by plugging LBGM on top of other gradient compression techniques.
Appendix Table of Contents
A Proof of Theorem
A.1 Condition on Learning Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Proof of Corollary
C Additional Pseudocodes
D Additional Preliminary Experiments
D.1 PCA Component Progression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Overlap of Actual and Principal Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Similarity among Consecutive Generated Gradients . . . . . . . . . . . . . . . . . . . . . . .
E Additional LBGM Experiments
E.1 LBGM as a Standalone Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Effect of δ threshold k on LBGM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 LBGM as a Plug-and-Play Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.4 Generalizability of LBGM to Distributed Training. . . . . . . . . . . . . . . . . . . . . . . . .
E.5 LBGM under Client Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A P ROOF OF T HEOREM
We start by introducing a lemma, that is used prominently throughout the analysis that follows.
Lemma 1.
For a sequence of vectors { a i } Ni =1 , such that E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Proof.
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Using the law of total expectation, assuming i < j , we get
[Equation]
[Equation]
which completes the proof.
Next, we deﬁne a few auxiliary variables that would be referenced in the proof later: as deﬁned in the main text, g ( t ) k
= P
[Equation]
[-1 b =
-]− [Equation] = [Equation]
g k ( θ ( t
[Equation]
b ) k ) is the accumulated stochastic gradient at worker k , where b ranging from 0 to τ − 1 denotes the rounds of local updates.
Using Assumption ( A2 ), error in stochastic gradient approximation g k ( θ ( t,b ) k ) can be deﬁned as ǫ ( t,b ) k
= g k ( θ ( t,b ) k ) − ∇ F k ( θ ( t,
[Equation]
) k ) . Consequently, we can write the stochastic gradient as g k ( θ ( t,b ) k ) =
∇ F k ( θ ( t,b ) k ) + ǫ ( t,b ) k where ∇ F k ( θ ( t,b ) k ) is the true gradient.
From Assumption ( A2 ) it follows that
[Equation]
We also introduce the normalized stochastic gradient d ( t ) k given by
[Equation]
where we deﬁne
Cumulative Average of the true gradient: h ( t ) k
= 1 τ
[Equation]
and (9)
Cumulative Average of the SGD error: ǫ ( t ) k = 1 τ
[Equation]
Next, we evaluate the ﬁrst and second moment of normalized SGD error, ǫ ( t ) k as follows [Equation] [Equation] h
[Equation]
i
[Equation]
E
[Equation]
(cid:13)
[Equation]
( t
[Equation]
(cid:13)
[Equation]
(cid:13)(cid:13) (cid:13)
[Equation] (cid:21) [Equation] (12) where ( 11) uses Assumption ( A2 ) and (12) uses Lemma 1.
Now we can proceed to the proof of Theorem 1.
We start with the update rule of LBGM , where a round of global update is given by
[Equation]
where e g ( t ) k is the approximate gradient shared by worker k with the server and [Equation] k is given by
[Equation]
14) where d ℓk = g ℓk /τ is the normalized stochastic gradient w.r.t.
the last LBG shared by worker k .
Also, from the trigonometric relationship shown in Fig. 4, we have,
[Equation]
Similar to d ( t ) k from (8), d ℓk can be split into the cumulative average of true look-back gradient h ℓk and the corresponding cumulative average of SGD error ǫ ℓk , given by
[Equation]
From Assumption ( A1 ), the global loss function is β smooth, which implies:
[Equation]
Taking expectation over ǫ ( t,b )
k
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Z
[Equation]
[Equation]
[Equation]
We evaluate Z 1 as follows
[Equation]
where Z 1 , 1 is given by,
[Equation]
K
X k = ω
k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
 −
[Equation]
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) ) −
K X k = ω k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   , (20) where ( i ) follows from (16) and ( ii ) uses 2 h a , b i =
k a k 2 + k b k 2 − k a − b k 2 for any two real vectors a and b .
We next upper bound the term Z 1 , 2 (although Z 1 , 2 has a negative sign in (19), Zalso appears with a negative sign in (18) which allows us to do the upper bound) as follows
[Equation]
(cid:13)(cid:13)(cid:13)   , (21) where ( i ) follows from h a, b i ≤ (1 / 4) k a k 2 + k b k 2 (result of Cauchy-Schwartz and Young’s inequalities).
Substituting (20) and (21) back in (19), we get
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 

[Equation]
2 E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
F (
[Equation]
( t ) )
[Equation]
K
X k
= ω
k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 

[Equation]
K
[Equation]
e
[Equation]
[Equation]
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
 . We next bound the term Z 2 in (18) as follows
[Equation]
  ( i )
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
e
[Equation]
(cid:19)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 
 , (23) where Z 2 , 1 is given by,
[Equation]
=
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]

[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
[Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] 

[Equation]
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   ( ii ) ≤
[Equation]   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] (24) where ( i ) follows from (8), ( ii ) uses Jensen’s inequality: k
[Equation]
P Kk =1 ω k = 1 , and ( iii ) uses (12).
Plugging (24) back into (23), we get
[Equation]
[Equation]
Substituting (22) and (25) back in (18), we get
E h F ( θ ( t +1) )
i −
F ( θ ( t ) ) ≤ − τη(cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) )
(cid:13)(cid:13)(cid:13) − τη 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K X k
= ω k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   + τη 2 E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) ) −
K X k
= ω k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   + τη E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
X k = ω k (cid:18) d ( t )
k − e d ( t ) k
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   + τ 2 η 2 β E   (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)
K X k
= ω k h ( t ) k (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)   + τη 2 βσ 2 + τ 2 η 2 β E   (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)
K X k = ω k (cid:18) d ( t )
k − e d ( t ) k
(cid:19)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) 
 = − τη (cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) )
(cid:13)(cid:13) (cid:13) − τη 2 (1 − 2 τηβ ) E  
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
K X k
= ω k h ( t ) k (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)   + τη 2 E  
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) ∇ F ( θ ( t ) )
−
K X k
= ω k h ( t ) k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   + τη (1 + τηβ ) E  
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)
K X k = ω k (cid:18) d ( t )
k − e d ( t ) k (cid:19) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)   + τη 2 βσ 2 .
Choosing τηβ ≤ 1 / 2 , implies that − τη 2 (1 − 2 τηβ ) ≤ 0 and 1 + τηβ ≤ 3 / 2 < 2 , which results in simpliﬁcation of (26) as: [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
) k − e d ( t ) [Equation] (cid:13)(cid:13)(cid:13)(cid:13) # | {z }
Z [Equation] ηβσ
[Equation]
(27) where ( i ) follows from Jensen’s inequality k [Equation]
and θ ( t ) k
= θ ( t ) , ∀ k due to local synchronization.
Now Z 3 can be bounded as follows:
Z 3 = E "
[Equation]
e [Equation] [Equation] (cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
d ℓk (cid:13)(cid:13)(cid:13)
[Equation] (cid:21) ( ii [Equation] E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d ( t )
[Equation]
D g (
[Equation]
g
[Equation] E [Equation] (cid:13)(cid:13)
g ℓk (cid:13)(cid:13)
[Equation]
d ℓk (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   ( iii ) = E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d [Equation]
D
[Equation]
E (cid:13)(cid:13)(cid:13) d ℓk (cid:13)(cid:13)(cid:13) d ℓk (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
 = E  (cid:13)(cid:13)(cid:13)
d ( t ) k (cid:13)(cid:13)(cid:13) − D d ( t ) k , d ℓk
E (cid:13)(cid:13)(cid:13) d ℓk (cid:13)(cid:13)(cid:13) 
 ( iv ) =
E (cid:20)(cid:13)(cid:13)(cid:13) d ( t ) k (cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13) d ( t ) k (cid:13)(cid:13)(cid:13) 2 cos 2 ( α ( t ) ,ℓ k ) (cid:21) = E (cid:20)(cid:13)(cid:13)(cid:13) d ( t ) k
(cid:13)(cid:13)(cid:13) (cid:16) 1 − cos 2 ( α ( t ) ,ℓ k ) (cid:17)(cid:21) = 2 E (cid:20)(cid:13)(cid:13)(cid:13) d ( t ) k (cid:13)(cid:13)(cid:13) 2 sin 2 ( α ( t ) ,ℓ k ) (cid:21) ( v ) ≤ E (cid:2) ∆(cid:3) =
∆ 2 , (28) where ( i ) uses (14), ( ii ) uses LBGM deﬁnition from ( D1 ), ( iii ) uses the fact that d ( t ) k = g ℓk /τ , ( iv ) uses h a, b i =
k a kk b k cos( α ) , and ( v ) follows from the condition in the theorem.
Substituting (28) back in (27), we get E
[Equation]
(29) where Z 4 is given by,
[Equation]
(cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
) (cid:17)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) 
 ≤ τ τ −
X b =
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ F k (
[Equation]
( t, 0) k )
−∇ F k ( θ ( t,b ) k ) (cid:13) (cid:13)(cid:13) (cid:21) ≤ βτ τ −
X b =
E (cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) (cid:21) . Also, using the local update rule θ ( t,b ) k ← θ ( t, 0) k − η P b − 1 s =
g k ( θ ( t,s ) k ) , where θ ( t,b ) k is the model parameter obtained at the b -th local iteration of the global round t at device k , we get: Z
[Equation]
t
[Equation]
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13) (cid:21) =
[Equation]
  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
X s =
g
[Equation]
[Equation]
( t,s ) k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
 ( i )
[Equation]
η
[Equation]
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) b
[Equation]
X s =0 ∇
F k ( [Equation] [Table] [Equation]
s )
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
 [Equation]
2 η
[Equation]
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
−
[Table]
X
[Table]
= [Table] ǫ ( [Table]
[Table]
k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 
 ( ii )
[Equation]
b b [Equation] [Equation] E (cid:20)(cid:13)(cid:13)(cid:13) ∇ F
[Equation]
[Equation] [Equation] s ) k ) (cid:13)(cid:13)(cid:13) (cid:21) + 2 ηb
[Equation]
where ( i ) uses Cauchy-Schwartz inequality and ( ii ) uses Lemma 1 and Cauchy-Schwartz inequality.
Also note that
[Equation]
[Equation]
Taking the cumulative sum of both hand sides of Z 5 from (31) over all batches, i.e., 1 τ
P τ − 1 b =0 , and using (32), we get:
[Equation]
Furthermore, term Z 6 can be bounded as follows:
[Equation]
(cid:21) ( i ) ≤ 2 E (cid:20)(cid:13)(cid:13)(cid:13) ∇ F k ( θ ( t,b ) k ) − ∇ F k ( θ ( t, 0) k ) (cid:13)(cid:13)(cid:13) (cid:21) + 2 E (cid:20)(cid:13)(cid:13)(cid:13)
∇ F k ( θ ( t, 0) k ) (cid:13)(cid:13)(cid:13) (cid:21) ( ii ) ≤ 2 β 2 E (cid:20)(cid:13)(cid:13)(cid:13) θ ( t,b ) k − θ ( t, 0) k (cid:13)(cid:13)(cid:13) (cid:21) + 2 E (cid:20)(cid:13)(cid:13)(cid:13) ∇ F k ( θ ( t, 0) k ) (cid:13)(cid:13)(cid:13) (cid:21) , (34) where ( i ) uses Cauchy-Schwartz inequality and ( ii ) uses ( A1 ).
Replacing Z 6 in (33) using (34), we get: τ
[Equation]
X
[Equation] = [Equation] E (cid:20)(cid:13)(cid:13)(cid:13) θ ( t, 0) k − θ ( t
[Equation]
) k (cid:13)(cid:13)(cid:13)
[Equation]
(cid:21) | {z }
[Equation]
≤
[Equation]
[Equation]
− 1)
+
[Equation]
[Equation]
− 1)
τ
[Equation]
X b =
E (cid:20)(cid:13)(cid:13)(cid:13) θ ( t,b ) k − θ ( t, 0) k
(cid:13)(cid:13)(cid:13) (cid:21) |
[Equation] Note that Z 5 , which is originally deﬁned in (31), appears both in the left hand side (LHS) and right hand side (RHS) of the above expression.
Rearranging the terms in the above inequality yields:
[Equation]
Deﬁning H = 2 η 2 β 2 τ ( τ − 1) , the above inequality can be re-written to evaluate Z 4 ,
[Equation]
Taking a weighted sum from the both hand sides of the above inequality across all the workers and using ( A3 )
[Equation]
we get:
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(38) where ( i ) follows from ( A3 ) and θ ( t )
k
= θ ( t ) , ∀ k since computation occurs at the instance of global aggregation.
Next, plugging (38) back in (29), we get: E
[Equation]
≤ −  (cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) )
(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
k ) −
[Equation]
(cid:21)
[Equation]
2∆
[Equation]
≤ −  (cid:13)(cid:13)(cid:13) ∇ F ( θ ( t ) )
(cid:13)(cid:13)(cid:13)
[Equation]
[Equation] If H ≤ 1 1+2 α Υ 2 for some constant α > [Equation] then it follows that
[Equation]
Choosing α = 2 we can simplify the above expression as follows:
[Equation]
Rearranging the terms in the above inequality and taking the average across all aggregation rounds from the both hand sides, yields:
[Equation]
where we used the fact that F is bounded below
[Equation]
since F k -s are presumed to be bounded below
[Equation]
and
F ⋆ ≤ F ( θ ) , ∀ θ ∈ R M .
This completes the proof of Theorem
[Equation]
A. C ONDITION ON L EARNING R ATE
From the two conditions on the learning rate used in the analysis above, we have
[Equation]
[Equation]
We can further tighten the second constraint as,
[Equation]
Combining the two we have,
[Equation]
B P ROOF OF C OROLLARY
Using (41) we have:
[Equation]
Next, using the assumption in the corollary statement
[Equation]
we have ∆ 2 ≤ η .
We then can upper bound the RHS of (46) to get:
[Equation]
Choosing η = q 1 τT [Equation] we get,
[Equation]
We can write the above expression as,
T
[Equation]
This completes the proof for Corollary 1.
C A DDITIONAL P SEUDOCODES
In this Appendix, we provide a psuedocode for generating the preliminary experimental results in
Sec. 2.
The actual implementation of the following function calls used in the pseudocode can be found in the listed ﬁles of our code repository: https://github.com/PairML/LBGM :
• get _ num _ PCA _ components : implemented in function estimate _ optimal _ ncomponents , ﬁle: src / common / nb _ utils .
py of the repository.
• get _ PCA _ components : implemented in function pca _ transform , ﬁle: src / common / nb _ utils .
py of the repository.
• cosine _ similarity : implemented using functions sklearn .
preprocessing .
normalize and numpy .
dot , such that cosine _ similarity ( a , b ) = normalize ( a ) .
dot ( normalize ( b )) where a and b are numpy .
array .
• plot _ 1 , plot _ 2 , and plot _ 3 : implemented in ﬁles src / viz / prelim _ 1 .
py , src / viz / prelim _ 2 .
py , and src / viz / prelim _ 3 .
py respectively.
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] t,b ) ; d ) / |B| .
Update parameter: θ ( t,b +1) ← θ ( t,b ) − η · g ( t,b ) .
Accumulate gradient: g ( t ) ← g ( t ) + g ( t,b ) .
13: end for 14:
Set θ ( t +1) ← θ ( t,B ) .
15: actual _ grads .
append ( g ( t ) )
⊲ append accumulated gradient to store 16:
pca95 _ store .
append ( get _ num _ PCA _ components ( actual _ grads , variance = 0 . 95 )) 17: pca99 _ store .
append ( get _ num _ PCA _ components ( actual _ grads , variance = 0 . 99 )) 18: end for 19: plot _
( pca95 _ store , pca99 _ store )
⊲ plot of PCA component progression 20: principal _ grads = get _ PCA _ components ( actual _ grads , variance = 0 . 99 ) 21: heatmap =
zeros ( len ( actual _ grads ) , len ( principal _ grads )) 22: for i = 0 to len ( actual _ grads ) − 1 do 23: for j = 0 to len ( principal _ grads ) − [Table] do 24: heatmap [ i , j ] = cosine _ similarity ( actual _ grads [ i ] , principal [Table]
grads [ j ]) 25: end for 26: end for 27: plot _ 2 ( heatmap )
⊲ overlap gradients 28: heatmap = zeros ( len ( actual [Table] grads ) , len ( actual
[Table] grads )) 29: for i = 0 to len ( actual [Table] grads ) − 1 do 30: for j = i to len (
[Table]
grads ) − 1 do 31: heatmap [ i , j ] = heatmap [ j , i ] = [Table] similarity (
[Table]
grads [ i ] , [Table] [ j ]) 32: end for 33: end for 34: plot _ 3 ( heatmap ) ⊲ plot of similarity among
[Table]
gradients
D A DDITIONAL P RELIMINARY E XPERIMENTS
We next present the additional experiments performed to test hypotheses, ( H1 ) & ( H2 ).
As mentioned in Section 2, we study the rank-characteristics of centralized training using SGD on multiple datasets:
FMNIST, MNIST, CIFAR-10, CelebA, COCO, and PascalVOC, and model architectures: CNN, FCN,
Resnet18, VGG19, and U-Net.
Section D.1 presents experiments complementary to the one presented in Fig.
1, while Section D.2 & D.3 present experiments complementary to those presented in Fig. 2, & 3, respectively.
Please follow the hyperlinks for the ease of navigating through the ﬁgures.
D. PCA C OMPONENT P ROGRESSION
Together with Fig.
1, Figs.
10-13 show that both N99-PCA and N95-PCA are signiﬁcantly lower than that the total number of gradients calculated during model training irrespective of model/dataset/learning task for multiple datasets and models which all agree with ( H1 ).
Speciﬁcally, the principal gradients (i.e., red and blue lines in the top row of the plots) are substantially lower (often as low as 10% of number of epochs, i.e., gradients generated) in these experiments.
Refer below for the details of the ﬁgures.
Fig.
9 repeats the experiment conducted in Fig.
1 on CIFAR-100 using FCN, CNN, Resnet18, & VGG19.
Fig.
10 repeats the experiment conducted in Fig.
1 on MNIST using FCN, CNN, Resnet18, & VGG19.
Fig.
11 repeats the experiment conducted in Fig.
1 on FMNIST using FCN, CNN, Resnet18, & VGG19.
Fig.
12 repeats the experiment conducted in Fig.
1 on CIFAR-10, FMNIST, and MNIST using SVM, suggesting that we can use LBGM for classic classiﬁers that are not necessary neural networks.
Fig.
13 repeats the experiment conducted in Fig.
1 on COCO, and PascalVOC using U-Net.
D. O VERLAP OF A CTUAL AND P RINCIPAL G RADIENT
Next, we perform experiments summarized in Fig.
14-35 to further validate our observation in Fig.
2: (i) cosine similarity of actual gradients with principal gradients varies gradually over time, and (ii) actual gradients have a high cosine similarity with one or more of the principal gradients.
Refer below for the details of the ﬁgures.
Each subplot is marked with #L, the layer number of the neural network, and #elem, the number of elements in each layer.
Fig.
14 repeats the experiment conducted in Fig.
2 on CelebA using VGG19.
Fig.
15 repeats the experiment conducted in Fig.
2 on CelebA using Resnet18.
Fig.
16 repeats the experiment conducted in Fig.
2 on CelebA using FCN.
Fig.
17 repeats the experiment conducted in Fig.
2 on CelebA using CNN.
Fig.
18 repeats the experiment conducted in Fig.
2 on CIFAR-10 using VGG19.
Fig.
19 repeats the experiment conducted in Fig.
2 on CIFAR-10 using Resnet18.
Fig.
20 repeats the experiment conducted in Fig.
2 on CIFAR-10 using FCN.
Fig.
21 repeats the experiment conducted in Fig.
2 on CIFAR-10 using CNN.
Fig.
22 repeats the experiment conducted in Fig.
2 on CIFAR-100 using VGG19.
Fig.
23 repeats the experiment conducted in Fig.
2 on CIFAR-100 using Resnet18.
Fig.
24 repeats the experiment conducted in Fig.
2 on CIFAR-100 using FCN.
Fig.
25 repeats the experiment conducted in Fig.
2 on CIFAR-100 using CNN.
Fig.
26 repeats the experiment conducted in Fig.
2 on FMNIST using VGG19.
Fig.
27 repeats the experiment conducted in Fig.
2 on FMNIST using Resnet18.
Fig.
28 repeats the experiment conducted in Fig.
2 on FMNIST using FCN.
Fig.
29 repeats the experiment conducted in Fig.
2 on FMNIST using CNN.
Fig.
30 repeats the experiment conducted in Fig.
2 on MNIST using VGG19.
Fig.
31 repeats the experiment conducted in Fig.
2 on MNIST using Resnet18.
Fig.
32 repeats the experiment conducted in Fig.
2 on MNIST using FCN.
Fig.
33 repeats the experiment conducted in Fig.
2 on MNIST using CNN.
Fig.
34 repeats the experiment conducted in Fig.
2 on PascalVOC using U-Net. 22.
Fig.
35 repeats the experiment conducted in Fig.
2 on COCO using U-Net.
D. S IMILARITY AMONG C ONSECUTIVE G ENERATED G RADIENTS
Furthermore, we perform experiments summarized in Fig.
Together with Fig.
3, these experiments show that there is a signiﬁcant overlap of consecutive gradients generated during SGD iterations, which further substantiates ( H2 ) and bolsters our main idea
that “gradients transmitted in
FL can be recycled/reused to represent the gradients generated in the subsequent iterations” .
Refer below for the details of the ﬁgures.
Fig.
36 repeats the experiment conducted in Fig.
3 on CelebA using VGG19.
Fig.
37 repeats the experiment conducted in Fig.
3 on CelebA using Resnet18.
Fig.
38 repeats the experiment conducted in Fig.
3 on CelebA using FCN.
Fig.
39 repeats the experiment conducted in Fig.
3 on CelebA using CNN.
Fig.
40 repeats the experiment conducted in Fig.
3 on CIFAR-10 using VGG19.
Fig.
41 repeats the experiment conducted in Fig.
3 on CIFAR-10 using Resnet18.
Fig.
42 repeats the experiment conducted in Fig.
3 on CIFAR-10 using FCN.
Fig.
43 repeats the experiment conducted in Fig.
3 on CIFAR-10 using CNN.
Fig.
44 repeats the experiment conducted in Fig.
3 on CIFAR-100 using VGG19.
Fig.
45 repeats the experiment conducted in Fig.
3 on CIFAR-100 using Resnet18.
Fig.
46 repeats the experiment conducted in Fig.
3 on CIFAR-100 using FCN.
Fig.
47 repeats the experiment conducted in Fig.
3 on CIFAR-100 using CNN.
Fig.
48 repeats the experiment conducted in Fig.
3 on FMNIST using VGG19.
Fig.
49 repeats the experiment conducted in Fig.
3 on FMNIST using Resnet18.
Fig.
50 repeats the experiment conducted in Fig.
3 on FMNIST using FCN.
Fig.
51 repeats the experiment conducted in Fig.
3 on FMNIST using CNN.
Fig.
52 repeats the experiment conducted in Fig.
3 on MNIST using VGG19.
Fig.
53 repeats the experiment conducted in Fig.
3 on MNIST using Resnet18.
Fig.
54 repeats the experiment conducted in Fig.
3 on MNIST using FCN.
Fig.
55 repeats the experiment conducted in Fig.
3 on MNIST using CNN.
Fig.
56 repeats the experiment conducted in Fig.
3 on PascalVOC using U-Net. 22.
Fig.
57 repeats the experiment conducted in Fig.
3 on COCO using U-Net.
E A DDITIONAL LBGM E XPERIMENTS
In this section, we present complimentary experiments to the properties studies in Section 4 of the main text.
In particular, we show that our observations hold for datasets: CIFAR-10, CIFAR-100,
CelebA, FMNIST, and MNIST.
We also present results when using shallower models FCN or deeper models Resnet18 different from CNNs.
Section E.1 gives further evidence of utility of LBGM as a standalone solution.
Section E.2 lists ﬁgures that summarize the effect of changing δ threshold k on model performance for mentioned datasets and models.
In Section E.3 we list ﬁgures that summarize additional experiments to support the observations made in Fig.
7 and Section E.4 lists ﬁgures for additional experiments corresponding to the observations made in Fig.
Finally, Section E.5 presents results on LBGM algorithm corresponding to the case wherein 50% of clients are randomly sampled during global aggregation.
Since the two layer FCN considered is a simple classiﬁer, it does not perform well on complex datasets such as CIFAR-10, CIFAR-100, and CelebA.
Thus, the respective results are omitted for FCN on these complicated datasets and the performance of FCN is only studied for MNIST and FMNIST datasets.
Similarly, the 4-layer CNN architecture does not perform well on CIFAR-100 dataset and hence the corresponding results are ommited.
We also present results using U-Net architecture for semantic segmentation on PascalVOC dataset.
E. LBGM AS A S TANDALONE A LGORITHM .
Fig.
58 shows the result of repeating the experiment conducted in Fig.
5 (CNNs on non-iid data distribution) on CNNs for iid data distribution for datasets CIFAR-10, FMNIST and MNIST, and U-Net for segmentation for dataset PascalVOC. 2.
Fig.
59 shows the result of repeating the experiment conducted in Fig.
(CNNs on non-iid data distribution) on FCNs for both non-iid and iid data distributions on datasets FMNIST and MNIST.
Fig.
shows the result of repeating the experiment conducted in Fig.
(CNNs on non-iid data distribution) on Resnet18s for non-iid data distribution on datasets CelebA, CIFAR-10, CIFAR-100, FMNIST
and MNIST using a setup similar to that of Wang et al
E. E FFECT OF δ threshold k ON LBGM .
Fig.
61 shows the result of repeating the experiment conducted in Fig.
6 (CNNs on non-iid data distribution) on CNNs for iid data distribution for datasets CIFAR-10, FMNIST, and MNIST, and U-Net for segmentation for dataset PascalVOC. 2.
Fig.
62 shows the result of repeating the experiment conducted in Fig.
6 (CNNs on non-iid data distribution) on FCNs for both non-iid and iid data distributions on datasets FMNIST and MNIST.
Fig.
63 shows the result of repeating the experiment conducted in Fig.
6 (CNNs on non-iid data distribution) on Resnet18s for non-iid data distribution on datasets CelebA, CIFAR-10 and
CIFAR-100 using a setup similar to that of Wang et al.
E. LBGM AS A P LUG - AND -P LAY A LGORITHM .
Fig.
64 shows the result of repeating the experiment conducted in Fig.
7 (CNNs on non-iid data distribution) on CNNs for iid data distribution for datasets CIFAR-10, FMNIST, and MNIST.
Fig.
65 shows the result of repeating the experiment conducted in Fig.
7 (CNNs on non-iid data distribution) on FCNs for both non-iid and iid data distributions on datasets FMNIST and MNIST.
Fig.
66 shows the result of repeating the experiment conducted in Fig.
7 (CNNs on non-iid data distribution) on Resnet18s for non-iid data distribution on datasets CelebA, CIFAR-10 and
CIFAR-100 using a setup similar to that of Wang et al.
E. G ENERALIZABILITY OF LBGM TO D ISTRIBUTED T RAINING .
Fig.
67 shows the result of repeating the experiment conducted in Fig.
8 (CNNs on non-iid data distribution) on CNNs for iid data distribution for datasets CIFAR-10, FMNIST, and MNIST.
Fig.
68 shows the result of repeating the experiment conducted in Fig.
8 (CNNs on non-iid data distribution) on FCNs for both non-iid and iid data distributions on datasets FMNIST and MNIST.
Fig.
69 shows the result of repeating the experiment conducted in Fig.
8 (CNNs on non-iid data distribution) on Resnet18s for non-iid data distribution on datasets CelebA, CIFAR-10 and CIFAR-100 using a setup similar to that of Wang et al. E. LBGM UNDER C LIENT S AMPLING .
Fig Fig.
71 shows the result of repeating the experiment conducted in Fig. 5 (CNNs on non-iid data distribution) under 50% client sampling using CNNs for iid data distribution for datasets CIFAR-10, FMNIST, and MNIST.
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]