F ED P ARA : L OW - RANK H ADAMARD P RODUCT FOR C OMMUNICATION -E FFICIENT F EDERATED L EARNING
A BSTRACT
In this work, we propose a communication-efﬁcient parameterization, FedPara , for federated learning (FL) to overcome the burdens on frequent model uploads and downloads.
Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product.
Compared to the conventional low-rank parameterization, our FedPara method is not restricted to lowrank constraints, and thereby it has a far larger capacity.
This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods.
The efﬁciency of our method can be further improved by combining with other efﬁcient FL optimizers.
In addition, we extend our method to a personalized FL application, pFedPara , which separates parameters into global and local ones.
We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.
I NTRODUCTION
Federated learning (FL; McMahan et al., 2017) has been proposed as an efﬁcient collaborative learning strategy along with the advance and spread of mobile and IoT devices.
FL allows leveraging local computing resources of edge devices and locally stored private data without data sharing for privacy.
FL typically consists of the following steps: (1) clients download a globally shared model from a central server, (2) the clients locally update each model using their own private data without accessing the others’ data, (3) the clients upload their local models back to the server and (4) the server consolidates the updated models and repeats these steps until the global model converges.
FL has the key properties (McMahan et al., 2017) that differentiate it from distributed learning:
• Heterogeneous data.
Data is decentralized and non-IID as well as unbalanced in its amount due to different characteristics of clients; thus, local data does not represent the population distribution.
• Heterogeneous systems.
Clients consist of heterogeneous setups of hardware and infrastructure; hence
those connections
are not guaranteed to be online, fast , or cheap
Besides, massive client participation is expected through different communication paths, causing communication burdens
These FL properties introduce challenges in the convergence stability with heterogeneous data and communication overheads.
To improve the stability and reduce the communication rounds, the prior works in FL have proposed modiﬁed loss functions or model aggregation methods (Li et al., 2020;
Karimireddy et al., 2020; Acar et al., 2021; Yu et al., 2020; Reddi et al., 2021).
However, the transferred data is still a lot for the edge devices with bandwidth constraints or countries having low-quality communication infrastructure.
1 A large amount of transferred data introduces an energy consumption issue on edge devices because wireless communication is signiﬁcantly more powerintensive than computation (Yadav & Yadav, 2016; Yan et al., 2019).
In this work, we propose a communication-efﬁcient re-parameterization for FL, FedPara , which reduces the number of bits transferred per round.
FedPara directly re-parameterizes each fully- connected (FC) and convolutional layers of the model to have a small and factorized form while preserving the model’s capacity.
Our key idea is to combine the Hadamard product with low-rank parameterization as W =
( X 1 Y (cid:62) 1 ) (cid:12) ( X 2 Y (cid:62) 2 )
∈ R m × n ,
called low-rank Hadamard product .
When rank( X 1 Y (cid:62) 1 ) = rank( X 2 Y (cid:62) 2 ) =
r , then rank( W ) ≤ r 2 .
This outstanding property facilitates spanning a full-rank matrix with much fewer parameters than the typical m × n matrix.
It signiﬁcantly reduces the communication burdens during training.
At the inference phase, we pre-compose and maintain W that boils down to its original structure; thus, FedPara does not alter computational complexity at inference time.
Compared to the aforementioned prior works that tackle reducing the required communication rounds for convergence, our FedPara is an orthogonal approach in that FedPara does not change the optimization part but re-deﬁnes each layer’s internal structure.
We demonstrate the effectiveness of FedPara with various network architectures, including VGG,
ResNet, and LSTM, on standard classiﬁcation benchmark datasets for both IID and non-IID settings.
The accuracy of our parameterization outperforms that of the traditional low-rank parameterization baseline given the same number of parameters.
Besides, FedPara has comparable accuracy to original counterpart models and even outperforms as the number of parameters increases at times.
We also combine FedPara with other FL algorithms to improve communication efﬁciency further.
We extend FedPara to the personalized FL application, named pFedPara , which separates the roles of each sub-matrix into global and local inner matrices.
The global and local inner matrices learn the globally shared common knowledge and client-speciﬁc knowledge, respectively.
We devise three scenarios according to the amount and heterogeneity of local data using the subset of
FEMNIST and MNIST.
We demonstrate performance improvement and robustness of pFedPara against competing algorithms.
We summarize our main contributions as follows:
• We propose FedPara , a low-rank Hadamard product parameterization for communicationefﬁcient FL.
Unlike traditional low-rank parameterization, we show that FedPara can span a full-rank matrix and tensor with reduced parameters.
We also show that FedPara requires up to ten times fewer total communication costs than the original model to achieve target accuracy.
FedPara even outperforms the original model by adjusting ranks at times.
• Our FedPara takes a novel approach; thereby, our FedPara can be combined with other FL methods to get mutual beneﬁts, which further increase accuracy and communication efﬁciency.
• We propose pFedPara , a personalization application of FedPara , which splits the layer weights into global and local parameters.
pFedPara shows more robust results in challenging regimes than competing methods.
M ETHOD
In this section, we ﬁrst provide the overview of three popular low-rank parameterizations in Section 2.1 and present our parameterization, FedPara , with its algorithmic properties in Section 2.2.
Then, we extend FedPara to the personalized FL application, pFedPara , in Section 2.3.
Notations.
We denote the Hadamard product as (cid:12) , the Kronecker product as ⊗ , n -mode tensor product as × n , and the i -th unfolding of the tensor T ( i
) ∈
R k i × (cid:81)
j (cid:54) =
i k
j given a tensor T ∈ R k 1 ×···× k n .
2. O VERVIEW OF L OW - RANK P ARAMETERIZATION
The low-rank decomposition in neural networks has been typically applied to pre-trained models for compression (Phan et al., 2020), whereby the number of parameters is reduced while minimizing the loss of encoded information.
Given a learned parameter matrix W ∈ R m × n , it is formulated as ﬁnding the best rank- r approximation, as arg min (cid:102) W || W − (cid:102) W
|| F such that (cid:102) W = XY (cid:62) ,
where
X ∈ R m × r , Y ∈ R n × r , and r (cid:28) min ( m, n ) .
It reduces the number of parameters from O ( mn ) to
O (( m + n ) r ) , and its closed-form optimal solution can be found by SVD.
This matrix decomposition is applicable to the FC layers and the reshaped kernels of the convolution layers.
However, the natural shape of a convolution kernel is a fourth-order tensor; thus, the lowrank tensor decomposition, such as Tucker and CP decomposition (Lebedev et al., 2015; Phan et al., 2020), can be a more suitable approach.
Given a learned high-order tensor T ∈ R k 1 ×···× k n , Tucker decomposition multiplies a kernel tensor K ∈ R r 1 ×···× r n with matrices X i ∈ R r i × n 1 , where r i = [Figure]
[Figure] set m = n = O = I = 256 , K 1 = K 2 = 3 , and R = 16 . rank( (cid:101) T ( i ) ) as (cid:101) T = K × 1 X 1 × 2 · · · × n X n , and CP decomposition is the summation of ranktensors as (cid:101) T =
(cid:80) i =
r i =1 x 1 i × x 2 i × · · · × x ni , where x ji ∈ R k j .
Likewise, it also reduces the number of model parameters.
We refer to these rank-constrained structure methods simply as conventional low-rank constraints or low-rank parameterization methods.
In the FL context, where the parameters are frequently transferred between clients and the server during training, the reduced parameters lead to communication cost reduction, which is the main focus of this work.
The post-decomposition approaches (Lebedev et al., 2015; Phan et al., 2020) using SVD, Tucker, and CP decompositions do not reduce the communication costs because those are applied to the original parameterization after ﬁnishing training.
That is, the original large-size parameters are transferred during training in FL, and the number of parameters is reduced after ﬁnishing training.
We take a different notion from the low-rank parameterizations.
In the FL scenario, we train a model from scratch with low-rank constraints, but speciﬁcally with low-rank Hadamard product reparameterization .
We re-parameterize each learnable layer, including FC and convolutional layers, and train the surrogate model by FL.
Different from the existing low-rank method in FL (Koneˇcn`y et al., 2016), our parameterization can achieve comparable accuracy to the original counterpart.
2. F ED P ARA : A C OMMUNICATION -E FFICIENT P ARAMETERIZATION
As mentioned, the conventional low-rank parameterization has limited expressiveness due to its lowrank constraint.
To overcome this while maintaining fewer parameters, we present our new low-rank
Hadamard product parameterization, called FedPara , which has the favorable property as follows:
Proposition 1 Let X 1 ∈ R m × r 1 , X 2 ∈ R m × r 2 , Y 1 ∈ R n × r 1 , Y 2 ∈ R n × r 2 , r 1 , r 2 ≤ min( m, n ) and the constructed matrix be W :=
( X 1 Y (cid:62) 1 ) (cid:12) ( X 2 Y (cid:62) 2 ) .
Then, rank( W ) ≤ r 1 r 2 .
All proofs can be found in the supplementary material including Proposition 1.
Proposition 1 implies that, unlike the low-rank parameterization, a higher-rank matrix can be constructed using the
Hadamard product of two inner low-rank matrices, W 1 and W 2 (Refer to Figure 1).
If we choose the inner ranks r 1 and r 2 such that r 1 r 2 ≥ min( m, n ) , the constructed matrix does not have a low-rank restriction and is able to span a full-rank matrix with a high chance (See Figure 6); i.e ., FedPara has the minimal parameter property achievable to full-rank.
In addition, we can control the number of parameters by changing the inner ranks r 1 and r 2 , respectively, but we have the following useful property to set the hyper-parameters to be a minimal number of parameters with a maximal rank.
Proposition 2 Given R ∈ N , r 1 = r 2 =
R is the unique optimal choice of the following criteria,
[Equation]
and its optimal value is 2 R ( m + n ) .
Equation 3 implies the criteria that minimize the number of weight parameters used in our parameterization with the target rank constraint of the constructed matrix as R 2 .
Proposition 2 provides an efﬁcient way to set the hyper-parameters.
It implies that, if we set r 1 = r 2 =
R and R 2 ≥ min( m, n ) ,
FedPara is highly likely to have no low-rank restriction 2 even with much fewer parameters than that of a na¨ıve weight, i.e ., 2 R ( m + n ) (cid:28) mn .
Moreover, given the same number of parameters, rank( W ) of FedPara is higher than that of the na¨ıve low-rank parameterization by a square factor, as shown in Figure 1 and Table 1.
To extend Proposition 1 to the convolutional layers, we can simply reshape the fourth-order tensor kernel to the matrix as R O ×
I × K 1 × K 2 → R O × ( IK 1 K 2 ) as a na ¨ ıve way, where O, I, K 1 , and K 2 are the output channels, the input channels, and the kernel sizes, respectively.
That is, our parameterization spans convolution ﬁlters with a few basis ﬁlters of size I × K
× K 2 .
However, we can derive more efﬁcient parameterization of convolutional layers without reshaping as follows:
Proposition
[Equation]
, Y 1 , Y 2 ∈ R k 2 × R , R ≤ min( k 1 , k 2 ) and the convolution kernel be W :=
( T 1 × 1 X 1 × 2 Y 1 ) (cid:12) ( T 2 × 1 X 2 × 2 Y 2 ) .
Then, the rank of the kernel satisﬁes rank( W (1) ) =
rank( W (2) ) ≤ R 2 .
Proposition 3 is the extension of Proposition 1 but can be applied to the convolutional layer without reshaping.
In the convolutional layer case of Table 1, given the speciﬁc tensor size, Propositionrequires 3 .
8 times fewer parameters than Proposition 1.
Hence, we use Proposition 3 for the convolutional layer since the tensor method is more effective for common convolutional models.
Optionally, we employ non-linearity and the Jacobian correction regularization, of which details can be found in the supplementary material.
These techniques improve the accuracy and convergence stability but not essential.
Depending on the resources of devices, these techniques can be omitted.
2. P F ED P ARA : P ERSONALIZED FL A PPLICATION
[Figure]
In practice, data are heterogeneous and personal due to different characteristics of each client, such as usage times and habits.
FedPer (Arivazhagan et al., 2019) has been proposed to tackle this scenario by distinguishing global and local layers in the model.
Clients only transfer global layers (the top layer) and keep local ones (the bottom layers) on each device.
The global layers learn jointly to extract general features, while the local layers are biased towards each user.
With our FedPara , we propose a personalization application, pFedPara , in which the Hadamard product is used as a bridge between the global inner weight W 1 and the local inner weight W 2 .
Each layer of the personalized model is constructed by
W = W 1 (cid:12) ( W 2 + 1 ) , where W 1 is transferred to the server while W 2 is kept in a local device during training.
This induces W 1 to learn globally shared knowledge implicitly and acts as a switch of the term ( W 2 + 1 ) .
Conceptually, we can interpret by rewriting W = W 1 (cid:12) W 2 + W 1 =
W per.
+ W glo.
, where W per.
=
W 1 (cid:12) Wand W glo.
= W 1 .
The construction of the ﬁnal personalized parameter W in pFedPara can be viewed as an additive model of the global weight W glo.
and the personalizing residue W per. .
pFedPara transfers only a half of the parameters compared to FedPara under the same rank condition; hence, the communication efﬁciency is increased further.
Intuitively, FedPer and pFedPara are distinguished by their respective split directions, as illustrated in Figure 2.
We summarize our algorithms in the supplementary material.
Although we only illustrate feed-forward network cases for convenience, it can be extended to general cases.
E XPERIMENTS
We evaluate our FedPara in terms of communication costs, the number of parameters, and compatibility with other FL methods.
We also evaluate pFedPara in three different non-IID scenarios.
We use the standard FL algorithm, FedAvg , as a backbone optimizer in all experiments except for the compatibility experiments.
More details and additional experiments can be found in the supplementary material.
3. S ETUP
Datasets.
In FedPara experiments, we use four popular FL datasets: CIFAR-10, CIFAR-
(Krizhevsky et al., 2009), CINIC-10 (Darlow et al., 2018), and the subset of Shakespeare (Shakespeare, 1994).
We split the datasets randomly into 100 partitions for the CIFAR-10 and CINICIID settings and 50 partitions for the CIFAR-100 IID setting.
For the non-IID settings, we use the
Dirichlet distribution for random partitioning and set the Dirichlet parameter α as 0 .
5 as suggested by He et al.
(2020b).
We assign one partition to each client and sample 16% of clients at each round during FL.
In pFedPara experiments, we use the subset of handwritten datasets: MNIST (LeCun et al., 1998) and FEMNIST (Caldas et al., 2018).
For the non-IID setting with MNIST, we follow
McMahan et al.
(2017), where each of 100 clients has at most two classes.
Models.
We experiment VGG and ResNet for the CNN architectures and LSTM for the RNN architecture as well as two FC layers for the multilayer perceptron.
When using VGG, we use the VGG16 architecture (Simonyan & Zisserman, 2015) and replace the batch normalization with the group normalization as suggested by Hsieh et al.
VGG16 ori .
stands for the original
VGG16 , VGG16 low the one with the low-rank tensor parameterization in a Tucker form by following
TKD (Phan et al., 2020), and VGG16 FedPara the one with our FedPara .
In the pFedPara tests, we use two FC layers as suggested by McMahan et al.
Rank Hyper-parameter.
We adjust the inner rank of W 1 and W 2 as r = (1 − γ ) r min + γr max , where r min is the minimum rank allowing FedPara to achieve a full-rank by Proposition 2, r max is the maximum rank such that the number of FedPara ’s parameters do not exceed the number of original parameters, and γ ∈ [0 , 1] .
We ﬁx the same γ for all layers for simplicity.
3 Note that γ determines the number of parameters.
3. Q UANTITATIVE R ESULTS
Capacity.
In Table 2, we validate the propositions stating that our FedPara achieves a higher rank than the low-rank parameterization given the same number of parameters.
We train VGG16 low and
VGG16 FedPara for the same target rounds T , and use 10 .
25% and 10 .
15% of the VGG16 ori .
parameters, respectively, to be comparable.
As shown in Table 2a, VGG16 FedPara surpasses VGG16 low on all the IID and non-IID settings benchmarks with noticeable margins.
The same tendency is observed in the recurrent neural networks as shown in Table 2b.
We train LSTM on the subset of
Shakespeare.
Table 2b shows that LSTM FedPara has higher accuracy than LSTM low , where the number of parameters is set to 19% and 16% of LSTM , respectively.
This experiment evidences that
our
FedPara has a better model expressiveness and accuracy than the low-rank parameterization.
[Table] [Table] [Figure] un i c a t i o n C o
[Figure]
[Figure]
E n e r g
[Figure]
C
[Figure]
Communication Cost.
We compare VGG16 FedPara and VGG16 ori .
in terms of accuracy and communication costs.
FL evaluation typically measures the required rounds to achieve the target accuracy as communication costs, but we instead assess total transferred bit sizes, 2 × (#participants) × (model size) ×
(#rounds) , which considers up-/down-link and is a more practical communication cost metric.
Depending on the difﬁculty of the datasets, we set the model size of VGG16 FedPara as 10 . 1% , 29 .
4% , and 21 .
8% of VGG16 ori .
for CIFAR-10, CIFAR-100, and
CINIC-10, respectively.
[Figure]
FedAvg
FedProx
SCAFFOLD
FedDyn
FedAdam
Accuracy ( T = 200 ) 82.
[Table]
In Figures 3a-3f, VGG16 FedPara has comparable accuracy but requires much lower communication costs than VGG16 ori . .
Figure 3g shows communication costs and energy consumption required for model training to achieve the target accuracy; we compute the energy consumption by the energy model of user-to-data center topology (Yan et al., 2019).
VGG16 FedPara needs 2.8 to 10.
times fewer communication costs and energy consumption than VGG16 ori .
to achieve the same target accuracy.
Because of these properties, FedPara is suitable for edge devices suffering from communication and battery consumption constraints.
Model Parameter Ratio.
We analyze how the number of parameters controlled by the rank ratio γ affects the accuracy of FedPara .
As revealed in Figure 4, VGG16 FedPara ’s accuracy mostly increases as the number of parameters increases.
VGG16 FedPara can achieve even higher accuracy than VGG16 ori . .
It is consistent with the reports from the prior works (Luo et al., 2017;
Kim et al., 2019) on model compression, where reduced parameters often lead to accuracy improvement, i.e ., regularization effects.
Compatibility.
We integrate the FedPara -based model with other FL optimizers to show that our FedPara is compatible with them.
We measure the accuracy during the target rounds and the required rounds to achieve the target accuracy.
Table 3 shows that VGG16 FedPara combined with the current state-of-the-art method, FedDyn , is the best among other combinations.
Thereby, we can further save the communication costs by combining FedPara with other efﬁcient FL approaches.
Personalization.
We evaluate pFedPara , assuming no sub-sampling of clients for an update.
We train two FC layers on the FEMNIST or MNIST datasets using four algorithms, FedPAQ , FedAvg ,
FedPer , and pFedPara , with ten clients.
FedPAQ denotes the local models trained only using their own local data; FedAvg the global model trained by the FedAvg optimization; and FedPer and pFedPara the personalized models of which the ﬁrst layer ( FedPer ) and half of the inner
[Figure]
matrices ( pFedPara ) are trained by sharing with the server, respectively, while the other parts are locally updated.
We validate these algorithms on three scenarios in Figure 5.
In Scenario 1 (Figure 5a), the FedPAQ accuracy is higher than those of FedAvg and FedPer because each client has sufﬁcient data.
Nevertheless, pFedPara surpasses the other methods.
In
Scenario 2 (Figure 5b), the FedAvg accuracy is higher than that of FedPAQ because local data are too scarce to train the local models.
The FedPer accuracy is also lower than that of FedAvg because the last layer of FedPer does not exploit other clients’ data; thus, FedPer is susceptible to a lack of local data.
The higher performance of pFedPara shows that our pFedPara can take advantage of the wealth of distributed data in the personalized setup.
In Scenario 3 (Figure 5c), the
FedAvg accuracy is much lower than the other methods due to the highly-skewed data distribution.
In most scenarios, pFedPara performs better or favorably against the others.
It validates that pFedPara can train the personalized models collaboratively and robustly.
Additionally, both pFedPara and FedPer save the communication costs because they partially transfer parameters; pFedPara transfers 3 .
4 times fewer parameters, whereas FedPer transfers 1 .
07 times fewer than the original model in each round.
The reduction of FedPer is negligible because it is designed to transfer all the layers except the last one.
Contrarily, the reduction of pFedPara is three times larger than FedPer because all the layers of pFedPara are factorized by the Hadamard product during training, and
only a half of each layer’s parameters are transmitted.
Thus, pFedPara is far more suitable in terms of both personalization performance and communication efﬁciency in FL.
Additional Experiments.
We conducted additional experiments of wall clock time simulation and other architectures including ResNet , LSTM , and Pufferfish (Wang et al., 2021), but the results can be found in the supplementary material due to the page limit.
R ELATED W ORK
Federated Learning.
The most popular and de-facto algorithm, FedAvg (McMahan et al., 2017), reduces communication costs by updating the global model using a simple model averaging once a large number of local SGD iterations per round.
Variants of FedAvg (Li et al., 2020; Yu et al., 2020; Diao et al., 2021) have been proposed to reduce the communication cost, to overcome data heterogeneity, or to increase the convergence stability.
Advanced FL optimizers (Karimireddy et al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) enhance the convergence behavior and improve communication efﬁciency by reducing the number of necessary rounds until convergence.
Federated quantization methods (Reisizadeh et al., 2020; Haddadpour et al., 2021) combine the quantization algorithms with FedAvg and reduce only upload costs to preserve the model accuracy.
Our FedPara is a drop-in replacement for layer parameterization, which means it is an orthogonal and compatible approach to the aforementioned methods; thus, our method can be integrated with other FL optimizers and model quantization.
Distributed Learning.
In large-scale distributed learning of data centers, communication might be a bottleneck.
Gradient compression approaches, including quantization (Alistarh et al., 2017; Bernstein et al., 2018; Wen et al., 2017; Reisizadeh et al., 2020; Haddadpour et al., 2021), sparsiﬁcation (Alistarh et al., 2018; Lin et al., 2018), low-rank decomposition (Vogels et al., 2019; Wang et al., 2021), and adaptive compression (Agarwal et al., 2021) have been developed to handle communication trafﬁc.
These methods do not deal with FL properties such as data distribution and partial participants per round (Kairouz et al., 2019).
Therefore, the extension of the distributed methods to
FL is non-trivial, especially for optimization-based approaches.
Low-rank Constraints.
As described in section 2.1, low-rank decomposition methods (Lebedev et al., 2015;
Tai et al., 2016; Phan et al., 2020) are inappropriate for FL due to additional steps; the post-decomposition after training and ﬁne-tuning.
In FL, Koneˇcn`y et al.
(2016) and Qiao et al.
(2021) have proposed low-rank approaches.
Koneˇcn`y et al.
(2016) train the model from scratch with low-rank constraints, but the accuracy is degraded when they set a high compression rate.
To avoid such degradation, FedDLR (Qiao et al., 2021) uses an ad hoc adaptive rank selection and shows the improved performance.
However, once deployed, those models are inherently restricted by limited low-ranks.
In particular, FedDLR requires matrix decomposition in every up/down transfer.
In contrast, we show that FedPara has no such low-rank constraints in theory.
Empirically, the models re-parameterized by our method show comparable accuracy to the original counterparts when trained from scratch.
D ISCUSSION AND C ONCLUSION
To overcome the communication bottleneck in FL, we propose a new parameterization method,
FedPara , and its personalized version, pFedPara .
We demonstrate that both FedPara and pFedPara can signiﬁcantly reduce communication overheads with minimal performance degradation or better performance over the original counterpart at times.
Even using a strong low-rank constraint, FedPara has no low-rank limitation and can achieve a full-rank matrix and tensor by virtue of our proposed low-rank Hadamard product parameterization.
These favorable properties enable communication-efﬁcient FL, of which regimes have not been achieved by the previous low-rank parameterization and other FL approaches.
We conclude our work with the following discussions.
Discussions.
FedPara conducts multiplications many times during training, including the
Hadamard product, to construct the weights of the layers.
These multiplications may potentially be more susceptible to gradient exploding, vanishing, dead neurons, or numerical instability than the low-rank parameterization with an arbitrary initialization.
In our experiments, we have not observed such issues when using He initialization (He et al., 2015) yet.
Investigating initializations appropriate for our model might improve potential instability in our method.
Also, we have discussed the expressiveness of each layers in neural networks in the view of rank.
As another view of layer characteristics, statistical analysis of weights and activation also offers ways to initialize weights (
He et al., 2015; Sitzmann et al., 2020) and understanding of neural networks (De & Smith, 2020), which is barely explored in this work.
It would be a promising future direction to analyze statistical properties of composited weights from our parameterization and activations and may pave the way for FedPara -speciﬁc initialization or optimization.
Through the extensive experiments, we show the superior performance improvement obtained by our method, and it appears to be with no extra cost.
However, the actual payoff exists in the additional computational cost when re-composing the original structure of W from our parameterization during training; thus, our method is slower than the original parameterization and low-rank approaches.
However, the computation time is not a dominant factor in practical FL scenarios as shown in Table 7, but rather the communication cost takes a majority of the total training time.
It is evidenced by the fact that FedPara offers better Pareto efﬁciency than all compared methods because our method has higher accuracy than low-rank approaches and much less training time than the original one.
In contrast, the computation time might be non-negligible compared to the communication time in distributed learning regimes.
While our method can be applied to distributed learning, the beneﬁts of our method may be diminished there.
Improving both computation and communication efﬁciency of FedPara in large-scale distributed learning requires further research.
It would be a promising future direction.
E THICS S TATEMENT
We describe the ethical aspect in various ﬁelds, such as privacy, security, infrastructure level gap, and energy consumption.
Privacy and Security.
Although FL is privacy-preserving distributed learning, personal information may be leaked due to the adversary who hijacks the model intentionally during FL.
Like other
FLs, this risk is also shared with FedPara due to communication.
Without care, the private data may be revealed by the membership inference or reconstruction attacks (Rigaki & Garcia, 2020).
The local parameters in our pFedPara could be used as a private key to acquire the complete personal model, which would reduce the chance for the full model to be hijacked.
It would be interesting to investigate whether our pFedPara guarantees privacy preserving or the way to improve robustness against those risks.
Infrastructure Level Gap.
Another concern introduced by FL is limited-service access to the people living in countries having inferior communication infrastructure, which may raise human rights concerns including discrimination, excluding, etc .
It is due to a larger bandwidth requirement of FL to transmit larger models.
Our work may broaden the countries capable of FL by reducing required bandwidths, whereby it may contribute to addressing the technology gap between regions.
Energy Consumption.
The communication efﬁciency of our FedPara directly leads to noticeable energy-saving effects in the FL scenario.
It can contribute to reducing the battery consumption of
IoT devices and fossil fuels used to generate electricity.
Moreover, compared to the optimizationbased FL approaches that reduce necessary communication rounds, our method allows more clients to participate in each learning round under the ﬁxed bandwidth, which would improve convergence speed and accuracy further. ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.2021-0-02068, Artiﬁcial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) grant funded by the
Korea government (MSIT) (No. NRF-2021R1C1C1006799), and the ”HPC Support” Project supported by the ‘Ministry of Science and ICT’ and NIPA.
S UPPLEMENTARY
In this supplementary material, we present additional details, results, and experiments that are not included in the main paper due to the space limit.
The contents of this supplementary material are listed as follows:
C ONTENTS A Maximal Rank Property B Additional Techniques C Details of Experiment Setup
D Additional Experiments
A M AXIMAL R ANK P ROPERTY
This section presents the proofs of our propositions and an additional analysis of our method’s algorithmic behavior in terms of the rank property.
A. P ROOFS
Proposition 1 Let X 1 ∈ R m × r 1 , X 2 ∈ R m × r 2 , Y 1 ∈ R n × r 1 , Y 2 ∈ R n × r 2 , r 1 , r 2 ≤ min( m, n ) and the constructed matrix be W :=
( X 1 Y (cid:62) 1 ) (cid:12) ( X 2 Y (cid:62) 2 ) .
Then, rank( W ) ≤ r 1 r 2 .
Proof.
X 1 Y (cid:62) 1 and X 2 Y (cid:62) 2 can be expressed as the summation of rank-1 matrices such that X i Y (cid:62) i = (cid:80)
j = r i j =1 x ij y (cid:62) ij , where x ij and y ij are the j -th column vectors of X i and Y i , and i ∈ { 1 , 2 } .
Then,
W = X 1 Y (cid:62) 1 (cid:12) X 2 Y (cid:62) 2 =
j = r 1 (cid:88)
j = x 1 j y (cid:62) 1 j (cid:12) j
= r 2 (cid:88)
j = x 2 j y (cid:62) 2 j = k
= r 1 (cid:88) k
=
j = r 2 (cid:88) j
=
[Equation]
W is the summation of r 1 r 2 number of rank-1 matrices; thus, rank( W ) is bounded above r 1 r 2 .
(cid:3)
Proposition 2 Given R ∈ N , r 1 = r 2 =
R is the unique optimal choice of the following criteria,
[Equation]
and its optimal value is 2 R ( m + n ) .
Proof.
We use arithmetic-geometric mean inequality and the given constraint.
We have
[Equation]
The equality holds if and only if r 1 = r
=
R by the arithmetic–geometric mean inequality.
(cid:3)
Corollary 1 Under Proposition 2, R 2 ≥ min( m, n ) is a necessary and sufﬁcient condition for achieving the maximal rank of W =
( X 1 Y (cid:62) 1 ) (cid:12) ( X 2 Y (cid:62) 2 )
∈
R m × n , where X 1 ∈ R m × r 1 , X 2 ∈ R m × r, Y 1 ∈ R n × r 1 , Y 2 ∈ R n × r 2 , and r 1 , r 2 ≤ min( m, n ) .
Proof.
We ﬁrst prove the sufﬁcient condition.
Given r 1 = r 2 =
R under Proposition 2 and R 2 ≥ min( m, n ) , rank( W ) ≤ min( r 1 r 2 , m, n ) =
min( R 2 , m, n ) = min( m, n ) .
The matrix W has no low-rank restriction; thus the condition, R 2 ≥ min( m, n ) , is the sufﬁcient condition.
The necessary condition is proved by contraposition; if R 2 < min( m, n ) , the matrix W cannot achieve the maximal rank.
Since r 1 = r 2 = R under Proposition 2and R 2 < min( m, n ) , then rank( W ) ≤ min( r 1 r 2 , m, n ) =
min( R 2 , m, n ) =
R 2 < min( m, n ) .
That is, rank( W ) is upperbounded by R 2 , which is lower than the maximal achievable rank of W .
Therefore, the condition,
R 2 ≥ min( m, n ) , is the necessary condition because the contrapositive is true.
(cid:3)
Corollary 1 implies that, with R 2 ≥ min( m, n ) , the constructed weight W does not have the lowrank limitation, and allows us to deﬁne the minimum inner rank as r min :
= min( (cid:100)√ m (cid:101) , (cid:100)√ n (cid:101) ) .
If we set r 1 = r 2 = r min , rank( W ) of our FedPara can achieve the maximal rank because r 1 r 2 = r 2 min ≥ min( m, n ) while minimizing the number of parameters.
Proposition 3 Let T 1 , T 2 ∈ R R
×
R × k 3 × k 4 , X 1 , X 2 ∈ R k 1 × R , Y 1 , Y 2 ∈ R k 2 × R , R ≤ min( k 1 , k 2 ) and the convolution kernel be W :=
( T 1 × 1 X 1 × 2 Y 1 ) (cid:12) ( T 2 × 1 X 2 × 2 Y 2 ) .
Then, the rank of the kernel satisﬁes rank( W (1) ) =
rank( W (2) ) ≤ R 2 .
Proof.
According to Rabanser et al.
(2017), the 1 st and 2 nd unfolding of tensors can be expressed as
[Equation]
where I (3) ∈ R k 3 × k 3 and I (4) ∈
R k 4 × k 4 are identity matrices.
Since W (1) and W (2) are matrices, we apply the same process used in Eq.
2, then we obtain rank( W (1) )
=
rank( W (2) ) ≤ R 2 .
(cid:3)
A.
A NALYSIS OF THE RANK PROPERTY
To demonstrate our propositions empirically, we sample the parameters randomly and count rank( W ) .
When applying our parameterization to W ∈ R 100 × 100 , we set r min = 10 by Corollary 1 to Proposition 2.
We sample the entries of X 1 , X 2 , Y 1 , Y 2 ∈ R 100 × 10 from the standard
Gaussian distribution and repeat this experiment 1 , 000 times.
As shown in Figure 6, we observe that our parameterization achieves the full rank with the probability of 100% but requires 2.5 times fewer entries than the original 100 × 100 matrix.
This empirical result demonstrates that our parameterization can span the full-rank matrix with fewer parameters efﬁciently.
[Figure]
B A DDITIONAL T ECHNIQUES
We devise additional techniques to improve the accuracy and stability of our parameterization.
We consider injecting a non-linearity in FedPara and Jacobian correction regularization to further squeeze out the performance and stability.
However, as will be discussed later, these are optional.
Non-linear Function.
The ﬁrst is to apply a non-linear function before the Hadamard product.
FedPara composites the weight as W = W 1 (cid:12) W 2 , and we inject non-linearity as W = σ ( W 1 ) (cid:12) σ ( W 2 ) as a practical design.
This makes our algorithm departing from directly applying the proofs, i.e ., empirical heuristics.
However, the favorable performance of our FedPara regardless of this non-linearity injection in practice suggests that our re-parameterization is a reasonable model to deploy.
The non-linear function σ ( · ) can be any non-linear function such as ReLU , Tanh , and Sigmoid , but
ReLU and Sigmoid restrict the range to only positive, whereas Tanh has both negative and positive values of range [ − 1 , 1] .
The ﬁlters may learn to extract the distinct features, such as edge, color, and blob, which require both negative and positive values of the ﬁlters.
Furthermore, the reasonably bounded range can prevent overshooting a large value by multiplication.
Therefore, Tanh is suitable, and we use Tanh through experiments except for those of pFedPara .
Jacobian Correction.
The second is the Jacobian correction regularization, which induces that the
Jacobians of X 1 , X 2 , Y 1 , and Y 2 follow the Jacobian of the constructed matrix W .
Suppose that
X 1 , X 2 ∈ R m × r and Y 1 , Y 2 ∈ R n × r are given.
We construct the weight as W = W 1 (cid:12) W 2 , where W 1 = X 1 Y (cid:62) 1 and W 2 = X 2 Y (cid:62) 2 .
Additionally, suppose that the Jacobian of W with respect to the objective function is given: J W = ∂
L ∂ W .
We can compute the Jacobians of X 1 , X 2 , Y 1 , and
Y 2 with respect to the objective function and apply one-step optimization with SGD.
For simplicity, we set the momentum and the weight decay to be zero.
Then, we can compute the Jacobian of other variables using the chain rule:
[Equation]
We update the parameters using SGD with the step size η as follows:
[Equation]
We can compute W (cid:48) , which is the constructed weight after one-step optimization:
[Equation]
This shows that gradient descent and ascent are mixed, as shown in the signs of each term.
We propose the Jacobian correction regularization to minimize the difference between W (cid:48) and W − η J W , which induces our parameterization to follows the direction of W − η J W .
The total objective function consists of the target loss function and the Jacobian correction regularization as:
[Equation]
Results.
We evaluate the effects of each technique in Table 4.
We train VGG16 with group normalization on the CIFAR-10 IID setting during the same target rounds.
We set γ = 0 .
1 and λ = 10 .
As
[Table]
shown, the model with both Tanh and regularization has higher accuracy and lower variation than the base model.
There is gain in accuracy and variance with both techniques, whereas only variance with only one technique.
Again, note that these additional techniques are not essential for FedPara to work; therefore, we can optionally use these techniques depending on the situation where the device has enough computing power.
C D ETAILS OF E XPERIMENT S ETUP
In this section, we explain the details of the experiments, including datasets, models, and hyperparameters.
We also summarize our FedPara and pFedPara into the pseudo algorithm.
For implementation, we use PyTorch Distributed library (Paszke et al., 2019) and 8 NVIDIA GeForce
RTX 3090 GPUs.
C. D ATASETS
CIFAR-10.
CIFAR-10 (Krizhevsky et al., 2009) is the popular classiﬁcation benchmark dataset.
CIFAR-10 consists of 32 × 32 resolution images in 10 classes, with 6 , 000 images per class.
We use 50 , 000 images for training and 10 , 000 images for testing.
For federated learning, we split training images into 100 partitions and assign one partition to each client.
For the IID setting, we split the dataset into 100 partitions randomly.
For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0 .
5 as suggested by He et al.
(2020b;a).
CIFAR-100.
CIFAR-100 (Krizhevsky et al., 2009) is the popular classiﬁcation benchmark dataset.
CIFAR-100 consists of 32 × 32 resolution images in 100 classes, with 6 , 000 images per class.
We use 50 , 000 images for training and 10 , 000 images for testing.
For federated learning, we split training images into 50 partitions.
For the IID setting, we split the dataset into 50 partitions randomly.
For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0 .
5 as suggested by He et al.
(2020b;a).
CINIC-10.
CINIC-10 (Darlow et al., 2018) is a drop-in replacement for CIFAR-10 and also the popular classiﬁcation benchmark dataset.
CINIC-10 consists of 32 × 32 resolution images inclasses and three subsets: training, validation, and test.
Each subset has 90 , 000 images with 9 ,per class, and we do not use the validation subset for training.
For federated learning, we split training images into 100 partitions.
For the IID setting, we split the dataset into 100 partitions randomly.
For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0 .
5 as suggested by He et al.
(2020b;a).
MNIST.
MNIST (LeCun et al., 1998) is a popular handwritten number image dataset.
MNIST consists of 70 , 000 number of 28 × 28 resolution images in 10 classes.
We use 60 , 000 images are for training and 10 , 000 images for testing.
We do not use MNIST IID-setting, and we split the dataset so that clients have at most two classes as suggested by McMahan et al.
(2017) for a highly-skew non-IID setting.
FEMNIST.
FEMNIST (Caldas et al., 2018) is a handwritten image dataset for federated settings.
FEMNIST has 62 classes and 3 , 550 clients, and each client has 226 .
83 data samples on average of 28 × 28 resolution images.
FEMNIST is the non-IID dataset labeled by writers.
Shakespeare.
Shakespeare (Shakespeare, 1994) is a next word prediction dataset for federated learning settings.
Shakespeare has 80 classes and 1 , 129 clients, and each client has 3 , 743 .
2 data samples on average (Caldas et al., 2018).
C. M ODELS
VGG16.
PyTorch library (Paszke et al., 2019) provides VGG16 with batch normalization, but we replace the batch normalization layers with the group normalization layers as suggested by Hsieh et al.
(2020) for federated learning.
We also modify the FC layers to comply with the number of classes.
The dimensions of the output features in the last three FC layers are 512 – 512 – (cid:104)
# classes (cid:105) , sequentially.
We do not apply our parameterization to the last three FC layers and set the same γ to all convolutional layers in the model for simplicity.
For reference purpose, Table 5 shows the number of parameters corresponding to each γ .
[Table]
Two FC Layers.
In personalization experiments, we use two FC layers as suggested by McMahan et al.
but modify the size of the hidden features corresponding to the number of classes in the datasets.
The dimensions of the output features in two FC layers are 256 and the number of classes, respectively; i.e ., 256 – (cid:104)
# classes (cid:105) .
We do not use other layers, such as normalization and dropout, and set γ = 0 . 5 for pFedPara .
LSTM.
For the Shakespeare dataset, we use two-layer LSTM as suggested by McMahan et al.
(2017) and Acar et al.
We set the hidden dimension as 256 and the number of classes as 80.
We also apply the weight normalization technique on original parameterization, low-rank parameterization, and FedPara .
C. F ED P ARA & P F ED P ARA
Algorithm Input: rounds T , parameters { X 1 l , X 2 l , Y 1 l , Y 2 l } l = L l =1 where { X 1 l , X 2 l , Y 1 l , Y 2 l } is the l th layer of the model and L is the number of layers for t = 1 , 2 , . .
, T do
Sample the subset S of clients; for each client c ∈ S do
Download { X 1 l , X 2 l , Y 1 l , Y 2 l } l = L l =1 from the server;
Optimize( { X 1 l , X 2 l , Y 1 l , Y 2 l } l = L l =1 );
Upload { X 1 l , X 2 l , Y 1 l , Y 2 l } l = L l =1 to the server; end
Aggregate { X 1 l , X 2 l , Y 1 l , Y 2 l } l = L l =1 ; end
We summarize our two methods, FedPara and pFedPara , into Algorithms 1 and 2 for apparent comparison.
In these algorithms, we mainly use the popular and standard algorithm, FedAvg , as a backbone optimizer, but we can switch with other optimizer and aggregate methods.
As mentioned in Section B, we can consider the additional techniques.
In the FedPara experiments, we use the regularization to Algorithm 1, and set the regularization coefﬁcient λ as 1.0.
In pFedPara experiments, we do not apply the additional techniques to Algorithm 2.
C. H YPER - PARAMETERS OF B ACKBONE O PTIMIZER
FedAvg (McMahan et al., 2017) is the most popular algorithm in federated learning.
The server samples S number of clients as a subset in each round, each client of the subset trains the model locally by E number of SGD epochs, and the server aggregates the locally updated models and repeats these processes during the total rounds T .
We use FedAvg as a backbone optimization algorithm, and its hyper-parameters of our experiments, such as the initial learning rate η , local batch size B , and learning rate decay τ , are described in Table 6.
[Table]
C. H YPER - PARAMETERS OF O THER O PTIMIZERS
For compatibility experiment, we combine FedPara with other optimization-based FL algorithms:
FedProx (Li et al., 2020), SCAFFOLD (Karimireddy et al., 2020), FedDyn (Acar et al., 2021), and FedAdam (Reddi et al., 2021).
FedProx (Li et al., 2020) imposes a proximal term to the objective function to mitigate heterogeneity; SCAFFOLD (Karimireddy et al., 2020) allows clients to reduce the variance of gradients by introducing auxiliary variables; FedDyn (Acar et al., 2021) introduces dynamic regularization to reduce the inconsistency between minima of the local device level empirical losses and the global one; FedAdam employs Adam (Kingma & Ba, 2015) at the server-side instead of the simple model average.
They need a local optimizer to update the model in each client, and we use the SGD optimizer for a fair comparison, and the SGD conﬁguration is the same as that of FedAvg .
The four algorithms have additional hyper-parameters.
FedProx has a proximal coefﬁcient µ , and we set µ as 0 . 1 .
SCAFFOLD has Options I and II to update the control variate, and we use Option II with global learning rate η g ( = 1 . 0 ).
FedDyn has the hyper-parameter α ( = 0 . 1 ) in the regularization.
FedAdam uses Adam optimizer to aggregate the updated models at the server-side, and we use the
[Table] [Table]
parameters β 1 = 0 .
9 , β 2 = 0 .
99 , the global learning rate η g =
D A DDITIONAL E XPERIMENTS
In this section, we simulate the computation time, the communication time, and total training time in Section D.1, experiment about other models in Section D.2, and compare our method and the quantization approach in Section D.3.
D. T RAINING TIME
We can compute the elapsed time during one round in FL by t = t comp.
+ t comm. , where t comp.
is the computation time of training the model on local data for several epochs and t comm.
is the communication time of downloading and uploading the updated model.
We estimate the computation time by measuring the elapsed time during local epochs.
We compute the communication time by 2 · [Table] network speed (Mbyte / s) , considering upload and download of the model in one round.
Since it is challenging to experiment in a real heterogeneous network environment, we follow the simple standard network simulation setting widely used in the communication literature (Zhu et al., 2020; Jeon et al., 2020).
They assume the homogeneous link quality by the average quality to simplify complex network environments in FL communication simulation, i.e., the network speeds are identical for all clients.
We compare the elapsed time per round of VGG16 ori .
and VGG16 FedPara on different network speeds such as 2, 10, and 50 Mbps.
As shown in Table 7, the communication time is larger than the computation time indicating that the communication is a bottleneck in FL as expected.
Although
VGG16 FedPara takes more computation time than VGG16 ori .
due to the weight composition time, VGG
FedPara decreases the communication time by about ten times and the total time by 4. to 9.52 times.
Table 8 shows the total training time to achieve the same accuracy on the CIFAR10 IID setting.
Although FedPara needs three rounds more than the original parameterization,
VGG16 FedPara requires 4.67 to 9.68 times less training time than VGG16 ori .
because of communication efﬁcient parameterization.
[Figure]
Acc.
of 200 rounds
Acc. of 1000 rounds (gain)
Original 83.
84.1 (+ 0.42)
FedPara ( γ = 0 . 1) 82.
83.35 (+ 0.47)
FedPara ( γ = 0 . 2) 82.
83.34 (+ 0.81)
FedPara ( γ = 0 . 3) 83.
83.94 (+ 0.83)
FedPara ( γ = 0 . 4) 84.
84.59 (+ 0.54)
FedPara ( γ = 0 . 5) 83.
84.57 (+ 0.75)
FedPara ( γ = 0 . 6) 83.
84.16 (+ 0.53)
FedPara ( γ = 0 . 7) 83.
84.24 (+ 0.45)
FedPara ( γ = 0 .
84.00 (+ 0.6)
D. O THER M ODELS
VGG16 As shown in Figure 7, we compare VGG16 ori .
and VGG16 FedPara with three different γ values.
Note that a higher γ uses more parameters than a lower γ .
Compared to
VGG16 ori . ,
VGG16 FedPara achieves comparable accuracy and even higher accuracy with a high γ .
VGG16 FedPara also requires fewer communication costs than VGG16 ori . .
We also investigate how much accuracy is increased for longer rounds.
Table 9 shows that the accuracy is increased in the long round experiment, but the tendency of the 1000 round training is consistent with the 200 round training.
ResNet18 We demonstrate the consistent effectiveness of our method with another architecture,
ResNet18 .
For ResNet18 , we train without replacing batch normalization layers, set γ of the ﬁrst layer, the second layer, and the 1 × 1 convolution layers as 1 . 0 , and adjust γ of remaining layers to control the ResNet18 FedPara size; γ = 0 .
1 for small model size, γ = 0 .
6 for mid model size, and γ = 0 .
9 for large model size.
To train the ResNet18 FedPara in FL, we set the batch size to 10 .
As revealed in Figure 8a, ResNet18 FedPara has comparable accuracy and uses fewer communication costs than ResNet18 ori . , of which the results are consistent with the VGG16 experiments.
Figure 8b shows the communication costs required for model training to achieve the same tar-
[Figure] [Figure]
A cc .
ResNet18 ori.
ResNet
[Figure]
[Figure] [Figure]
( =
[Figure]
ResNet18 FedPara ( =0.9) (a) Accuracy ResNet18 ori.
ResNet18 FedPara ( =0.1) ResNet18 FedPara ( =0.6) ResNet18 FedPara ( =0.9)
C o mm un i c a t i o n C o s t
[ G
B ] 169.
[Figure]
[Table]
get accuracy.
Our ResNet18 FedPara needs 1 .
17 to 5 .
1 times fewer communication costs than
ResNet18 ori . , and the results demonstrate that FedPara is also applicable to the ResNet structure.
Pufferﬁsh Pufferﬁsh (Wang et al., 2021) is similar to our method, where they use partially prefactorized networks and employ the hybrid architecture by maintaining original size weights to minimize the accuracy loss due to the low-rank constraints.
Since this work can be directly applied in the FL setup, we compare the parameterization of PufferFish and FedPara as follows.
We train VGG16 with PufferFish and FedPara on the CIFAR-10 IID dataset in FL, and evaluate the models according to varying the number of parameters.
As shown in Table 10, our FedPara has higher accuracy with fewer parameters compared with PufferFish.
Although PufferFish is superior to the naive low-rank pre-decomposition method due to its hybrid architecture, we think the hybrid architecture of PufferFish still suffers from the low-rank constraints on the top layers.
However, our method is free from such limitations both theoretically and empirically.
LSTM We train LSTM ori .
, LSTM low , and LSTM FedPara on the Shakespeare dataset.
The parameters ratio of LSTM low
and LSTM FedPara are about 16% and 19% of the LSTM ori .
parameters, respectively.
Table 11 shows that LSTM FedPara outperforms LSTM low and LSTM ori .
on the IID setting.
In the nonIID setting, LSTM FedPara accuracy is higher than LSTM low and slightly lower than LSTM ori .
with only 19% of parameters.
Therefore, our parameterization can be applied to general neural networks.
D. Q UANTIZATION
FedPAQ (Reisizadeh et al., 2020) is a quantization approach to reduce communication costs for
FL.
To compare the accuracy and transferred size per round, we train VGG16 on the CIFAR-10 IID setting.
We consider both downlink and uplink to evaluate communication costs.
We quantize the model from 32 bits ﬂoating-point numbers to 16 bits ﬂoating-point numbers.
[Table]
[Table]
Table 12 shows the comparison between FedPara and FedPAQ as well as those integration.
Compared to FedPAQ , FedPara transfers 1.96 times lower bits per round because FedPAQ only reduces the uplink communication cost.
Compared with FedAvg , FedPara achieves 0.14 % higher accuracy, but FedPAQ 1.01 % lower accuracy.
FedPara can transfer the full information of updated weights to the server, whereas FedPAQ loses the updated weight information due to the quantization.
Furthermore, since the way FedPara reduces communication costs is different from quantization, we integrate FedPara with FedPAQ as an extension.
Combining our method with
FedPAQ reduces communication costs by 25% further from our FedPara without the integration while having a minor accuracy drop, 0.1 % , from that of FedAvg .