AQUILA: C OMMUNICATION E FFICIENT F EDERATED L EARNING WITH A DAPTIVE Q UANTIZATION OF L AZILY -A GGREGATED G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
The development and deployment of federated learning (FL) have been bottlenecked by the heavy communication overheads of high-dimensional models between the distributed client nodes and the central server.
To achieve better errorcommunication tradeoffs, recent efforts have been made to either adaptively reduce the communication frequency by skipping unimportant updates, a.k.a.
lazilyaggregated quantization (LAQ), or adjust the quantization bits for each communication.
In this paper, we propose a unifying communication efficient framework for FL based on a daptive qu ant i zation of l azily- a ggregated gradients ( AQUILA ), which adaptively adjusts two mutually-dependent factors, the communication frequency and the quantization level, in a synergistic way.
Specifically, we start from a careful investigation on the classical LAQ scheme and formulate AQUILA as an optimization problem where the optimal quantization level per communication is selected by minimizing the gradient loss caused by updates skipping.
Meanwhile, we adjust the LAQ strategy to better fit the novel quantization criterion and thus keep the communication frequency at an appropriate level.
The effectiveness and convergence of the proposed AQUILA framework are theoretically verified.
The experimental results demonstrate that AQUILA can reduce around 50% of overall transmitted bits compared to existing methods while achieving the same level of model accuracy in a number of non-homogeneous FL scenarios, including Non-IID data distribution and heterogeneous model architecture.
The proposed AQUILA is highly adaptive and compatible to existing FL settings.
I NTRODUCTION
With the deployment of ubiquitous sensing and computing devices, the Internet of things (IoT) as well as many other distributed systems have gradually grown from concept to reality, bringing dramatic convenience to people’s daily life (Du et al., 2020; Liu et al., 2020a; Hard et al., 2018).
To fully utilize such distributed computing resources, distributed learning provides a promising framework that can achieve comparable performance with the traditional centralized learning scheme.
However, the privacy and security of sensitive data during the updating and transmitting processes in distributed learning have been a growing concern.
In this context, federated learning (FL) (McMahan et al., 2017; Yang et al., 2019) has been developed, enabling distributed devices to collaboratively learn a global model without privacy leakage by keeping private data sets isolated and masking transmitted information with secure approaches like differential privacy (Abadi et al., 2016), secret sharing techniques (Bonawitz et al., 2017) and homomorphic encryption (Liu et al., 2020b).
Due to its privacy and security preserving property and great potentials in some distributed but privacysensitive fields like finance and health, FL has attracted tremendous attentions from both academia and industry in recent years.
Unfortunately, in many FL applications like image classification and objective recognition, the models to be trained tend to be high-dimensional, which lead to heavy communication overheads, for example, a Resnet-152 network has over 58 million parameters (He et al., 2016).
Hence, communication efficiency has become one of the key bottlenecks of FL.
To this end, recent researches have tried to reduce the communication frequency, for example, Sun et al.
(2020) proposes the lazily-aggregated quantization (LAQ) method to reduce communication rounds by skipping some unnecessary parameter uploads.
To further reduce transmitted bits per communication, LAQ can be used jointly with gradient compression techniques, e.g. quantization and sparsification (Strom, 2015; Wangni et al., 2018; Lin et al., 2018; Han et al., 2020).
Moreover, Mao et al.
(2021) develops the Adaptive Quantized Gradient (AQG) for LAQ to adjust the quantization bit among multiple given levels during training.
However, the AQG is not sufficiently adaptive, for example, in the two-level AQG with 4 bit and 2 bit, the situation of 3 bit and 1 bit is not covered at all.
In a separate line of work, Jhunjhunwala et al.
(2021) develops an adaptive quantization rule (AdaQuantFL) for FL which can search in a given range for an optimal quantization level and achieve a better error-communication tradeoff.
Previous work has investigated how to optimize communication frequency or adjust quantization levels in a highly adaptive fashion, but not both.
Intuitively, we ask the question, can we adaptively adjust the quantization level in LAQ to further reduce communication rounds and transmitted bits simultaneously?
A straightforward approach is to train LAQ jointly with state-of-the-art adaptive quantization methods like AdaQuantFL.
However, quantization level and communication frequency are mutually dependent in establishing the model convergence and must be adjusted together.
For example, lazy aggregation leads to the skip of some gradients that no longer require quantization, while the choice of quantization levels directly affects the quality of a gradient update and thus whether it is selected for transmission and aggregation.
Therefore, the key question of our work is how to jointly leverage these two complementary yet mutually-dependent degrees of freedom for further optimizing communication efficiency in FL.
The key idea of this paper is to select the optimal quantization bits for each communication round in LAQ by optimizing the gradient loss caused by skipping quantized updates, which gives a novel quantization criterion that can cooperate with LAQ strategy to further reduce overall transmitted bits while maintaining the desired convergence properties of LAQ.
The contributions of this paper are trifold and summarized as follows.
We propose a FL framework with a daptive qu ant i zation of l azily- a ggregated gradients termed AQUILA, which simultaneously adjusts the communication frequency and quantization level in a synergistic fashion.
We formulate AQUILA as an optimization problem and develop an upperbound for the gradient loss caused by communication skipping, which gives a novel adaptive quantization criterion which is theoretically proven to be more efficient compared to AdaQuantFL while maintaining the same convergence properties.
We experimentally evaluate the performance of AQUILA in a number of non-homogeneous FL settings, including Non-IID data distribution and various heterogeneous model architecture.
The experimental results show that AQUILA can significantly mitigate the communication overhead over a number of baselines including fixed-bit LAQ and the naive combination of LAQ and AdaQuantFL.
Our approach is highly adaptive and compatible to existing FL settings.
B ACKGROUND AND R ELATED W ORK
Consider a FL system with one central server and a clients set M of M distributed clients to collaboratively train a global model parameterized by θ ∗ .
At iteration k , each client m ∈ M first trains the global model θ k on its local data D km , and sends the local gradient g km = ∇ f m ( D km ; θ k ) to the central server.
Then the server aggregates the parameters and updates the global parameter by:
[Equation]
To reduce communication overheads with gradient quantization, the stochastic uniform quantizer (Alistarh et al., 2017) is usually adopted.
For any local gradient g ∈ R d , the quantized value of its i -th dimension with quantization level b is defined as:
[Equation]
where ξ i ( g , b ) is a random variable defined as follows.
Let l ∈ { 0 , 1 , 2 , ..., b − 1 } be an integer satisfying |
g i | / ∥ g ∥ 2 ∈ [ l/b, ( l + 1) /b ) , then: ξ i ( g , b ) = (cid:26) ( ℓ + 1) /b with probability ( b · ∥ g i ∥ ) /
∥ g ∥ 2 − l ℓ/b otherwise.
It is clear that with quantization level b , the number of bits for transmitting a quantized gradient from a client to the central server is C b = d ⌈ log 2 ( b + 1) ⌉
+ d + 32 , with 32 bits for ∥ g ∥ 2 , 1 bit for each sign ( g i ) , and log 2 ( b + 1) bits for each ξ i ( g , b ) .
For communication rounds reduction, the LAQ proposes to let the client m ∈ M upload its newlyquantized local gradient Q b ( g km ) at iteration k only when the change in local gradient is sufficiently large, i.e.,
[Equation]
where Q b (ˆ g k − 1 m ) is the last quantized upload from client m , ε b (ˆ g k − 1 m ) and ε b ( g km ) denote quantization errors, and { ξ d } Dd =1 are some predetermined constant weights.
Notice here a fixed quantization level b is used.
In LAQ, if the difference between client m ’s newly-quantized local gradient Q b ( g km ) and the last upload is smaller than a threshold involving quantization errors and global model’s innovation, client m will skip the upload of Q b ( g km ) at iteration k and the central server will reuses
Q b (ˆ g k − 1 m ) for such lazy aggregation:
[Equation]
where M k 0 denotes the subset of clients that skip the new gradient update and reuse the old quantized gradient at iteration k .
Besides, ˆ g km represents the actual gradient for aggregation from client m , which is g km for m ∈ M \ M k 0 , while ˆ g k − 1 m for m ∈ M k 0 .
Recently, AdaQuantFL is proposed to achieve a better error-communication tradeoff by adaptively adjusting the quantization levels during FL training (Jhunjhunwala et al., 2021).
Specifically,
AdaQuantFL computes iteration k ’s optimal quantization level b ∗ k based on the following criterion involving training loss and initial quantization level b 0 :
[Equation]
where f ( θ 0 ) and f ( θ k ) are the global training loss at iteration 0 and k , respectively.
However, AdaQuantFL transmits quantized gradients at every iteration .
In order to skip unnecessary communication rounds and adaptively adjust quantization level for each communication jointly, an naive approach is to quantize lazily aggregated gradients with AdaQuantFL.
Nevertheless, it fails to achieve efficient communication due to a number of reasons.
Firstly, given the descending trend of training loss, AdaQuantFL’s criterion (6) may lead to high quantization bit number even exceeding 32 late in the training process, which is too much for cases where the global convergence is already approaching.
Secondly, higher quantization level results in smaller quantization error, which will lead to lower communication threshold in LAQ’s criterion (4) and thus higher frequency of transmission.
Therefore it is desirable to develop more efficient adaptive quantization method in the lazily-aggregated setting to systematically improve communication efficiency in FL.
M ETHOD 3. A DAPTIVE Q UANTIZATION OF L AZILY -A GGREGATED G RADIENTS (AQUILA)
Given the above limitations of the naive joint use of existing adaptive quantization criterion and lazy aggregation strategy, this paper aims to design a unifying framework for communication efficiency optimization where the quantization level and communication frequency are adjusted in a synergistic and interactive way.
Based on a careful investigation on LAQ, we design a novel quantization criterion where the optimal quantization level is selected by minimizing the expected gradient loss caused by skipping quantized updates.
The rationale behind such strategy is, by formulating the adaptive quantization problem of lazily-aggregated gradients as optimizing the expected gradient loss with respect to the number of quantization bits, we can get an adaptive quantization criterion based on local gradient updates while maintaining or even improving the convergence property of LAQ.
To get the optimization target, we first derive an upperbound for the expected gradient loss in terms of quantization bits as elaborated in Section 3.2, which gives a novel adaptive quantization criterion (7) that selects the quantization level for client m at iteration k based on initial quantization level b 0 and the change between client m ’s newly computed gradient g km and the last uploaded gradient ˆ g k − 1 m :
[Equation]
The superiority of (7) comes from the following two aspects.
Firstly, the gradient updates tend to fluctuate along with the training process instead of keeping descending like the loss value, and thus prevent the quantization level from increasing tremendously compared with the initial level.
Secondly, with lazy aggregation criterion based on gradient updates like (4), the transmitted bits in AQUILA is further controlled, since the gradient update for actual transmission in (7) is lower bounded by the lazy aggregation criterion, and therefore high-bit transmission for small update is more likely to be skipped.
To better fit the larger quantization errors induced by fewer quantization bits in (7), we modify the communication criterion as follows to avoid the potential expansion of clients group to be skipped:
[Equation]
k − 1 m ) (cid:13)(cid:13)(cid:13) 2 2 , (8) where all the notations are the same as in (4) except the heterogeneous quantization level b km and ˆ b k − 1 m for each client.
For detailed development of (8), please refer to the Appendix.
(cid:2189) (cid:2778)(cid:2193) (cid:28663)
[Figure]
Client Client
Client m Server (cid:28663) (cid:2189) (cid:2779)(cid:2193) (cid:2189) (cid:2780)(cid:2193) (cid:2189) (cid:2195)(cid:2193) skip
[Table]
(cid:3286)
[Table]
(cid:2779)(cid:2193) (cid:1843)
[Table]
(cid:2780)(cid:2193) (cid:1843)
[Table]
(cid:3286) [Table]
(cid:2195)(cid:2193) (cid:2242)
[Table]
=
[Table]
) (cid:1854)
[Table]
LAQ
[Table]
=
[Table] [Table] [Figure] The cooperation of the novel adaptive quantization criterion (7) and the modified lazy aggregation strategy (8) is illustrated in Fig.
1a.
Compared to the naive combination of AdaQuantFL and
LAQ where the mutual influence between adaptive quantization and lazy aggregation has not been considered as shown in Fig.
1b, our AQUILA framework adaptively optimizes the allocation of quantization bits throughout training to promote the convergence of lazy aggregation, and at the same time utilizes the lazy aggregation strategy to improve the efficiency of adaptive quantization by skipping high-bit transmission.
The proposed AQUILA’s effect of suppressing the transmission of high quantization bits has been verified in our experiments, as shown in Fig.
20 in the Appendix. Besides, with the adjusted lazy aggregation strategy (8), AQUILA well addresses the problem of high communication frequency in the late training process of naive combination of AdaQuantFL and LAQ, as indicated by Fig. 2.
The proposed AQUILA is summarized as follows in Algorithm 1.
At iteration k = 0 , all clients are forced to transmit local gradients quantized with the initial level b 0 .
At iteration k ∈ { 1 , 2 , ..., K } , the server first broadcasts the global model θ k to all clients.
Each client m computes g km with local training data, and then uses it to select an optimal quantization level by (7).
Then, each client computes its gradient update after quantization and determines whether to upload the update or not based on the communication criterion (8).
Finally, the server updates the new global model θ k + with up-to-date quantized gradients Q b km ( g km ) for those clients who send the uploads at iteration k , while reusing the old quantized gradients Q ˆ b k − 1 m (ˆ g k − 1 m ) for those who skip the uploads.
[Figure]
Input: the number of communication rounds K , the learning rate α , the maximum communication level b max Initialize do do 9: quantization 11: if 12: b
13: end if 14: if holds then 15:
Client m computes quantized Q
Set sides.
17: else 18: Client m sends nothing.
[-[Figure] -IID     
 N u m b e r-]Set and on both sides.
20: end if 21: end for 22:
Server updates
23: end for
3. T HEORETICAL D ERIVATION AND A NALYSIS OF AQUILA
As mentioned before, in this work we bound the expected gradient loss caused by skipping updates with respect to quantization bits.
Specifically, if the communication criterion (8) holds for client m at iteration k , it does not contribute to iteration k ’s gradient loss, otherwise, the loss caused by client m will be minimized with the optimal quantization criterion (7).
In this section, the theoretical derivation of the target upper bound is based on following standard assumptions:
Assumption 1.
Loss function f ( θ ) =
(cid:88) m ∈ M f m ( θ ) is L-smooth, and f m ( θ ) is L m -smooth.
Assumption 2.
The quantization operation is unbiased with E [ Q b ( w ) | w ] = w , and its variance satisfy: ∀ w ∈ R d , E [ ∥ Q b ( w ) − w ∥ 22 | w ] ≤ q b ∥
w ∥ 22 , where q b is a positive constant.
Here we adopt the definition of B in AdaQuantFL, but modified it as follows:
Definition 1 (Number of Bits sent from client m to the server, B m ).
The total number of bits that has been sent from client m to the central server until a given time instant is denoted by B m .
With the definition of B m , the upper bound of the loss induced by gradient skipping is indicated by the following theorem:
Theorem 1.
Under Assumption 1 and 2, the expected gradient loss caused by lazy aggregation has the following upper bound:
[Equation]
where σ 2 denotes the maximum ∥ w ∥ 22 , and d m is a positive constant determined by L m .
See the
Appendix for proof details.
For each client m , the optimal b km is selected by setting the first derivative of the upper bound (9) to zero:
[Equation]
Therefore, an adaptive quantization criterion which does not rely on unknown parameters such as σ and d m can be achieved through dividing b km by b 1 m , which gives the adaptive selection criterion in (7) if b 1 m for each client m is chosen as the initial quantization level b 0 .
Comparison with AdaQuantFL.
In AdaQuantFL, the optimal quantization level is selected by minimizing the upper bound of the expected squared norm of the gradient ∇ f ( θ k ) of a non-convex objective function, and thus the convergence of AdaQuantFL is guaranteed.
However, the proposed
AQUILA is proven to be more efficient with sufficiently small learning rates as shown in Theorem 2.
Besides, it is theoretically guaranteed that the squared norm of the global gradient ∇ f ( θ k ) is also able to converge under AQUILA’ s adaptive quantization criterion developed for lazy aggregation.
Theorem 2.
Under Assumption 1 and 2, if the learning rate α satisfies α ≤ (1 /ML ) · (cid:112)
Pξ/L , the following inequality holds:
[Equation]
where ξ and P are both positive constants.
See the Appendix for proof details.
Theorem 2 indicates that with appropriate learning rate, the proposed AQUILA uses less number of bits per communication compared to AdaQuantFL.
Convergence analysis.
The Lyapunov function of AQUILA is defined in the same way as LAQ and
AQG (Mao et al., 2021):
[Equation]
(12) where θ ∗ is the optimal solution of min θ f ( θ ) .
Convergence guarantee of federated learning with lazy aggregation has been well discussed in Sun et al.
More specifically, both the objective residual f ( θ k ) − f ( θ ∗ ) and the parameter differences term in Lyapunov function are guaranteed to descend along with the training process.
Therefore, the squared norm of the gradient ∇ f ( θ k ) is also guaranteed to converge based on the
L -smoothness assumption which results in: ∥∇ f
( θ k ) ∥ 22 ≤ 2 L [ f ( θ k ) − f
( θ ∗ )] .
E XPERIMENTS AND D ISCUSSION 4. E XPERIMENT S ETUP
Dataset.
In this paper, we evaluate our method with MNIST and CIFAR10 dataset, considering both
IID and Non-IID data distribution.
To simulate Non-IID situation, each client is assigned with two classes of data at most and the amount of data for each class is balanced.
Parameters.
We set total client number M = 10 , and follow the settings in LAQ, where D =and ξ 1 = ξ 2 = · · · = ξ D = 0 . 8 /D .
In terms of initial quantization level, a low level b 0 = 2 is chosen for simpler tasks with MNIST, and a larger level b 0 = 6 is selected for more complex tasks with CIFAR10.
For both AdaQuantFL and AQUILA, we set an upperbound for quantization level as b max = 16 in case that the level grows too high.
Training.
We train a CNN with MNIST and a Resnet18 network with CIFAR10.
The hyperparameters of our experiments are shown in Table 1 in Appendix.
We first evaluate our proposed AQUILA with homogeneous settings where all the local models share the same architecture as the global model.
The performance of AQUILA is compared with several state-of-the-art methods including FedAvg (McMahan et al., 2017), QSGD (Alistarh et al., 2017),
AdaQuantFL, fixed-bit LAQ and the naive combination of AdaQuantFL with LAQ.
For the choice of quantization level for fixed-bit LAQ, since AQUILA’s initial level b 0 is 2 for MNIST and 6 for
CIFAR10, we compare AQUILA with LAQ-2 and LAQ-6 for MNIST and CIFAR 10 respectively.
Besides, the performance of LAQ with the upperbound b max = 16 is also evaluated.
We also evaluate our proposed AQUILA with HeteroFL (Diao et al., 2020), where the local models trained at clients’ side are heterogeneous.
Assume the global model at iteration
k is θ kg and its size is d g · h g , then the local model of each client m can be selected by θ km =
θ kg
[: d m , : h m ] , where d m = r m d g and h m = r m h g respectively.
In this paper, we choose three various model complexity levels r
= { a, b, c } = { 1 , 0 . 5 , 0 . 25 } .
Fig.
3 shows the training loss vs total transmitted bits curve of experiments with IID MNIST / CIFAR10 for homogeneous model architecture, IID MNIST / CIFAR10 for 100 % -50 % heterogeneous model architecture, and Non-IID MNIST /
CIFAR10 for 100 % -25 % heterogeneous model architecture.
The corresponding transmitted bits vs steps curves of the above experiment settings are shown in Fig.
4, which represents how many bits are transmitted in each step.
Methods without adaptive quantization and lazy aggregation like FedAvg and QSGD are not included in Fig.
4 for simplicity.
All the other experiment results are provided in Appendix, with Fig.
5 to Fig.
9 for homogeneous models, and Fig.
10 to Fig.
19 for heterogeneous models.
4. P ERFORMANCE A NALYSIS
In this part, we analyze the performance of AQUILA with various experiment settings including
Non-IID data distribution and heterogeneous model architecture.
From figures in this paper, we can observe that:
• AQUILA achieves a significant transmission reduction on both MNIST and CIFAR10 as compared to AdaQuantFL and the naive combination of LAQ and AdaQuantFL.
For instance, AQUILA saves 65.98% of transmitted bits compared with AdaQuantFL and 62.05% compared with the naive combination of LAQ and AdaQuantFL in the IID scenario with CIFAR10 dataset as shown in Fig.
3d, and other figures all show an obvious reduction in terms of the total transmitted bits required for convergence.
Fig.
4 verifies that the quantization level selected by AQUILA will not continuously increase during training like AdaQuantFL and LAQ with AdaQuantFL.
• Similarly, AQUILA outperforms all fixed-level LAQ in terms of overall transmitted bits for both MNIST and CIFAR10, as shown in Fig.
For example, Fig. 3a indicates that AQUILA reduces 42.6% of total transmitted bits compared with LAQ-2 and 92.32% compared with LAQ-16 in MNIST.
The transmission reduction is 38.76% for LAQ-6 and 78.09% for LAQ-16 in CIFAR10 as shown in Fig.
3e.
[-M B ) 100%-25%-CIFAR10-Non-IID Parameters LAQ-
-]
[-LAQLAQ-
 AdaQuantFL AdaQuantFL+LAQ AQUILA ( [Figure] [Figure]-]Besides, Fig. 3 and Fig.
4 indicate that although LAQ with fixed but low quantization level like LAQ-2 and LAQ-4 sometimes transmit smaller amount of bits per step compared with AQUILA, they suffer from lower accuracy and slower convergence.
It further verifies the necessity and effectiveness of our well-designed adaptive quantization criterion which achieves fast convergence with similar low-bit transmission but without degradation of the model performance.
• AQUILA also works well with heterogeneous local models.
With the two various ways of distributing the heterogeneous local models in this paper, our proposed AQUILA still outperforms other methods by significantly reducing overall transmitted bits while maintaining the same convergence property and model accuracy.
Please refer to Fig.
10 to Fig.
19 in the Appendix for more detailed information.
These experimental results in nonhomogeneous FL settings prove that our proposed AQUILA can be used in a more general and complicated federated learning scenarios. [Figure]
C ONCLUSIONS AND F UTURE W ORK
This paper proposes a communication efficient FL framework to simultaneously adjust two mutually-dependent degrees of freedom: communication frequency and quantization level.
With the close cooperation of the novel adaptive quantization and adjusted lazy aggregation strategy developed in this paper, the proposed AQUILA has been proven to be capable of reducing the transmitted bits while maintaining the same convergence property and model performance compared against existing methods both theoretically and experimentally.
The evaluation with Non-IID data distribution and various heterogeneous model architectures demonstrates that AQUILA is compatible to existing FL settings.
Future works include further improvements and theoretical guarantee for FL systems with heterogeneity.
[Figure]
A A PPENDIX A. S UPPLEMENTARY E XPERIMENTAL R ESULTS
The appendix includes supplementary experimental results, mathematical proof of Theorem 1 and
Theorem 2, and detailed derivation of the novel adaptive quantization criterion and lazy aggregation strategy.
Compared to Fig.
3 and Fig.
4 in the main text, result figures in appendix show a more comprehensive evaluation with AQUILA, which contain more detailed information including but not limited to accuracy vs steps and training loss vs steps curves
[Table] [Table]
[Figure]
T r a i n i n g L o ss
[Figure]
FedAvg QSGD
LAQ
[Figure]
2 LAQ [Figure] AdaQuantFL AdaQuantFL+LAQ
AQUILA
300 Steps 0.
T r a i n i n g L o ss
Homo-MNIST-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL Adaptive+LAQ
AQUILA
[Figure]
6 Total Transmitted Bits(GB)
A cc u r a c y ( % )
Homo-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
[Figure]
100 Steps
A cc u r a c y ( % )
Homo-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA (
[Figure]
6 Total Transmitted Bits(GB) 0.
T r a i n i n g L o ss
Homo-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
[Figure]
100 Steps 0.
T r a i n i n g L o ss
Homo-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL Adaptive+LAQ
AQUILA
[Figure] [Figure]
i
[Figure]
L
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Total Transmitted Bits     100 Steps
A cc u r a c y ( % ) 100%-50%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA (
[Figure]
Accuracy vs Steps 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Total Transmitted Bits(GB) 0.
T r a i n i n g L o ss 100%-50%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
100 Steps 0.
T r a i n i n g L o ss 100%-50%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL Adaptive+LAQ
AQUILA (
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Transmitted Bits(GB)
A cc u r a c
[Figure]
( % ) 100%-25%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
[Figure]
100 Steps
A cc u r a c y ( % ) 100%-25%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
[Figure]
Accuracy
[Figure]
3.0 Total Transmitted Bits(GB) 0.
T r a i n i n g L o ss 100%-25%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL AdaQuantFL+LAQ
AQUILA
[Figure]
100 Steps 0.
T r a i n i n g L o ss 100%-25%-MNIST-Non-IID Parameters
FedAvg QSGD
LAQ-2 LAQAdaQuantFL Adaptive+LAQ
AQUILA
[Figure] [Figure]
[Figure] [Figure]
100 Steps 5.
L o c a l Q u a n t
i z a t i o n L e v e l Hetero-CIFAR10-IID Communication skipping (a) 1st client in system     100 Steps 5. L o c a l Q u a n t
i z a t i o n L e v e l
Hetero-CIFAR10-IID Communication skipping (b [Figure]
in system     100 Steps 5. L o c a l Q u a n t
i z a t i o n L e v e l Hetero-CIFAR10-IID Communication skipping (
[Figure]
in
[Figure]
A. M ATHEMATICAL P ROOF A.2. P ROOF OF T HEOREM
Without lazy aggregation, the aggregated model at iteration k should be:
[Equation]
With lazy aggregation, the actual aggregated model at iteration k is:
[Equation]
[Equation]
With (13) and (14), the gradient loss caused by skipping can be written as:
[Equation]
2 ) , (15) where the two inequalities follow from: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  ≤ n n (cid:88)
i =
∥ a i ∥ 22 .
Take expectation of both sides of (15):
[Equation]
where the second inequality follows from Assumption 2 with σ 2 denoting the maximum ∥ w ∥ 22 and q b
= d/b 2 , and the last inequality is resulted from the conclusion in LAQ (Sun et al., 2020), which indicates that the client m communicates with the server at most d m rounds in total K iterations, where d m is a constant related to L m .
With the definition of B m and C km = d (cid:6) log 2 ( b km + 1) (cid:7) + d +
[Equation]
the expected gradient loss can be written as:
[Equation]
This completes the proof for Theorem
[Equation]
A.2. D ERIVATION D ETAILS OF A DAPTIVE Q UANTIZATION C RITERION (7)
Let H be the upper bound to be minimized:
[Equation]
The first derivative is:
[Equation]
Let ∂H ∂b k m = 0 , there is:
[Equation]
Dividing ( b km ) ∗ by ( b 1 m ) :
[Equation]
For iteration k = 1 , we set b 1 m for each client m as the intial quantization level b 0 , and thus we get the adaptive quantization criterion (7) for iteration k = 1 , 2 , ..., K :
[Equation]
A.2. P ROOF OF T HEOREM
With the adjusted communication criterion (8)
[Equation]
all transmitted gradient updates are larger than a threshold after quantization.
Therefore, all transmitted gradient updates satisfy:
[Equation]
then,
[Equation]
[Table]
[Equation]
(24) where the second inequality comes from Assumption
[Equation]
Let P = f
( θ 0 )
− f
( θ ∗ )
L (cid:13)(cid:13) θ 1 − θ
(cid:13)(cid:13) 2 2 .
Therefore
[Equation]
if α ≤ (1 /ML ) (cid:112) 2 Pξ/L holds
[Equation]
then in the experiment setting where ξ 1 = ξ
= ... =
ξ D = ξ and
L ≈ L m for all m ∈ M
[Equation]
there is:
[Equation]
A.2. D EVELOPMENT OF L AZY A GGREGATION C RITERION (8)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the second inequality comes from:
[Equation]
Following LAQ’s definition
[Equation]
redefine d m
[Equation]
m ∈ M as:
[Equation]
With the definition of d m and ξ 1 ≥ ξ 2 ≥ ... ≥ ξ D , there is:
[Equation]
[Equation]
With (26) and (31)
[Equation]
we have:
[Equation] [Equation]