AQUILA: C OMMUNICATION E FFICIENT F EDERATED L EARNING WITH A DAPTIVE Q UANTIZATION OF L AZILYA GGREGATED G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
The development and deployment of federated learning (FL) have been bottlenecked by heavy communication overheads of high-dimensional models between the distributed device nodes and the central server.
To achieve better errorcommunication trade-offs, recent efforts have been made to either adaptively reduce the communication frequency by skipping unimportant updates, e.g., lazy aggregation, or adjust the quantization bits for each communication.
In this paper, we propose a unifying communication efficient framework for FL based on a daptive qu ant i zation of l azily- a ggregated gradients ( AQUILA ), which adaptively balances two mutually-dependent factors, the communication frequency, and the quantization level.
Specifically, we start with a careful investigation of the classical lazy aggregation scheme and formulate AQUILA as an optimization problem where the optimal quantization level is selected by minimizing the model deviation caused by update skipping.
Furthermore, we devise a new lazy aggregation strategy to better fit the novel quantization criterion and retain the communication frequency at an appropriate level.
The effectiveness and convergence of the proposed
AQUILA framework are theoretically verified.
The experimental results demonstrate that AQUILA can reduce around 60% of overall transmitted bits compared to existing methods while achieving identical model performance in a number of non-homogeneous FL scenarios, including Non-IID data and heterogeneous model architecture.
I NTRODUCTION
With the deployment of ubiquitous sensing and computing devices, the Internet of things (IoT), as well as many other distributed systems, have gradually grown from concept to reality, bringing dramatic convenience to people’s daily life (Du et al., 2020; Liu et al., 2020; Hard et al., 2018).
To fully utilize such distributed computing resources, distributed learning provides a promising framework that can achieve comparable performance with the traditional centralized learning scheme.
However, the privacy and security of sensitive data during the updating and transmission processes in distributed learning have been a growing concern.
In this context, federated learning (FL) (McMahan et al., 2017) has been developed, allowing distributed devices to collaboratively learn a global model without privacy leakage by keeping private data isolated and masking transmitted information with secure approaches.
On account of its privacy-preserving property and great potentiality in some distributed but privacy-sensitive fields such as finance and health, FL has attracted tremendous attention from both academia and industry in recent years.
Unfortunately, in many FL applications, such as image classification and objective recognition, the trained model tends to be high-dimensional, resulting in significant communication costs.
Hence, communication efficiency has become one of the key bottlenecks of FL.
To this end, Sun et al.
(2020) proposes the lazily-aggregated quantization (LAQ) method to skip unnecessary parameter uploads by estimating the value of gradient innovation — the difference between the current unquantized gradient and the previously quantized gradient.
Moreover, Mao et al.
(2021) devises an adaptive quantized gradient (AQG) strategy based on LAQ to dynamically select the quantization level within some artificially given numbers during the training process.
Nevertheless, the AQG is still not sufficiently adaptive because the pre-determined quantization levels are difficult to choose in complicated FL environments.
In another separate line of work, Jhunjhunwala et al.
(2021) introduces an adaptive quantization rule for FL (AdaQuantFL), which searches in a given range for an optimal quantization level and achieves a better error-communication trade-off.
Most previous research has investigated optimizing communication frequency or adjusting the quantization level in a highly adaptive manner, but not both.
Intuitively, we ask a question, can we adaptively adjust the quantization level in the lazy aggregation fashion to simultaneously reduce transmitted amounts and communication frequency?
In the paper, we intend to select the optimal quantization level for every participated device by optimizing the model deviation caused by skipping quantized gradient updates (i.e., lazy aggregation), which gives us a novel quantization criterion cooperated with a new proposed lazy aggregation strategy to reduce overall communication costs further while still offering a convergence guarantee.
The contributions of this paper are trifold.
• We propose an innovative FL procedure with a daptive qu ant i zation of l azily- a ggregated gradients termed AQUILA, which simultaneously adjusts the communication frequency and quantization level in a synergistic fashion.
• Instead of naively combining LAQ and AdaQuantFL, AQUILA owns a completely different device selection method and quantitative level calculation method.
Specifically, we derive an adaptive quantization strategy from a new perspective that minimizes the model deviation introduced by lazy aggregation.
Subsequently, we present a new lazy aggregation criterion that is more precise and saves more device storage.
Furthermore, we provide a convergence analysis of AQUILA under the generally non-convex case and the Polyak-Łojasiewicz condition.
• Except for normal FL settings, such as independent and identically distributed (IID) data environment, we experimentally evaluate the performance of AQUILA in a number of non-homogeneous FL settings, such as non-independent and non-identically distributed (Non-IID) local dataset and various heterogeneous model aggregations.
The evaluation results reveal that AQUILA considerably mitigates the communication overhead compared to a variety of state-of-art algorithms.
B ACKGROUND AND R ELATED W ORK
Consider an FL system with one central parameter server and a device set M with M = |M| distributed devices to collaboratively train a global model parameterized by θ ∈ R d .
Each device m ∈ M has a private local dataset D m = { ( x ( m ) 1 , y ( m ) 1 ) , · · · , ( x ( m ) n m , y ( m ) n m ) } of n m samples.
The federated training process is typically performed by solving the following optimization problem
[Equation]
with
[Equation]
where f : R d → R denotes the empirical risk, f m : R d
→ R denotes the local objective based on the private data D m of the device m , l denotes the local loss function, and h θ denotes the local model.
The FL training process is conducted by iteratively performing local updates and global aggregation as proposed in (McMahan et al., 2017).
First, at communication round k , each device m receives the global model θ k from the parameter server and trains it with its local data D m .
Subsequently, it sends the local gradient ∇ f m ( θ k ) to the central server, and the server will update the global model with learning rate α by
[Equation]
Definition 2.1 (Quantized gradient innovation).
For more efficiency, each device only uploads the quantized deflection between the full gradient ∇ f m ( θ k ) and the last quantization value q k − 1 m utilizing a quantization operator Q : R d → R d , i.e.,
[Equation]
For communication frequency reduction, the lazy aggregation strategy allows the device m ∈ M to upload its newly-quantized gradient innovation at epoch k only when the change in local gradient is sufficiently larger than a threshold.
Hence, the quantization of the local gradient q km of device m at epoch k can be calculated by
[Equation]
If the device m skips the upload of ∆ q km , the central server will reuse the last gradient q k − 1 m for aggregation.
Therefore, the global aggregation rule can be changed from (2) to:
[Equation]
where M k denotes the subset of devices that upload their quantized gradient innovation, and M kc =
M \ M k denotes the subset of devices that skip the gradient update and reuse the old quantized gradient at epoch k .
For AdaQuantFL, it is proposed to achieve a better error-communication trade-off by adaptively adjusting the quantization levels during the FL training process.
Specifically, AdaQuantFL computes the optimal quantization level ( b k ) ∗ by ( b k ) ∗ =
⌊ (cid:113) f
( θ 0 ) /f
( θ k ) · b 0 ⌋ , where f ( θ 0 ) and f ( θ k ) are the global objective loss defined in (1).
However, AdaQuantFL transmits quantized gradients at every communication round .
In order to skip unnecessary communication rounds and adaptively adjust the quantization level for each communication jointly, a naive approach is to quantize lazily aggregated gradients with AdaQuantFL.
Nevertheless, it fails to achieve efficient communication for several reasons.
First, given the descending trend of training loss, AdaQuantFL’s criterion may lead to a high quantization bit number even exceeding 32 bits in the training process (assuming a floating point is represented by 32 bits in our case), which is too large for cases where the global convergence is already approaching and makes the quantization meaningless.
Second, a higher quantization level results in a smaller quantization error, leading to a lower communication threshold in the lazy aggregation criterion (4) and thus a higher transmission frequency.
Consequently, it is desirable to develop a more efficient adaptive quantization method in the lazilyaggregated setting to improve communication efficiency in FL systematically.
A DAPTIVE Q UANTIZATION OF L AZILY -A GGREGATED G RADIENTS
Given the above limitations of the naive joint use of the existing adaptive quantization criterion and lazy aggregation strategy, this paper aims to design a unifying procedure for communication efficiency optimization where the quantization level and communication frequency are considered synergistically and interactively.
3. O PTIMAL QUANTIZATION LEVEL
First, we introduce the definition of a deterministic rounding quantizer and a fully-aggregated model.
Definition 3.1.
(Deterministic mid-tread quantizer.)
Every element of the gradient innovation of device m at epoch k is mapped to an integer [ ψ km ] i as
[Equation]
[Equation]
where ∇ f ( θ km ) denotes the current unquantized gradient, R km = ∥∇ f m ( θ k ) − q k − 1 m ∥ ∞ denotes the quantization range, b k m denotes the quantization level, and τ k m := 1 / (2 b km − 1) denotes the quantization granularity.
Definition 3.2 (Fully-aggregated model).
The fully-aggregated model ˜ θ without lazy aggregation at epoch k is computed by
[Equation]
Lemma 3.1.
The influence of lazy aggregation at communication round k can be bounded by
[Equation]
Corresponding to Lemma 3.
[Equation]
since R km is independent of τ km
[Equation]
we can formulate an optimization problem to minimize the upper bound of this model deviation caused by update skipping for each device m :
[Equation]
to
[Equation]
Solving the below optimization problem gives AQUILA an adaptive strategy (10) that selects the optimal quantization level based on the quantization range R km , the dimension d of the local model, the current gradient ∇ f m ( θ k )
[Equation]
and the last uploaded quantized gradient q k − 1 m :
[Equation]
The superiority of (10) comes from the following three aspects.
First, since R km ⩾ [ ∇ f m ( θ k )] i − [ q k − 1 m ]
i ⩾ − R km , the optimal quantization level ( b km ) ∗ must be greater than or equal to 1 .
Second,
AQUILA can personalize an optimal quantization level for each device corresponding to its own gradient, whereas, in AdaQuantFL, each device merely utilizes an identical quantization level according to the global loss.
Third, the gradient innovation and quantization range R km tend to fluctuate along with the training process instead of keeping descending, and thus prevent the quantization level from increasing tremendously compared with AdaQuantFL.
3. P RECISE LAZY AGGREGATION CRITERION
Definition 3.3 (Quantization error).
The global quantization error ε k is defined by the subtraction between the current unquantized gradient ∇ f ( θ k ) and its quantized value q k − 1 + ∆ q k , i.e.,
[Equation]
where ∇ f ( θ k ) =
(cid:80) m ∈M ∇ f m ( θ k ) , q k − 1 = (cid:80) m ∈M q k − 1 m , ∆ q k = (cid:80) m ∈M ∆ q km .
To better fit the larger quantization errors induced by fewer quantization bits in (10), AQUILA possesses a new communication criterion to avoid the potential expansion of the devices group being skipped:
[Equation] [Equation]
where β ⩾ 0 is a tuning factor.
Furthermore, we provide an intuitive method to choose the value of β in Appendix G. Note that this skipping rule is employed at epoch k , in which each device m calculates its quantized gradient innovation ∆ q km and quantization error ε km , then utilizes this rule to decide whether uploads ∆ q k m .
Instead of storing a large number of previous model parameters as LAQ, the strength of (12) is that
AQUILA directly utilizes the global model for two adjacent rounds as the skip condition, which does not need to estimate the global gradient (more precise), requires fewer hyperparameters to adjust, and considerably reduces the storage pressure of local devices.
This is especially important for small-capacity devices (e.g., sensors) in practical IoT scenarios.
q
[Equation]
k
=
[Equation]
k
[Equation]
(cid:0)
[Equation]
+ 1 /M (cid:80) m ∈M
∆ q
The detailed process of AQUILA is comprehensively summarized in Algorithm 1.
At epoch k = 0 , each device calculates b 0 m by setting q k − 1= 0 and uploads ∆ q k 0 to the server since the (12) is not satisfied.
At epoch k ∈ { 1 , 2 , ..., K } , the server first broadcasts the global model θ k to all devices.
Each device m computes ∇ f ( θ km ) with local training data and then utilizes it to calculate an optimal quantization level by (10).
Subsequently, each device computes its gradient innovation after quantization and determines whether or not to upload based on the communication criterion (12).
Finally, the server updates the new global model θ k +1 with up-to-date quantized gradients q k − 1 m + ∆ q km for those devices who transmit the uploads at epoch k , while reusing the old quantized gradients q k − 1 m for those who skip the uploads.
T HEORETICAL D ERIVATION AND A NALYSIS OF AQUILA
As aforementioned, we bound the model deviation caused by skipping updates with respect to quantization bits.
Specifically, if the communication criterion (12) holds for the device m at epoch k , it does not contribute to epoch k ’s gradient.
Otherwise, the loss caused by device m will be minimized with the optimal quantization level selection criterion (10).
In this section, the theoretical convergence derivation of AQUILA is based on the following standard assumptions.
Assumption 4.1 (L-smoothness).
Each local objective function f m is L m -smooth, i.e., there exist a constant L m > 0 , such that ∀ x , y ∈ R d ,
[Equation]
(13) which implies that the global objective function f is L -smooth with L ≤ ¯ L = 1 m (cid:80) mi =1 L m .
Assumption 4.2 (Uniform lower bound).
For all x ∈ R d , there exist f ∗ ∈ R such that f ( x ) ≥ f ∗ .
Lemma 4.1.
Following the assumption that the function f is L-smooth, we have f ( θ k +1 )
− f ( θ k ) ⩽ − α 2 (cid:13)(cid:13)(cid:13) ∇ f ( θ k ) (cid:13)(cid:13)(cid:13) 2 2 + α   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 M (cid:88) m ∈M kc
[Equation]
q km (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
4. C ONVERGENCE ANALYSIS FOR G ENERALLY N ON - CONVEX C ASE .
Theorem 4.1.
Suppose Assumptions 4.1, 4.2, and B.1 (29) be satisfied.
If M kc ̸ = ∅ , the global objective function f satisfies
[Equation]
Corollary 4.1.
Let all the assumptions of Theorem 4.1 hold and L 2 − 12 α + βγα
⩽
, then the AQUILA requires
[Equation]
communication rounds with ω 1 = f (cid:0) θ 1 (cid:1) − f ( θ ∗ )+ βγα (cid:13)(cid:13)
θ 1 − θ
(cid:13)(cid:13) 2 2 to achieve min k ∥∇ f
( θ k ) ∥
22 ⩽ ϵ 2 .
Compared to LAG.
Corresponding to Eq.(70) in Chen et al.
(2018), LAG defines a Lyapunov function V k :=
f ( θ k ) − f ( θ ∗ )
+ (cid:80) Dd =1 β d ∥
θ k +1 − d − θ k − d ∥ 22 and claims that it satisfies
[Equation]
where ˜ c ( α, β 1 ) = L/ 2 − 1 / (2 α ) + β 1 , β 1 =
Dξ/ (2 αη ) , ξ < 1 /D , and ρ > 0 .
The above result (17) indicates that LAG requires
[Equation]
communication rounds to converge.
Since the non-negativity of the term ˜ c ( α, β 1 ) (1 + ρ ) α 2 , we can readily derive that α < α − 2˜ c ( α, β 1 ) (1 + ρ ) α 2 , which demonstrates AQUILA achieves a better convergence rate than LAG with the appropriate selection of α .
4. C ONVERGENCE ANALYSIS U NDER P OLYAK -Ł OJASIEWICZ CONDITION .
Assumption 4.3 ( µ − PŁ condition).
Function f satisfies the PL condition with a constant µ > 0 , that is,
[Equation]
Theorem 4.2.
Suppose Assumptions 4.1, 4.2, and 4.3 be satisfied and M kc ̸ = ∅ , if the hyperparameters satisfy βγα ⩽ (1 − αµ ) (cid:0) 12 α − L 2 (cid:1) , then the global objective function satisfies
[Equation]
−
[Equation]
−
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
20) and the AQUILA requires
[Equation] [Equation]
communication round with ω 1 = f
( θ 1 ) − f ( θ ∗ ) + (cid:0) 12 α − L 2 (cid:1) (cid:13)(cid:13) θ 1 − θ
(cid:13)(cid:13) 2 2 to achieve f ( θ K +1 )
− f ( θ ∗ )
+ ( 12 α − L 2 )
∥
θ K +1 − θ K ∥
22 ⩽ ϵ .
Compared to LAG.
According to Eq.(50) in Chen et al.
(2018), we have that
[Equation]
where ξ < 1 /D .
Thus, we have that LAG requires
[Equation]
communication rounds to converge.
Compared to Theorem 4.2, we can derive that log(1 − αµ ) <
log(1 − αµ + αµ √ Dξ ) , which indicates that AQUILA has a faster convergence than LAG under the
PŁ condition.
Remark.
We want to emphasize that LAQ introduces the Lyapunov function into its proof, making it extremely complicated.
In addition, LAQ can only guarantee that the final objective function converges to a range of the optimal solution rather than an accurate optimum f ( θ ∗ ) .
Nevertheless, as discussed in Section 3.2, we utilize the precise model difference in AQUILA as a surrogate for the global gradient and thus simplify the proof.
E XPERIMENTS AND D ISCUSSION 5. E XPERIMENT S ETUP
In this paper, we evaluate AQUILA on CIFAR-10 , CIFAR-100 (Krizhevsky et al., 2009), and
WikiText-2 dataset (Merity et al., 2016), considering IID, Non-IID data scenario, and heterogeneous model architecture (which is also a crucial challenge in FL) simultaneously.
The FL environment is simulated in Python 3.9 with PyTorch 11.1 (Paszke et al., 2019) implementation.
For the diversity of the neural network structures, we train ResNet-18 (He et al., 2016) at CIFAR-10 dataset, MobileNet-v2 (Sandler et al., 2018) at CIFAR-100 dataset, and
Transformer (Vaswani et al., 2017) at WikiText-2 dataset.
As for the FL system setting, in the majority of our experiments, the whole system exists M = 10 total devices.
However, considering the large-scale feature of FL, we also validate AQUILA on a larger system of M = 100 / 80 total devices for CIFAR / WikiText-2 dataset.
The hyperparameters and additional details of our experiments are revealed in Appendix A.3.
[Figure]
epoch.
5. H OMOGENEOUS E NVIRONMENT
We first evaluate AQUILA with homogeneous settings where all local models share the same model architecture as the global model.
To better demonstrate the effectiveness of AQUILA, its performance is compared with several state-of-the-art methods, including AdaQuantFL, LAQ with fixed levels,
LENA (Ghadikolaei et al., 2021), MARINA (Gorbunov et al., 2021), and the naive combination of
AdaQuantFL with LAQ.
Note that based on this homogeneous setting, we conduct both IID and NonIID evaluations on CIFAR-10 and CIFAR-100 dataset, and an IID evaluation on WikiText-2 .
To simulate the Non-IID FL setting as (Diao et al., 2020), each device is allocated two classes of data in CIFAR-10 and 10 classes of data in CIFAR-100 at most, and the amount of data for each label is balanced.
The experimental results are presented in Fig.
1, where 100% implies all local models share a similar structure with the global model (i.e., homogeneity), 100% (80) denotes the experiment is conducted in an 80 devices system, and LAdaQ represents the naive combination of AdaQuantFL and LAQ.
For better illustration, the results have been smoothed by their standard deviation.
The solid lines represent values after smoothing, and transparent shades of the same colors around them represent the true values.
Additionally, Table 2 shows the total number of bits transmitted by all devices throughout the FL training process.
The comprehensive experimental results are established in Appendix A.4.
5. N ON - HOMOGENEOUS S CENARIO
In this section, we also evaluate AQUILA with heterogeneous model structures as HeteroFL (Diao et al., 2020), where the structures of local models trained on the device side are heterogeneous.
Suppose the global model at epoch k is θ k and its size is d = w g ∗ h g , then the local model of each device m can be selected by θ km = θ
k
[: w m , : h m ] , where w m = r m w g and h m =
r m h g , respectively.
In this paper, we choose model complexity levels r m = 0 .
[Figure]
Most of the symbols in Fig.
2 are identical to the Fig.
100%-50% is a newly introduced symbol that implies half of the devices share the same structure with the global model while another half only have 50% * 50% parameters as the global model.
Performance Analysis.
First of all, AQUILA achieves a significant transmission reduction compared to the naive combination of LAQ and AdaQuantFL in all datasets, which demonstrates the superiority of AQUILA’s efficiency.
Specifically, Table 2 indicates that AQUILA saves 57.49% of transmitted bits in the system of 80 devices at the WikiText-2 dataset and reduces 23.08% of transmitted bits in the system of 100 devices at the CIFAR-100 dataset, compared to the naive combination.
And other results in Table 3 also show an obvious reduction in terms of the total transmitted bits required for convergence.
Second, in Fig.
1 and Fig.
2, the changing trend of AQUILA’s communication bits per each round clearly verifies the necessity and effectiveness of our well-designed adaptive quantization level and skip criterion.
In these two figures, the number of bits transmitted in each round of AQUILA fluctuates a bit, indicating the effectiveness of AQUILA’s selection rule.
Meanwhile, the value of transmitted bits remains at quite a low level, suggesting that the adaptive quantization principle makes training more efficient.
Moreover, the figures also inform that the quantization level selected by
AQUILA will not continuously increase during training instead of being as AdaQuantFL.
In addition, based on these two figures, we can also conclude that AQUILA converges faster under the same communication costs.
Finally, AQUILA is capable of adapting to a wide range of challenging FL circumstances.
In the
Non-IID scenario and heterogeneous model structure, AQUILA still outperforms other algorithms by significantly reducing overall transmitted bits while maintaining the same convergence property and objective function value.
In particular, AQUILA reduces 60.4% overall communication costs compared to LENA and 57.2% compared to MARINA on average.
These experimental results in non-homogeneous FL settings prove that AQUILA can be stably employed in more general and complicated FL scenarios.
5. A BLATION STUDY ON THE IMPACT OF TUNING FACTOR β
One key contribution of AQUILA is presenting a new lazy aggregation criterion (12) to reduce communication frequency.
In this part, we evaluate the effects of the tuning factor β in Fig.
As β grows within a certain range, the convergence speed of the model will slow down (due to lazy aggregation).
Still, it will eventually converge to the same model performance while considerably reducing the communication overhead.
Nevertheless, increasing the value of β will lead to a decrease in the final model performance since it skips so many essential uploads that make the training deficient.
To sum up, we should choose the value of factor β to maintain the model’s performance and minimize the total transmitted amount of bits.
Specifically, we select the value of β = 0 .
25 , 1 . 25 for
CIFAR-10 , CIFAR-100 , and WikiText-2 datasets, respectively.
[Figure]
C ONCLUSIONS AND F UTURE W ORK
This paper proposes a communication-efficient FL procedure to simultaneously adjust two mutuallydependent degrees of freedom: communication frequency and quantization level.
With the close cooperation of the novel adaptive quantization and adjusted lazy aggregation strategy derived in this paper, the proposed AQUILA has been proven to be capable of reducing the transmitted costs while maintaining the convergence guarantee and model performance compared to existing methods.
The evaluation with Non-IID data distribution and various heterogeneous model architectures demonstrates that AQUILA is compatible in a non-homogeneous FL environment.
R EPRODUCIBILITY
We present the overall theorem statements and proofs for our main results in the Appendix, as well as necessary experimental plotting figures.
Furthermore, we submit the code of AQUILA in the supplementary material part, including all the hyperparameters and a requirements to help the public reproduce our experimental results.
Our algorithm is straightforward, well-described, and easy to implement.
E THICS S TATEMENT
All evaluations of AQUILA are performed on publicly available datasets for reproducibility purposes.
This paper empirically studies the performance of various state-of-art algorithms, therefore, probably introduces no new ethical or cultural problems.
This paper does not utilize any new dataset.
A A PPENDIX
The appendix includes supplementary experimental results, mathematical proof of the aforementioned theorems
[Figure]
and a detailed derivation of the novel adaptive quantization criterion and lazy aggregation strategy.
Compared to Fig.
1 and Fig.
2 in the main text, the result figures in the appendix show a more comprehensive evaluation with AQUILA, which contains more detailed information including but not limited to accuracy vs steps and training loss vs steps curves.
A. O VERALL F RAMEWORK OF AQUILA
The cooperation of the novel adaptive quantization criterion (10) and lazy aggregation strategy (12) is illustrated in Fig. 4a.
Compared to the naive combination of AdaQuantFL and LAQ, where the mutual influence between adaptive quantization and lazy aggregation has not been considered, as shown in Fig.
4b, AQUILA adaptively optimizes the allocation of quantization bits throughout training to promote the convergence of lazy aggregation
[Figure]
and at the same time utilizes the lazy aggregation strategy to improve the efficiency of adaptive quantization by compress the transmission with a lower quantization level
[Figure]
gradient innovation, considering the heterogeneous FL environment as in our evaluation part.
For instance, the gradient innovation ∆ q k 3 of a PC is larger than ∆ q k 2 of a mobile phone.
A. D IFFERENCE OF OUR SKIP RULE AND LAQ’ S
The skip rule of LAQ is measured by the summation of the accumulated model difference and quantization error:
[Equation]
where ξ d ′ is a series of manually selected scalars and D is also predetermined.
ε km is the quantization error of client m at epoch k, and ˆ ε k − 1 m is the quantization of client m at last time it uploads its gradient innovation.
Please refer to Sun et al.
(2020) for more details on (24).
In order to compute the LAQ skip threshold, each client has to store much previous information.
A. E XPERIMENT S ETUP
In this section, we provide some extra hyperparameter settings for our evaluation.
For the LAQ, we set D = 10 and ξ 1 = ξ 2 = · · · = ξ D = 0 .
8 /D as the same as the setting in their paper.
For
LENA, we set β LENA = 40 in their trigger condition.
And for MARINA, we calculate the uploading probability of Bernoulli distribution as p = ξ Q /d as announced in their paper.
In addition, we choose the CrossEntropy function as our objective function in the experiment part.
Table 1 shows the hyperparameter details of our evaluation.
[Table]
A. C OMPREHENSIVE EXPERIMENT RESULTS
This section will cover all the experimental results in our paper.
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
QSGDLAQLAdaQ
AdaQLENA
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
i t
[Figure] [Figure]
[Figure]
B B ASIC F ACTS AND SOME LEMMAS
Notations: Bold fonts denote vectors (e.g., θ ).
Normal fonts denote scalars (e.g., α ).
Subscript m is used to describe functions about a local device m (e.g., f m ( θ ) ).
A function without a subscript is used to describe an average among all devices (e.g., f ( θ ) ).
Frequently used norm inequalities Suppose n ∈ N + and ∥ · ∥ 2 denotes the ℓ 2 − norm.
For p in R + , x i , a , b ∈ R d , there holds
Norm summation inequality.
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i = x i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  ⩽ n n (cid:88) i =
∥ x i ∥ 22 .
Inner-product inequality.
[Equation]
Young’s Inequality.
[Equation]
Minkowski’s Inequality.
[Equation]
Assumption B.1.
All devices’ quantization errors ε k will be constrained by the total error of the omitted devices.
, i.e., ∀ k = 0 , 1 , · · · , K , if M kc ̸ = ∅ , ∃ γ ⩾ 1 , such that (cid:13) (cid:13)
[Equation]
(cid:13) (cid:13) 2
=
(cid:13) (cid:13) (cid:13)(cid:13)(cid:13)
M (cid:88) m ∈M ε km (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)  ⩽ γ
M(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (cid:88) m ∈M kc ε k m (cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13)  , (29) where K denotes the termination time, and ε km = ∇ f m ( θ k ) − (cid:0) q k − 1 m + ∆ q km (cid:1) .
This lemma is easy to verify when
M kc ̸ = ∅ , a bounded variable (here is ε k ) will always be bounded by a part of itself ( 1 M (cid:80) m ∈M kc ε km ) multiplied by a real number ( γ ).
Note that there is another nontrivial scenario that M kc ̸ = ∅ but ε km = 0 for all m ∈ M kc , which implies that γ = 0 or not exists and conflicts with our assumption.
However, this situation only happens when all entries of ε km = 0 , i.e., [ ∇ f m ( θ k )]
i =
[ q k − 1 m ] i for all 0 ⩽
i ⩽ d .
Lemma B.1 .
The summation of quantized gradient innovation and quantization error is bounded by the global model difference:
[Equation]
Proof.
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (a) follows Assumption B.1, (b) follows γ is larger than 1 by definition, and (c) utilizes our novel trigger condition (12).
Lemma B.2 .
From Definition 3.1, we can derive that the relationship between quantized gradient innovation ∆ q km and its quantization representation ψ km which applies b km bits for each dimension:
[Equation]
where 1 ∈ R d denotes a vector filled with scalar value 1 .
Remark: We can utilize (32) to calculate the quantized gradient innovation in the experimental implementation.
C M ISSING P ROOF OF L EMMA 3.1 AND THE D ERIVATION OF b km
With lazy aggregation, the actual aggregated model at epoch k is:
[Equation]
Suppose ∆ km denotes the quantization loss of device m at epoch k and ψ km denotes the quantization representation of local gradient innovation as in Definition 3.1, i.e.,
[Equation]
With (7), (33)
[Equation]
and (34)
[Equation]
the model deviation ∥
˜ θ k − θ k ∥ 22 caused by skipping gradients can be written as:
[Equation]
[Table]
[Equation] [Equation]
where 1 ∈ R d denotes the vector filled with scalar value 1 , (a) ∆ km ∈ ( − 1 , 0] , (b) R km ⩾ τ km R km ⩾ 0 .
Since R km is independent of τ km
[Equation]
we can formulate an optimization problem about τ km for device m at communication round k as follows:
[Equation]
Therefore, the optimal solution of τ km in (36) is
[Equation]
Then, the optimal adaptive quantization level ( b km ) ∗ is equal to
[Equation]
Notice that ( b km ) ∗ ⩾ 1 is always true since ( τ km ) ∗ ⩽
D M ISSING P ROOF OF L EMMA 4.1, T HEOREM 4.1 AND C OROLLARY 4.1.
Proof.
Suppose Assumptions 4.1, 4.2, and 4.3 are satisfied and M kc ̸ = ∅ .
For the simplicity of the convergence proof
[Equation]
we assume Φ k = 1 M (cid:80) m ∈M kc ∆ q km .
First
[Equation]
we prove Lemma 4.
[Equation]
[Table]
Hence, we have
[Table]
− θ
[Equation]
40) which gives us Theorem 4.1.
Sum it up for k = 1 , 2 , · · · , K , we have
[Equation]
Notice that inequality (41) holds for both M kc ̸ = ∅ and M kc = ∅ .
Therefore, for (cid:16) L 2 − 12 α + βγα (cid:17) ⩽ 0 and all hyperparameters are chosen properly, considering the minimum of ∥∇ f ( θ k ) ∥
[Equation]
For (cid:16) L 2 − 12 α + βγα (cid:17) ⩽ 0 and all hyperparameters are chosen properly, we have that
[Equation]
which demonstrates AQUILA requires K
= O
[Equation]
communication round with ω 1 = f ( θ 1 ) − f ( θ ∗ ) + βγα (cid:13)(cid:13)
θ 1 − θ
(cid:13)(cid:13) 2 2 to achieve min k =1 , 2 ,
[Equation]
E M ISSING P ROOF OF C OROLLARY 4.1 WHEN M kc = ∅ .
Proof.
Since the skipping subset of devices are the empty set, from (5), we have
[Equation]
From (14) we have:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
=
[Equation]
If the factor of (cid:13)(cid:13) ε k (cid:13)(cid:13) 2 2 in (45) is less than or equal to 0 , that is,
[Equation]
46) then the factor of (cid:13)(cid:13)(cid:13) ∇ f (cid:16) θ k (cid:17)(cid:13)(cid:13)(cid:13) 2 2 will be less than − α 2 , which indicates that
[Equation]
Note that it is not difficult to demonstrate that (46) and L 2 − 12 α + βγα
⩽ 0 can actually be satisfied at the same time.
For instance, we can set p = 0 .
1 , α = 0 .
1 , β = 0 .
25 , γ = 2 , L = 2 . 5 that satisfies both of them.
F M ISSING P ROOF OF T HEOREM 4.2.
Proof.
Based on the intermediate result (40) of Theorem 4.1 and Assumption 4.4 ( µ − PŁ condition), we have
[Equation]
⩽ − α 2 (cid:13)(cid:13)(cid:13) ∇ f (cid:16) θ k (cid:17)(cid:13)(cid:13)(cid:13) 2 2 + (cid:18)
L 2 − 1 2 α (cid:19) (cid:13)(cid:13)(cid:13) θ k +1 − θ k (cid:13)(cid:13)(cid:13) 2 2 + βγ α (cid:13)(cid:13)(cid:13) θ k − θ k − 1 (cid:13)(cid:13)(cid:13) 2(19)
⩽ − αµ ( f ( θ k ) − f
( θ ∗ ))
+ (cid:18) L 2 − 1 2 α (cid:19) (cid:13)(cid:13) (cid:13) θ
k +1 − θ k
(cid:13)(cid:13) (cid:13) 2
+ βγ α (cid:13)(cid:13) (cid:13) θ k − θ k − 1 (cid:13)(cid:13)
(cid:13) 2 2 , (48) which is equivalent to f (cid:16) θ
[Equation]
− f ( θ ∗ ) (19) ⩽ (1 − αµ )
( f ( θ k ) − f
( θ ∗ ))
+ (cid:18) L 2 − 1 2 α (cid:19) (cid:13)(cid:13)(cid:13) θ k +1 − θ k (cid:13)(cid:13)(cid:13) 2
+ βγ α (cid:13)(cid:13)(cid:13) θ k − θ k − 1 (cid:13)(cid:13)(cid:13) 2 2 .
Suppose βγα ⩽ (1 − αµ ) (cid:0) 12 α − L 2 (cid:1) , we can show that
[Equation]
Therefore, after multiply k = 1 , 2 , · · · , K , we have
[Equation]
which demonstrates AQUILA requires K
= O (cid:16) −
[Equation] [Equation]
) log ω 1 ϵ (cid:17) communication round with ω= f (cid:0)
[Equation] [Equation]
+
[Equation]
12 α − L
[Equation] [Equation]
θ
[Equation]
2 2 to achieve f (cid:16) θ K +
[Equation]
G A N INTUITIVE METHOD TO CHOOSE β .
From the vanilla gradient descent update, we know it yields the following descent:
[Equation]
(52) where ∆ kGD = − (1 − αL 2 )
α (cid:13)(cid:13)(cid:13) ∇ f (cid:16) θ k (cid:17)(cid:13)(cid:13)(cid:13) 2 2 , α denotes the step size, and L > 0 denotes the Lipschitz constant.
At the same time, we denote ∆ kAQ by − α 2 (cid:13)(cid:13)(cid:13) ∇ f (cid:16) θ k (cid:17)(cid:13)(cid:13)(cid:13) 2 2 + (cid:0) L 2 − 12 α (cid:1) (cid:13)(cid:13)(cid:13) θ k +1 − θ k (cid:13)(cid:13)(cid:13) 2
+ α (cid:13)(cid:13)
ε k (cid:13)(cid:13) 2
+ α (cid:13)(cid:13)(cid:13) 1 M (cid:80) m ∈M kc ∆ q km (cid:13)(cid:13)(cid:13) 2 2 according to (14).
For simplicity, we temporarily set α = 1 /L .
If we want AQUILA to be more efficient than the vanilla gradient descent, then
[Equation]
(53) that is,
[Equation]
[Table]
[Equation]
Thus
[Equation]
for each device m at communication round k
[Equation]
if it satisfies
[Equation]
then (54) will also be satisfied, that is, β ⩽ 0 .
to (12)
In paper we β =
while evaluating the of
[Figure]