P ER F ED M ASK : P ERSONALIZED F EDERATED L EARNING WITH O PTIMIZED M ASKING V ECTORS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, various personalized federated learning (FL) algorithms have been proposed to tackle data heterogeneity.
To mitigate device heterogeneity, a common approach is to use masking.
In this paper, we first show that using random masking can lead to a bias in the obtained solution of the learning model.
To this end, we propose a personalized FL algorithm with optimized masking vectors called
PerFedMask.
In particular, PerFedMask facilitates each device to obtain its optimized masking vector based on its computational capability before training.
Finetuning is performed after training.
PerFedMask is a generalization of a recently proposed personalized FL algorithm, FedBABU
(Oh et al., 2022).
PerFedMask can be combined with other FL algorithms including HeteroFL (Diao et al., 2021) and Split-Mix FL (Hong et al., 2022).
Results based on CIFAR-10 and CIFAR100 datasets show that the proposed PerFedMask algorithm provides a higher test accuracy after fine-tuning and lower average number of trainable parameters when compared with six existing state-of-the-art FL algorithms in the literature.
I NTRODUCTION
Federated learning (FL) is a distributed artificial intelligence (AI) framework, which allows multiple edge devices to train a single model collaboratively (Kone ˇ cn ` y et al., 2015; McMahan et al., 2017).
The model is trained under the orchestration of a central server.
In a typical FL algorithm, each communication round includes the following steps: (1) the edge devices download the latest model from the server to be used as their local model; (2) each device performs multiple local update iterations for updating the local model based on its local dataset; (3) the devices upload their updated local models to the server; (4) the server computes the new model by aggregating the local models.
In practical systems, the devices may have diverse and limited computation, communication, and storage capabilities.
Moreover, the local datasets available to the devices may be different in size, and contain non-independent and identically distributed (non-IID) data samples across the devices.
Under these heterogeneous settings, the performance of the conventional FL algorithms can degrade (Wang et al., 2020; Li et al., 2021).
Most of the recent works on FL attempt to tackle data and device heterogeneity issues in FL separately.
To handle the case when the data is non-IID, some works (Li et al., 2020a; Karimireddy et al., 2020) have introduced new optimization frameworks to obtain a more stable global model for the devices.
Another approach to address the data heterogeneity issue is by designing a personalized model for each device (Arivazhagan et al., 2019;
Fallah et al., 2020;
Collins et al., 2021;
Oh et al., 2022).
In personalized FL algorithms, instead of obtaining a single model for all the devices, an initial model is obtained.
This initial model can then be personalized for each device based on its local data samples.
To overcome the computation limitation of the heterogeneous devices, one common approach is to use masking vectors (Chen et al., 2021;
Babakniya et al., 2022; Bibikar et al., 2022).
Masking vectors can be used to train only a sub-network of the learning model for each device based on the computational capability of that device.
Masking vectors can be combined with pruning and freezing methods.
Pruning methods utilize masking vectors to keep the important parameters of the learning model and remove those which are unimportant from the model architecture.
However, leveraging pruning in FL incurs additional communication overhead between the central server and devices (Babakniya et al., 2022; Bibikar et al., 2022).
Moreover, pruning methods may lead to different model architectures for the devices in the local update iterations (Guo et al., 2016).
Hence, the available local data samples at each device may not be used effectively for training of all the model architectures (Hong et al., 2022).
This may lead to accuracy loss, especially when there is data heterogeneity in the system.
In the freezing methods, the masking vectors are used to freeze some parts of the learning model for each device.
Unlike pruning, the masked parameters are not removed but are frozen during local updates.
Hence, a more stable FL algorithm is obtained without changing the learning model architecture.
Sidahmed et al.
(2021) and Pfeiffer et al.
(2022) have shown that freezing methods can reduce the computational and communication resources required for training the learning model in FL.
However, the aforementioned works have considered random masking, which is a simple approach for designing the masking vectors, and do not provide a theoretical analysis for their choice.
Also, the aforementioned works do not address the data heterogeneity issue in their proposed algorithms.
We aim to answer the following question: By exploiting freezing method in FL, what is a systematic approach to determine the masking vectors which can improve the final test accuracy in a setting with data and device heterogeneities?
In this work, we first show that using the masking vectors to freeze the model parameters for the devices may lead to a bias in the convergence bound.
This bias can hinder the success of employing masking vectors to tackle the device heterogeneity issue in FL.
Using the insights from our analysis, we propose PerFedMask, Per sonalized Fed erated Learning with Optimized Mask ing Vectors (see
Fig. 1).
Specifically, by decoupling the learning model into a global model and a local head model, we first freeze the local head model for all the devices.
Then, we freeze a portion of the global model for each device based on its computational capability.
In our work, the masking vectors are determined before training through minimizing the bias term in the convergence bound.
Although the bias can be mitigated by properly selecting the masking vectors, it may not be eliminated completely.
Thus, after training of the global model, the frozen parameters of the local head model can assist to fine-tune the entire learning model for each device by using the available local data samples.
We demonstrate empirically the effectiveness of PerFedMask under the heterogeneous settings when compared with six existing state-of-the-art FL algorithms.
PerFedMask has several distinct advantages: (1) PerFedMask generalizes the recently proposed personalized FL algorithm, FedBABU
(Oh et al., 2022).
In particular, FedBABU is a special case of PerFedMask when all the devices have the same computational capability.
(2) PerFedMask is flexible.
Since PerFedMask does not change the model architecture, it can be combined with other
FL algorithms such as HeteroFL (Diao et al., 2021) and Split-Mix FL (Hong et al., 2022) to further improve the performance.
(3) PerFedMask can address the objective inconsistency problem, which arises due to different number of local update iterations.
Unlike other FL algorithms (e.g., FedNova (Wang et al., 2020)) which need to modify the device optimizers to tackle this issue, in PerFedMask, we consider the same number of local update iterations for all the devices, while adjusting the required number of computations for those devices with lower computational capabilities.
RELATED WORK
FL Algorithms with non-IID Data.
FedAvg (McMahan et al., 2017) is the most popular FL algorithm, which aims to find a single model for all the devices.
However, the local data samples at the devices are usually non-IID, which can lead to data heterogeneity among the devices.
To address this problem, Li et al.
(2020a) proposed FedProx by introducing a proximal term in the objective function.
However, this method does not attempt to find a personalized model for each device.
To tackle this issue, different personalized FL algorithms have been proposed (Arivazhagan et al., 2019;
Fallah et al., 2020; Collins et al., 2021;
Oh et al., 2022) and most of them utilize fine-tuning to obtain a higher accuracy.
In particular, after training, each device uses its local data samples to fine-tune the obtained model by performing several iterations of stochastic gradient descent (SGD).
Personalization by fine-tuning may be limited if we focus on increasing the performance of a single model across all devices (Deng et al., 2020;
Oh et al., 2022).
Oh et al.
(2022) proposed FedBABU, where after decoupling the learning model into a global model and a local head model, the same randomly initialized local head model is considered for all the devices.
In FedBABU, the local head model is not updated during training.
Nevertheless, the aforementioned methods are not designed specifically to tackle the device heterogeneity issue.
FL Algorithms with Heterogeneous Devices.
Since different types of devices (e.g., smartphones, tablets, laptops) may participate in FL, the devices’ capabilities vary from one another.
Pruning method is one approach to address the device heterogeneity issue.
Jiang et al.
; Babakniya et al.
; Bibikar et al.
(2022) proposed different algorithms for consistent mask adjustment at the devices and mask aggregation at the server during training.
Some works have considered static pruning at initialization (i.e., before training).
Diao et al.
(2021) proposed HeteroFL, where the local model parameters for each device are considered to be a subset of the learning model parameters.
Hong et al.
(2022) proposed Split-Mix FL, where a large model is split into several small base subnetworks according to the model widths.
Each device selects some of the base models for training.
In general, the efficiency of different pruning techniques for obtaining an appropriate masking vector even in the centralized machine learning setting is still under investigation (Liu et al., 2022).
Since the pruned connections are no longer available in the model architecture, pruning at initialization may inevitably degrade the performance of FL algorithms.
Freezing method is another approach to address the device heterogeneity issue.
Sidahmed et al.
; Chen et al.
Pfeiffer et al.
(2022) proposed different algorithms, where the masking vectors are used to freeze a portion of the learning model for each device in FL.
Hence, while the required computation and communication resources can be reduced for each device, a more stable FL algorithm is obtained without changing the learning model architecture.
In this paper, we apply freezing method in FL.
We leverage the optimized masking vectors and fine-tuning to address both data and device heterogeneity issues.
Appendix A contains a full discussion of the related work.
P ROBLEM S ETTING
We consider one server and N edge devices.
Each device n ∈ [ N ] = { 1 , 2 , . .
, N } has its own set of local data samples D n .
In a supervised learning setting, each device aims to find a learning model θ n ∈ R d θ for predicting the true label y n given the input x n , ( x n , y n ) ∈ D n , where d θ denotes the dimension of the learning model.
Let f n ( θ n ) represent the expected loss over the data distribution of device n .
We have f n ( θ n ) =
E ( x n ,y n ) ∼ p n L ( θ n ; x n , y n ) , where L ( θ n ; x n , y n ) is the loss function that measures the prediction error of θ n over data samples ( x n , y n ) ∈ D n , and p n is the distribution over D n .
Formally, the optimization problem min θ 1 N (cid:80) Nn =1 f n ( θ ) is solved in the conventional FL.
All our results can easily be extended to the weighted averaging case, where the devices have different size of data samples.
In this work, we study both data and device heterogeneity issues using model personalization and masking vectors.
Model Personalization.
In the heterogeneous data setting, the probability distribution p n varies across the devices.
Unlike conventional FL problems, to obtain a more personalized solution for each device, the learning models θ n , n ∈ [ N ] , are not equal to each other.
Similar to FedBABU (Oh et al., 2022), we decouple each learning model θ n into a global representation model w g ∈ R d w and a device-specific head model ϕ n ∈ R d ϕ , where d w and d ϕ denote the dimensions of the global model and local head model, respectively.
We have d w + d ϕ =
d θ .
To further improve accuracy after training, we freeze the local head model parameters during training.
In particular, before training, ϕ g is initialized randomly by the server to be used for initialization of all the devices’ local head models. We have ϕ n = ϕ g , n ∈ [ N ] .
ϕ n remains unchanged during training until the convergence is reached for the global model w g .
The global model is obtained by minimizing the objective function F ( { w g , ϕ g } ) = 1 N (cid:80) Nn =1 f n ( { w g , ϕ n } ) throughout all the communication rounds, where the operator { ., . } denotes the concatenation of the learning model parameters.
Since ϕ g and ϕ n are never updated during training, with some abuse of notation, we consider the objective function as
F ( w g ) = 1 N (cid:80) Nn =1 f n ( w g ) . After convergence of w g , each device n obtains its personalized local head model ϕ n by fine-tuning the learning model θ n using its local data samples.
Masking Vectors.
In the heterogeneous device setting, devices vary in their computational and communication capabilities.
We consider that in each communication round, the devices perform τ local update iterations.
When deploying the same local models on all the devices, some devices with limited computational capability are not able to complete τ local update iterations and send their final local models to the server for aggregation in a timely manner.
To address the device heterogeneity issue, masking vectors are used to freeze a part of the local model parameters by customization.
A masking vector m n ∈ { 0 , 1 } d w is selected for each device n ∈ [ N ] based on its computational capability.
During local update iterations, each device n only updates those parameters in the global model that correspond to non-zero values of the masking vector m n .
Other parameters, which correspond to the elements of m n with zero values, are frozen during local updates.
Note that in FedBABU (Oh et al., 2022), all the elements of vector m n are equal to one for all devices n ∈ [ N ] .
Let w i n ( t ) denote the local model of device n at the beginning of local update iteration i in communication round t .
At initialization (i.e., i = 1 ), we set w 1 n ( t ) =
w g ( t ) .
At local update iteration i > 1 , the local model of device n is updated using SGD as follows:
[Equation]
where ⊙ denotes the element-wise product, η ( t ) is the learning rate, and b in ( t ) is the local batch sample chosen uniformly at random from the local dataset D n .
After performing τ local update iterations, each device n sends its final local model, i.e., w τ +1 n ( t ) to the server.
We have
[Equation]
Algorithm 2 in Appendix B describes the DeviceLocalUpdate function based on (1) and (2).
In the aggregation step, we consider that the server aggregates the received final local models by taking the masking vectors of the devices into account.
The global model for the next communication round can thus be determined through stable aggregation of unfrozen parameters, as follows:
[Equation]
where ( k n ) l = ( m n ) l (cid:80) n ′∈N ( t ) ( m n ′ )
l denotes the l -th element of vector k n .
N ( t ) denotes the set of devices participating in training in communication round t .
Specifically, the server selects a fraction c of N devices at random as participating devices in each communication round.
For each device n , vector k n is obtained as a normalized masking vector using the vectors m n , n ∈ N ( t ) .
Using k n in (3) indicates that the server only aggregates the updated parameters from the participating devices.
T HEORETICAL R ESULTS ON THE C ONVERGENCE R ATE OF THE G LOBAL M ODEL
In this section, we analyze the convergence rate of the global model when masking vectors are used by the devices.
Without loss of generality, we focus on non-convex loss functions.
We also present the convergence results for strongly convex loss functions in Appendix C.
In both cases, we assume that the loss functions are smooth.
For simplicity, we obtain our convergence results for the full device participation scenario 1 (i.e., c = 1 , N ( t ) =
[ N ] for all t ).
The analysis relies on the following assumptions, which are commonly used for obtaining the convergence rate of different
FL algorithms in the literature (Li et al., 2020b; Reddi et al., 2021; Amiri et al., 2022).
Assumption 1.
The function f n ( w ) , n ∈ [ N ] , is L -smooth and satisfies:
[Equation]
where f ∗ n denotes the minimum value of f n ( w ) .
Assumption 2.
∇ f n ( w in ( t ) , b in ( t )) is an unbiased stochastic gradient of function f n ( w ) .
The variance of the masked stochastic gradients is bounded for each device n ∈ [ N ] .
We have
[Equation]
Assumption 3.
The expected squared l 2 -norm of the masked stochastic gradients for all the devices is uniformly bounded.
We have
[Equation]
When the masking vectors are determined based on the computational capability of the devices, we define the term γ n = max l ( k n ) l to quantify the degree of device heterogeneity in the network.
Note that in the full device participation scenario, 1 N ≤ γ n ≤ 1 , n ∈ [ N ] .
In addition, γ n is inversely proportional to the minimum non-zero element of the vector (cid:80)
Nn ′ =
1 m n ⊙ m n ′ .
Hence, larger γ n implies a higher degree of device heterogeneity for device n ∈ [ N ] .
We first present the following lemma, which is derived using Theorem 3 in Fang et al.
From Lemma 1, we can quantify the impact of freezing the parameters by masking vectors on the convergence bound.
Lemma 1.
The following inequality holds for any vectors x and z ∈ R d , for which there exists
Q > 0 satisfying | min l ( x ⊙ z ) l | ≤ Q , and for any vector y ∈ R d :
[Equation]
where ⟨ .,
⟩ denotes the inner product operator in R d .
We use Lemma 1 to prove the following theorem concerning the device heterogeneity effect on the
FL convergence bound.
Devices with lower computational capability partially train the global model due to the zero-valued elements in their masking vectors.
We show that employing the masking vectors to address the device heterogeneity issue in FL leads to a bias term in the convergence bound.
However, it does not affect the convergence rate.
Theorem 1.
Under Assumptions 1 − 3, and for smooth and non-convex loss functions, if the total number of communication rounds T is pre-defined and the learning rate η ( t ) is small enough such that η ( t ) =
η ≤ [Equation] 2 τ , we have
[Equation]
where Ψ is a constant satisfying (cid:12)(cid:12) max l (cid:0) ∇ f n ( w in ( t )) ⊙ ∇ F ( w g ( t )) (cid:1) l (cid:12)(cid:12) ≤ Ψ for all n ∈ [ N ] , i = 1 , . . .
, τ , and t = 1 , . .
, T .
F ∗ =
F ( w ∗ ) , where w ∗ is the global optimal point.
L , ξ 2 n , and G are constants defined in Assumptions 1 − 3.
The proof for Lemma 1 and Theorem 1 can be found in Appendices D and E, respectively.
Remark 1.
By employing the masking vectors in FL, the term (cid:80) Nn =1 (cid:16) d w γ n − (cid:80) d w l =1 ( k n ) l (cid:17) appears on the right-hand side of (8).
Since this term does not scale with the number of communication rounds T , it is considered as a bias term, which remains as a residual in the convergence bound.
Hence, those FL algorithms, which use masking vectors to reduce the computational and communication costs for the devices, may converge to a local minimum of the objective function F ( w g ) .
In PerFedMask, we design the masking vectors by minimizing (cid:80) Nn =1 (cid:16) d w γ n − (cid:80) d w l =1 ( k n ) l (cid:17) to mitigate the performance degradation due to this bias term.
In Appendix F, we present a simple example to gain insight regarding the selection of masking vectors by minimizing the bias term.
O UR P ROPOSED A LGORITHM
In this section, we propose a novel algorithm called PerFedMask, which aims to mitigate the performance degradation caused by bias described in Remark 1 through: (1) systematically designing the the masking vectors via an optimization framework, and (2) fine-tuning the local head models.
First, each device determines the maximum number of parameters which can be trained during local update iterations based on its computational capability 2 .
These values are sent to the server.
The server then determines the masking vector for each device before training.
Let ψ n denote the maximum number of parameters that can be trained by device n ∈ [ N ] , where ψ n ≤ d w .
Given ψ n , the server determines the masking vector m n for each device n before training by minimizing the bias term.
Here, we present the formulation of layer-wise masking 3 , which decides whether or not to freeze all the parameters in each layer of the global model.
We adopt layer-wise masking since most of the current machine learning frameworks such as PyTorch (PyTorch, 2022) and TensorFlow (Abadi et al., 2016) run at the granularity of a full tensor, with no APIs which can provide parameter freezing at a finer granularity.
By considering layer-wise masking, the number of optimization variables can be reduced.
Thus, the complexity of obtaining the masking vectors can also be reduced accordingly.
Let Λ and | Λ | denote the set and number of layers in the global model, respectively.
Let π j and | π j | denote the set and number of parameters in layer j ∈ Λ , respectively.
We define ˜ m n ∈ { 0 , 1 } | Λ | as the layer-wise masking vector.
If ( ˜ m n ) j = 1 , then all the elements ( m n ) l , l ∈ π j are equal to one.
In PerFedMask, the following optimization problem is solved by the server to obtain the masking vectors for the devices:
[Equation]
minimize ˜
[Equation]
subject to
[Equation]
where ϵ n is a slack variable, which prevents to train more than ψ n parameters for each device n in the layer-wise masking.
Problem P mask is a mixed-integer nonlinear program, which is NP-hard and difficult to solve.
In Appendix H, we show how to obtain a close-to-optimal solution by using successive convex approximation (Shen et al., 2016) and MOSEK solver (MOSEK ApS, 2022).
Algorithm 1 summarizes the training procedure of PerFedMask.
EXPERIMENTAL RESULTS 6. E XPERIMENT S ETUP
Datasets and Model Architectures.
We conduct our experiments on CIFAR-10 and CIFARimage classification tasks 4 .
Our experiments are performed with ResNet (PreResNet18)
(He et al., 2016) for CIFAR-10, and with MobileNet (Howard et al., 2017) for CIFAR-100.
In both cases, we set the number of devices to 100 .
The data samples are uniformly divided among the devices.
Each device has 450 training data samples, 50 validation data samples, and 100 test data samples.
The batch size is set to 50 .
To enable non-IID data partitioning among the devices, we distribute 3 and 10 classes per device for CIFAR-10 and CIFAR-100 datasets, respectively.
The same classes are considered in the training, validation, and test datasets.
Implementation Details.
We denote the fraction of devices with the maximum computational capability by ν .
That is, ν represents the ratio of devices which can completely update the entire global model during the local update iterations.
Those devices are able to train × 1 parameters of the global model.
Since the remaining devices have lower computational capability, they should mask some parts of the global model during the local update iterations based on their capabilities.
Unless stated otherwise, we set ν = 0 .
For all the experiments, the learning rate starts with 0 . 1 and is decayed by a factor of 0 .
1 in communication round t ∈ { 12 T, 34 T } .
Similar to FedBABU (Oh et al., 2022), we fix the product of the local epochs E and the maximum number of communication rounds T to 320 .
After T communication rounds, we choose the model with the maximum validation accuracy and perform 5 local update iterations for fine-tuning the learning model using each device’s local training data samples.
We perform the experiments using PyTorch library (PyTorch, 2022) in Python 3.7.
We apply layer-wise masking in our experiments.
Baselines.
We compare the performance of our proposed algorithm, PerFedMask, with the following FL algorithms: FedBABU
(Oh et al., 2022) and FedProx (Li et al., 2020a), which have been proposed to tackle data heterogeneity; FedNova (Wang et al., 2020), which has been proposed to address objective inconsistency problem; HeteroFL (Diao et al., 2021) and Split-Mix FL (Hong et al., 2022), which have been proposed to tackle device heterogeneity in the non-IID data settings.
Performance Metrics.
We consider the test accuracy as one of the performance metrics.
Finetuning steps are performed for PerFedMask and FedBABU algorithms to personalize the learning model and obtain the local head model for each device.
For fair comparison, the obtained learning model has been also fine-tuned for each device in the other FL algorithms.
Thus, we report the test accuracy before and after fine-tuning.
We also assess the average number of floating-point operations (FLOPs) in each communication round for both the forward and backward propagation paths to show the required computation for the algorithms.
We use PyPAPI (PyPAPI, 2017) to obtain the number of FLOPs.
Since the average number of trainable parameters is equal to the average number of parameters transmitted from the devices to the server in each communication round, we report this number to show the communication cost of each FL algorithm.
6. B ENCHMARK E XPERIMENTS
We consider a heterogeneous device setting, where half of the devices (i.e., devices with maximum computational capability) perform four local epochs (i.e., E = 4 ).
Due to the limited computational capability, the remaining devices perform two local epochs for updating all the parameters.
Using PerFedMask, instead of considering different local epochs for the devices, we address the device heterogeneity issue by freezing some parts of the global model for those devices with lower computational capability.
For fair comparison, the number of frozen parameters are selected in a way such that the considered algorithms have the same number of FLOPs for each device.
Performance Comparison with the Baselines.
Table 1 shows the obtained test accuracy after fine-tuning and the number of trainable parameters for CIFAR-10 and CIFAR-100 datasets 5 .
Our observations are as follows: (1) our proposed algorithm, PerFedMask, has comparable performance We show some of the training curves in Appendix J. Also, in Appendix K, we present additional statistics for some of the results in Table 1. [Table] to FedBABU and outperforms the other baselines in terms of test accuracy after fine-tuning.
By increasing the number of devices participating in FL (i.e., by increasing c ), a higher test accuracy can be achieved.
(3) PerFedMask, HeteroFL, and Split-Mix FL can provide lower number of trainable parameters.
Split-Mix FL has the lowest number of trainable parameters because it trains several low-width base models instead of the original learning model.
(4) Different number of local update iterations for the devices may lead to the objective inconsistency problem.
PerFedMask, HeteroFL, and Split-Mix FL algorithms can address this problem by decreasing the number of FLOPs for the less capable devices.
Thus, the same number of local update iterations can be considered for all the devices.
Also, this problem is tackled in FedNova through modifying the optimizer.
However, other algorithms suffer from the objective inconsistency problem, which may degrade their performance.
Combining PerFedMask with HeteroFL and Split-Mix FL.
Table 2 shows the performance results for PerFedMask and its combination with HeteroFL and Split-Mix FL algorithms 6 .
Tablealso shows the performance results for FedBABU, HeteroFL, and Split-Mix FL algorithms.
The number of FLOPs for forward and backward indicate the number of required computations in the forward and backward propagation paths, respectively.
In general, backpropagation dominates the computational cost during training of a learning model (Xu et al., 2022).
In PerFedMask, by using masking vectors, there is no need to compute the partial derivative of the objective function with respect to the frozen parameters.
Although PerFedMask has reduced the number of trainable parameters and the backward FLOPs, it can still achieve higher test accuracy compared to other algorithms including FedBABU.
Since PerFedMask does not change the architecture of the learning model, it can easily be combined with other FL algorithms.
Table 2 shows that combining PerFedMask with
Split-Mix FL and HeteroFL algorithms can further reduce the number of FLOPs in the backward path and the number of trainable parameters.
This combination provides a higher test accuracy after fine-tuning than Split-Mix FL and HeteroFL algorithms.
[Table]
6. A BLATION S TUDIES
Effect of Fine-Tuning Steps.
We investigate the impact of the number of fine-tuning steps on the final test accuracy of PerFedMask and FedBABU 7 .
Fine-tuning steps equal to zero means the test accuracy is obtained before fine-tuning.
Also, since the number of batches for each device’s training dataset is equal to 9 , increasing the fine-tuning steps by one leads to 9 more local update iterations.
Results from Table 3 show that similar to FedBABU, PerFedMask can achieve better accuracy with a small number of fine-tuning steps.
This characteristic is important when fine-tuning is restricted or costly for the devices.
[Table]
Effect of Increasing the Number of Devices with Maximum Computational Capability.
We investigate the impact of increasing ν on the performance of PerFedMask in Table 4.
More devices in the network are able to train the entire global model as ν increases.
This leads to an increase in the number of trainable parameters and number of backward FLOPs in PerFedMask.
We can observe that by increasing ν , the test accuracy before fine-tuning is improved.
Note that PerFedMask can provide a comparable test accuracy after fine-tuning even for ν = 0 .
2 , when compared with the case in which all devices have the maximum computational capability (i
[Table]
e., ν = 1 )
[Table] [Table]
Effect of Masking Vectors Design.
In Table 5, we consider three masking approaches: in sequential masking, the layers are masked sequentially; in random masking, the layers are masked randomly; and in optimized masking, the layers are masked by solving problem P mask .
Optimized masking minimizes the bias described in Remark 1.
As shown in Table 5, optimized masking can provide lower training loss and higher training and test accuracies.
These results are compatible with our theoretical analysis.
As shown in Table 5, PerFedMask enhances FL performance before fine-tuning by employing optimal masking vectors.
The final test accuracy can then be improved by fine-tuning.
[Table]
C ONCLUSION
In this work, we proposed
a flexible and easy to implement personalized FL algorithm called PerFedMask.
We provided theoretical and empirical grounds to justify the utility of PerFedMask in heterogeneous data and device settings.
In particular, PerFedMask employs (1) optimized masking vectors obtained by minimizing the bias term in the convergence bound, and (2) fine-tuning.
The masking vectors are exploited to freeze some parts of the global model for each device based on its computational capability.
Fine-tuning is performed by each device after training to improve the final test accuracy.
When compared with some existing state-of-the-art FL algorithms, PerFedMask can achieve higher test accuracy.
It can also decrease the average number of trainable parameters and the average number of FLOPs in each communication round without changing the learning model architecture.
A future direction is to consider freezing priority for different layers in the neural network architecture.
For example, in Frankle et al.
(2021), it has been shown that batch normalization layers in convolutional networks are important to be considered as the trainable parameters.
Also, the approach of masking vectors design in this work can be emulated in pruning methods.
A R ELATED W ORK
FL Algorithms with non-IID Data.
In FedAvg (McMahan et al., 2017), the edge devices perform multiple local update iterations on their local models before sending them to the server for aggregation.
In this way, the communication cost can be reduced.
When there is data heterogeneity, conventional FL algorithms such as FedAvg may have slow and unstable convergence (Karimireddy et al., 2020).
To handle the case when the data is non-IID, some works (Li et al., 2020a; Karimireddy et al., 2020) have introduced new optimization frameworks to obtain a more stable model for the devices.
Although the obtained model may still perform well on average across the devices, some of the devices may not be satisfied with the obtained final accuracy (Collins et al., 2021).
To address this issue, more personalized models are required for the devices.
The personalized models can be obtained by using different methods such as meta-learning (Fallah et al., 2020), multi-task learning (Marfoq et al., 2021), clustering (Mansour et al., 2020), and decomposition.
In the decomposition method, the learning model is decomposed into a global model and a device-specific head model (Arivazhagan et al., 2019; Collins et al., 2021).
In each communication round, the global model is updated by the devices and is aggregated by the server, whereas the head model is updated by each device but is not transmitted to the server.
Oh et al.
(2022) proposed FedBABU using the decomposition method.
To improve the personalization ability, in FedBABU, the local head model parameters are frozen during training.
After convergence to a global model, each device updates the local head model parameters during the fine-tuning steps by using its local data samples.
FL Algorithms with Heterogeneous Devices.
Given the disparities in devices’ hardware, it is crucial to address device heterogeneity in FL.
In general, this problem can be tackled by reducing the computational complexity of model training based on the devices’ hardware capabilities.
Lin et al.
; Afonin & Karimireddy (2022) proposed FL algorithms using knowledge distillation, which aim to generate compact models by transferring knowledge of a large model to smaller ones.
Pruning method, where a compact model for each device can be obtained by removing the parameters with little impact on the performance of the original learning model, is another approach to accommodate device heterogeneity in FL.
Due to the dynamic sparse training, pruning method may suffer from instability in convergence for finding the sparse masking vectors, which are used for training the sparse sub-networks at the devices.
This issue can be resolved by consistent mask adjustment procedure at the devices, at the expense of additional communication overhead (Jiang et al., 2022;
Babakniya et al., 2022; Bibikar et al., 2022).
The ordered dropout technique has been proposed in
Horvath et al.
(2021) to dynamically adapt the model size used by each device based on its capabilities.
Another pruning method that does not require additional communication cost is to use a fixed masking vector for each device before training based on its computational capability.
This would result in training of the heterogeneous local models for the devices.
In this regard, HeteroFL (Diao et al., 2021) aims to facilitate efficient training and stable aggregation of devices’ local models.
However, the available local data samples at the devices may not be used efficiently in HeteroFL.
In particular, HeteroFL considers different model architectures as the local models for the devices.
Thereby, the available data samples at each device can be used only for training a specific model architecture.
In Split-Mix FL (Hong et al., 2022), based on its computational capability, each device randomly selects some of the base models in each communication round to train them.
After training, by using ensemble learning approach, each device mixes the selected base models to construct its desired model size.
The potential drawback of this approach is that none of the devices train the original learning model.
Another line of research is related to the objective inconsistency problem in FL.
Due to the different size of the devices’ local dataset and their computational capability, some of the devices may finish their local update iterations faster.
To prevent the devices from being idle, Wang et al.
(2020) proposed to let those faster devices continue their local update iterations until the slowest device finishes its local update.
FedNova is proposed by Wang et al.
(2020) to resolve the objective inconsistency problem due to different number of local update iterations.
Another approach is to consider the same number of local update iterations for all the devices.
However, to enable all the devices to finish their local update iterations in a timely manner, the required number of computations should be adjusted for those devices with lower computational capabilities.
Our proposed algorithm as well as other FL algorithms which aim to adjust the required computations for each device based on its computational capability can be employed to address the objective inconsistency problem.
B L OCAL U PDATES AT D EVICES
Return 11: end
C C ONVERGENCE A NALYSIS FOR S TRONGLY C ONVEX L OSS F UNCTIONS
To show the convergence of PerFedMask for smooth and strongly convex loss functions, in addition to Assumptions 1 − 3, we make the following assumption:
Assumption 4.
f n ( w ) , n ∈ [ N ] , is µ -strongly convex and satisfies:
[Equation]
To quantify the degree of data heterogeneity at each device n ∈ [ N ] , the term Γ n = f n ( w ∗ ) − f ∗ n is defined (Li et al., 2020b).
Let δ ( t ) =
E ∥ w g ( t ) − w ∗ ∥ 2 .
We first prove the following useful lemma.
Lemma 2.
Under Assumptions 1 − 4, if the learning rate is small enough, i.e., η ( t ) ≤ 1 L ( Nτ +1) , for all t = 1 , . .
, T , we have
[Equation]
where
[Equation]
and Ω and Υ should satisfy (cid:12)(cid:12) min l (cid:0)(cid:0)
[Equation]
( t ) (cid:1) ⊙ ∇ [Equation] n ( w in ( t )
[Equation]
(cid:1) l (cid:12)(cid:12) ≤ Ω , for all n ∈ [ N ] , i = 1 , . . .
, τ, t = 1 , . .
, T and (cid:12)(cid:12) min
n ∈ [ N ] ( f n ( w g ( t )
[Equation]
− f n ( w
[Equation]
≤ Υ , t = 1 , . . .
, T , respectively.
Proof.
See Appendix O.
Using Lemma 2, we can state the following theorem for the convergence rate of smooth and strongly convex loss functions:
Theorem 2.
Given Assumptions 1 − 4, if we choose κ = 2 Lq
( Nτ + 1) and the learning rate η ( t ) = 2 q 0 ( t + κ ) , under the full device participation scenario (i.e., c = 1 ), after T communication rounds, we have
[Equation]
Proof.
See Appendix P.
Remark 2.
The first term on the right-hand side of (15) appears in the convergence bound due to the device heterogeneity.
In particular, Theorem 2 shows that for q 1 ̸ = 0 , the FL algorithm converges to a local optimal solution at the rate of O (1 /T ) .
This convergence rate is similar to the results presented in Li et al.
(2020b); Amiri et al.
(2022), where device heterogeneity has not been considered.
Hence, using the masking vectors in FL do not degrade the convergence rate of FL.
Remark 3.
The result in Theorem 2 shows that when q 1 → 0 , the FL algorithm converges to the global optimal solution for the smooth and strongly convex functions.
Based on (13), it is straightforward to verify that without device heterogeneity in the network (i.e., when all the devices have the maximum computational capability and can update all the parameters of the global model), all the elements of vector k n , n ∈ [ N ] are equal to 1 N .
Hence, in this case, we have γ n = 1 N , and q 1 = 0 .
In general, based on (13), one way to reduce the bias incurred by the device heterogeneity is to design the masking vectors by minimizing q 1 .
For example, one can search for the masking vectors, which minimize (cid:80) N n =1 (cid:16) d w γ n − (cid:80) d w l =1 ( k n ) l (cid:17) .
D P ROOF OF L EMMA
Given vectors x , y , and z , we form diagonal matrices X , Y , and Z , respectively.
Note that we can write ⟨ x , y ⊙ z ⟩ as the form of the trace of matrices X , Y , and Z product, i.e., ⟨ x , y ⊙ z ⟩ =
Tr( XY Z ) .
By using Theorem 3 in Fang et al.
(1994), we have the following inequality:
[Equation]
where λ 1 ( Y ) and λ d ( XZ ) are the largest eigenvalue of matrix Y and the smallest eigenvalue of matrix XZ , respectively.
Since the considered matrices are diagonal, we have λ 1 ( Y ) =
max l ( y ) l and λ d ( XZ )
=
min l ( x ⊙ z ) l .
Hence, we have
[Equation]
Since d max l ( y ) l − (cid:80) dl
=1 ( y ) l ≥ 0 , by considering min l ( x ⊙ z ) l ≥ − Q , Lemma 1 is proved using inequality (17).
E P ROOF OF T HEOREM
Considering the smoothness of f n ( w ) , n ∈ [ N ] , in each communication round t ≥ 1 , we have
[Equation]
We first find an upper bound for ∥ w g ( t + 1) − w g ( t ) ∥ 2 as follows:
[Equation]
where equality (a) results from (2) and (3).
Equality (b) is obtained via basic equality E ∥ z
∥ 2 =
E ∥
z − E z ∥ 2 + ∥ E z ∥ 2 for any random vector z .
By using Assumption 2, we can obtain an upper bound of A 1 as follows:
[Equation]
By considering the convexity of ∥·∥ 2 and by using γ n = max l ( k n ) l , we can obtain an upper bound of A 2 as follows:
[Equation]
By combining (19), (20), and (21), we have the following inequality:
[Equation]
Now, we aim to obtain an upper bound of E ⟨ w g ( t + 1) − w g
( t ) , ∇ F ( w g ( t )) ⟩ .
We have
E
[Equation]
g
[Equation]
23) where equality (a) results from (2) and (3).
Equality (b) follows from E ∇ f n ( w in ( t ) , b in ( t ))
= ∇ f n ( w i n ( t )) .
Inequality (c) holds by using Lemma 1.
Inequality (d) follows from γ n ≥ 1 N .
To find an upper bound for − E (cid:68) 1 N (cid:80) Nn =1 ∇ f n ( w in ( t )) , ∇ F ( w g ( t )) (cid:69) , we first represent it as follows:
[Equation]
Then, E
[Equation]
=
[Equation]
t
[Equation]
t ))
(cid:1)(cid:13)(cid:13)(cid:13) 2 is bounded as follows:
[Equation]
where inequality (a) results from the convexity of ∥·∥ 2 .
Inequality (b) results from Assumption 1.
Now
[Equation]
we aim to bound E (cid:13)(cid:13) w g ( t ) − w in ( t ) (cid:13)(cid:13) 2 for i = 2 , . . .
, τ .
By using (1), we have
[Equation] [Equation]
where the last inequality results from Assumption 3.
By combining (25) and (26)
[Equation]
we have
[Equation]
By combining (18), (22), (23), and (27), we have
[Equation]
[Equation]
Since η ( t ) =
η ≤ 1 LN 2 τ , we have − η ( t ) 2 (cid:80) [Equation] f
[Equation]
(cid:13)(cid:13)
[Equation]
≤
[Equation]
By rearranging the terms in (28), we obtain
[Equation]
Finally, we multiply both sides of (29) by 1 T and sum over t = 1 , . . .
, T .
Then, Theorem 1 is concluded by considering that the first term on the right-hand side of (29) is a telescoping series.
We have
[Equation]
where the last inequality is obtained by considering that E F ( w g ( t + 1)) ≥ F ∗ .
F S ELECTION OF M ASKING V ECTORS IN A T OY E XAMPLE
In a heterogeneous device setting, devices with lower computational capability partially train the learning model.
Those devices use masking vectors to freeze a portion of the learning model during training based on their computational capability.
As shown in Theorem 1, employing masking vectors lead to a bias term in the convergence bound.
In this section, we demonstrate how the bias value is impacted by the design of masking vectors using a simple example.
We consider three devices.
One device has the maximum computational capability.
However, different scenarios are considered for the computational capability of the other two devices.
The devices aim to train a model with four parameters.
Hence, their masking vectors have four elements.
Fig.
2 shows the considered scenarios and the possible masking vectors selections, which may lead to different bias values.
For each scenario and for each possible selection, Table 6 shows vector k n and the value of γ n for each device as well as the obtained bias value.
For this example, the results in Table 6 illustrate that the bias value is minimized by freezing the same parameters for two devices with lower computational capability, in case each parameter cannot be trained at least by one of those devices (Figs. 2(a) and 2(b)).
However, when each parameter can be trained at least by one of those less capable devices, the bias value is minimized by freezing distinct parameters for the devices (Figs.
2(c) − 2(f)).
These results are compatible with the empirical results obtained by Pfeiffer et al.
(2022) and Yang et al.
(2022), where random selection is used by the devices to prevent freezing of the same parameters.
The results in Table 6 also show that by increasing the computational capability of the devices, the bias value can be decreased.
In the extreme case that all three devices have the maximum computational capability (i.e., Fig. 2(g)), the bias value is zero.
G E FFECT OF M ASKING R ATE ON THE C OMPUTATIONS
By increasing the masking rate (i.e., 1 − ψ n d w ), each device can reduce the number of trainable parameters and the number of FLOPs based on its computational capability.
Fig.
3 shows the reduction percentage 8 , which can be obtained for the number of trainable parameters and the number of FLOPs versus the masking rate.
For the results in Fig.
3, we have considered that a device performs each local update iteration on a batch containing 50 data samples of CIFAR-10 and CIFAR100 datasets using ResNet (PreResNet18) and MobileNet, respectively.
H S OLVING P ROBLEM P MASK BY SCA
In this section, we first transform the non-convex constraints in optimization problem P mask into convex functions or a difference of two convex functions.
Then, by using successive convex approximation (SCA) method, we can obtain a suboptimal solution for problem P mask in polynomial time.
We first rewrite constraint (9a) in the form of the following inequalities:
[Equation]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
γ 2 = 12 k 3 =
[ 12 , 13 , 12 , 1] γ 3 =
Selectionk 1 =
[ 12 , 12 , 0 , 0] γ 1 =2 k 2 = [0 , 0 , 12 , 12 ] γ 2 = 12 k 3 =
[ 12 , 12 , 12 , 12 ] γ 3 =
Fig
Selectionk 1 = [ 13 , 13 , 0 , 0] γ 1 =103 k 2 = [ 13 , 13 , 12 , 0] γ 2 = 12 k 3 =
[ 13 , 13 , 12 , 1] γ 3 =
Selectionk 1 =
[ 12 , 13 , 0 , 0] γ 1 =2 k 2 = [0 , 13 , 12 , 12 ] γ 2 = 12 k 3 =
[ 12 , 13 , 12 , 12 ] γ 3 =
Fig k 1 = k 2 = [ 13 , 13 , 13 , 0] γ 1 = γ 2 =83 k 3 =
[ 13 , 13 , 13 , 1] γ 3 =
Selectionk 1 =
[ 12 , 13 , 13 , 0] γ 1 =2 k 2 = [0 , 13 , 13 , 12 ] γ 2 = 12 k 3 =
[ 12 , 13 , 13 , 12 ] γ 3 =
Fig k 1 = k 2 = k 3 = [ 13 , 13 , 13 , 13 ] γ 1 = γ 2 = γ 3 =
[Figure]
c ti
[Figure]
P e r ce n t a g e p
e r L o ca l U pd a t e
I t e r a ti on
[Figure]
e du c ti
[Figure]
P e r ce n t a g e p e r L o
[Figure]
l U pd a t e
I t e r
[Figure] ti [Figure] Since ( ˜ m n ) j is a binary variable, the non-convex constraint (31b) can be expressed as the following convex constraint:
[Equation]
The non-convex constraint (31a) can be rewritten as follows:
[Equation]
Equality
[Equation]
can be used to express the left-hand side of (33) as a difference of two convex functions.
We have
[Equation]
Next [Equation] we relax the binary constraint (9c) in the form of the difference of two convex functions as follows: [Equation]
Finally
[Equation]
we define the following functions:
[Equation]
, (36b) where vector ˜ M j
= (( ˜ m n ′ ) j , n ′ ∈ [ N ]) . Algorithm 3 describes the SCA algorithm for solving problem P mask .
Let i denote the iteration index.
In Line 1, we initialize the maximum number of iterations i max .
In Line 2, the decision variables ˜ m (1) n and ˜ k
(1) n , n ∈
[ N ] are initialized in iteration i = 1 with a feasible solution of problem P mask .
In Line 4, the optimal solution of problem P mask-SCA (i.e., ˜ m ∗ n and ˜ k ∗ n , n ∈
[ N ] ) are determined.
In Line 5, using ˜ m ∗ n and ˜ k ∗ n , we update ˜ m
( i +1) n and ˜ k
( i +1) n , n ∈
[ N ] to be used for obtaining the first-order approximations ˆΘ(( ˜ m n ) j ) and ˆ ϑ ((˜ k n ) j , ˜ M j ) of functions Θ(( ˜ m n ) j ) and ϑ ((˜ k n ) j , ˜ M j ) , respectively.
The iteration index is updated in Line 6.
The steps within Lines 3 to 7 are repeated until the algorithm converges to a solution or we reach to i max .
The convex problem
P mask-SCA , which is solved in each iteration i , is as follows:
[Equation]
  subject to constraints (9b), (32), (35b), and (9d) ,    [Equation] [Equation]
N (cid:88) n
[Equation]
( ˜ m n
[Equation]
) j (cid:33)
[Equation]

[Equation] [ N ] , [Equation] I P ERFORMANCE C OMPARISON FOR D OMAIN N ET D ATASET
Different from the considered class non-IID configuration for CIFAR-10 and CIFAR-100 datasets, in this section, we evaluate the performance of our proposed algorithm under feature non-IID configuration.
We perform our experiment with AlexNet on DomainNet dataset.
The dataset contains images of six distinct domains including Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.
We consider 30 devices in the network, and split each domain among 5 devices.
We set the number of communication rounds and the learning rate to be 100 and 0 .
01 , respectively.
Table 7 shows the obtained test accuracy after fine-tuning and the number of trainable parameters for PerFedMask compared to other baseline algorithms.
The results in Table 7 indicate that PerFedMask can achieve a higher test accuracy compared to HeteroFL and Split-Mix FL algorithms, while the number of trainable parameters is much less than FedBABU, FedProx, FedNova
[Table]
and FedAvg algorithms
[Table]
[Table]
[Table]
[Figure]
J C OMPARISON OF T RAINING C URVES
This section includes some of the training curves for our experiments. Fig. 4 illustrates the training loss over communication rounds for PerFedMask and the baseline algorithms on CIFARand DomainNet datasets.
In Fig.
4, we have considered the full device participation scenario.
The difference between the training loss of PerFedMask and FedBABU is the bias resulted by the device heterogeneity.
Although we have minimized this bias through solving P mask , it has not been completely eliminated.
Fine-tuning after training helps to improve the performance of PerFedMask.
When c = 0 .
[Figure]
Fig.
5 shows the evolution of the validation accuracy over communication rounds for PerFedMask, FedBABU, FedAvg
[Figure]
and Split-Mix FL algorithms on CIFAR-10 and CIFARdatasets
[Figure]
[Figure]
[Figure]
K P ERFORMANCE C OMPARISON U SING B OX P LOTS
[Figure]
T e s t
A cc u r ac y a f t e r F i n e - T un i ng
[Figure]
L A VERAGE P ROCESSING T IME C OMPARISON
[Table]
M C OMBINATION OF P ER F ED M ASK WITH H ETERO FL AND S PLIT -M IX FL
[Table]
N E FFECT OF F REEZING THE L OCAL H EAD MODELS D URING T RAINING
In this section, we investigate the impact of freezing the local head models on the test accuracy of
PerFedMask.
In particular, we compare the test accuracy of PerFedMask with (w/) and without (w/o) freezing the local head models.
In the w/o head freezing scenario, the learning model is not decoupled into the global model and the local head model.
Table 10 shows that the test accuracy of
PerFedMask is improved by keeping the local head models frozen during training.
[Table]
O P ROOF OF L EMMA
We define two auxiliary sequences ¯ g ( t )
=
(cid:80) Nn =1 k n ⊙ (cid:80) τi
=1 ∇ f n ( w in ( t )) and g ( t ) =
(cid:80) Nn = k n ⊙ (cid:80) τi
=1 ∇ f n ( w in ( t ) , b in ( t )) .
We have
E ∥ w [Equation] [Equation]
+ 1) [Equation] w ∗ [Equation]
2 = E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) n = k n ⊙ w τ +1 n ( t ) − w ∗ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) n = k n ⊙ w g ( t ) − η ( t )
N (cid:88) n = k n ⊙ τ (cid:88)
[Equation]
Since E g ( t ) = ¯ g ( t ) [Equation] we have E A 3 = 0 .
Next [Equation] we focus on bounding A 1 .
[Equation]
By considering the convexity of ∥·∥ 2 and by using Assumption [Equation] we can bound B 1 as follows: [Equation]
n =
[Equation]
=
[Equation]
≤ Nτ
N
[Equation]
n
[Equation]
Nτ
[Equation] n [Equation]
=
[Equation]
≤
[Equation]
LNτ
N
[Equation] [Equation]
Next, we aim to bound B 2 .
We have
[Equation]
We first obtain an upper bound of C 1 .
We have
[Equation] [Equation]
where inequality (a) results from triangle and H ¨ older’s inequalities.
Inequality (b) results from the inequality of arithmetic and geometric means (AM-GM) inequality.
For inequality (c), we use
Assumption
[Equation]
Next, we obtain an upper bound of C 2 by using Lemma 1.
In particular, by considering x = w ∗ − w in ( t ) , y = k n , and z = ∇ f n ( w in ( t )) in Lemma 1, we can bound C 2 as follows:
[Equation]
42) where the last inequality results from Assumption 4.
Now, we focus on bounding D .
Considering the inequality ∥ x + y ∥ 2 ≤ 2 ∥ x ∥ 2 +2 ∥ y
∥ 2 for any x , y ∈ R d , and by replacing x with w i n ( t ) − w ∗ and y with w g ( t ) − w in ( t ) we have
[Equation]
By combining (38) − (43), we obtain
[Equation]
By rearranging the terms in (44)
[Equation]
we have
[Equation]
Now
[Equation]
we aim to bound f n ( w in ( t )) − f n ( w ∗ ) as follows: [Equation]
[Equation]
where inequality (a) results from the convexity of f n ( w ) , inequality (b) is obtained by using CauchySchwarz and AM-GM inequalities, and inequality (c) is due to Assumption 1.
By combining (26), (45), and (46), we have
[Equation]
Nτγ
[Equation]
N
[Equation]
n = γ n (1 − Lη ( t ) ( Nτγ n + 1)) E ( f n ( w g ( t ))
− f n ( w
[Equation]
+
[Equation]
( t )
N (cid:88) n = γ n (
[Equation]
(1 − Lη ( t ))
( Nτγ n + 1)) E ( f n ( w ∗ ) − f ∗ n )
[Equation]
t
[Equation]
Ω
N
[Equation]
n [Equation] d [Equation]
Lemma 3. For η ( t ) ≤ 1 L ( Nτ +1) , for all t = 1 , . . . , T , we have − 2 η ( t ) τ (1 − Lη ( t ))
N (cid:88) n = γ n (1 − Lη ( t ) ( Nτγ n + 1)) E ( f n ( w g ( t ))
− f n ( w ∗ )) ≤ 2 η ( t ) τ (1 − Lη ( t )) Υ
N (cid:88) n = (cid:18) γ n − 1 N (cid:19) − Lη ( t ) (cid:18) γ n ( Nτγ n + 1) − 1 N ( τ +
[Equation]
Proof.
See Appendix Q.
Using Lemma 3, we can simplify (47) as follows:
[Equation]
Finally, we use (20) to bound E
A 2 in (37).
Lemma 2 is concluded by combining (20), (37), and (49).
P P ROOF OF T HEOREM
First, through induction, we show that for a diminishing stepsize η ( t ) = 2 q 0 ( t + κ ) , we have δ ( t ) ≤ β ( t ) t + κ , where β ( t ) =
max { 2 q 1 q 0 ( t + κ ) + 4 q 2 q 20 , ( κ + 1) δ (1) } .
Note that the considered η ( t ) satisfies the mentioned condition in Lemma 2.
Moreover, the definition of β ( t ) ensures that the inequality holds for δ (1) .
Now, we assume that the inequality holds for t .
We show that it also holds for t
+ 1 .
From Lemma 2, we have
[Equation]
Finally, by the L -smoothness assumption for F , we have
[Equation]
From the definition of β ( t ) , we have β ( T ) ≤ 2 q 1 q 0 ( T + κ ) + 4 q 2 q 20 + ( κ + 1) δ (1) .
Combining this with (51) completes the proof of Theorem 2.
Q P ROOF OF L EMMA
First, we define two vectors x and y ∈ R N , where ( x ) n = γ n (1 − Lη ( t ) ( Nτγ n + 1)) and ( y ) n = f n ( w g ( t )) − f n ( w ∗ ) , n ∈ [ N ] , respectively.
Let X and Y , respectively, denote the corresponding diagonal matrices of vectors x and y .
By using Theorem 3 in Fang et al.
(1994), we have the following inequality: [Equation]
[Equation]
where λ N ( X ) and λ N ( Y ) are the smallest eigenvalue of matrices X and Y , respectively.
Since
X and Y are diagonal matrices, we have λ N ( X ) =
min n ∈ [ N ] ( x ) n and λ N ( Y ) =
min n ∈ [ N ] ( y ) n .
By considering η ( t ) ≤ 1 L ( Nτ +1) , all the elements of vector x including λ N ( X ) are nonnegative.
Also, ( x ) n is a quadratic function with respect to γ n .
For 1 N ≤ γ n ≤ 1 , by considering that η ( t ) ≤ 1 L ( Nτ +1) , it can be verified that the minimum value of ( x ) n is obtained at γ n = 1 N .
Thus, λ N ( X ) is lower bounded as λ N ( X ) ≥ 1 N (1 − Lη ( t ) ( τ + 1)) .
Hence, we have
[Equation]
γ [Equation] where for inequality (a), we use (52).
We also use the fact that − 2 η ( t ) τ (1 − Lη ( t )) ≤ 0 .
Note that since η ( t ) ≤ 1 L ( Nτ +1) , ∀ t , the quadratic function − 2 η ( t ) τ (1 − Lη ( t )) is always nonpositive.
Inequality (b) results from the definition of F ( w ) .
For inequality (c), since F ( w g ( t )) ≥ F ∗ , we use the fact that − 2 η ( t ) τ (1 − Lη ( t ))
λ N ( X ) N ( F ( w g ( t ))
− F ∗ ) ≤ 0 .
Finally, Lemma 3 is concluded by using the obtained lower bound for λ N ( X ) and by rearranging the terms in (53).