F ED P ROF : S ELECTIVE F EDERATED L EARNING WITH R EPRESENTATION P ROFILING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated Learning (FL) has shown great potential as a privacy-preserving solution to learning from decentralized data that are only accessible to end devices (i.e., clients).
In many scenarios however, a large proportion of the clients are probably in possession of low-quality data that are biased, noisy or even irrelevant.
As a result, they could significantly slow down the convergence of the global model we aim to build and also compromise its quality.
In light of this, we propose F ED P ROF , a novel algorithm for optimizing FL under such circumstances without breaching data privacy.
The key of our approach is a data representation profiling and matching scheme that uses the global model to dynamically profile data representations and allows for low-cost, lightweight representation matching.
Based on the scheme we adaptively score each client and adjust its participation probability so as to mitigate the impact of low-value clients on the training process.
We have conducted extensive experiments on public datasets using various
FL settings.
The results show that F ED P ROF effectively reduces the number of communication rounds and overall time (up to 4.5x speedup) for the global model to converge and provides accuracy gain.
I NTRODUCTION
With the advances in Artificial Intelligence (AI), we are seeing a rapid growth in the number of
AI-driven applications as well as the volume of data required to train them.
However, a large proportion of data used for machine learning are often generated outside the data centers by distributed resources such as mobile phones and IoT (Internet of Things) devices.
It is predicted that the data generated by IoT devices will account for 75% of the total in 2025 (Meulen, 2018).
Under this circumstance, it will be very costly to gather all the data for centralized training.
More importantly, moving the data out of their local devices (e.g., mobile phones) is now restricted by law in many countries, such as the General Data Protection Regulation (GDPR) 1 enforced in EU.
We face three main difficulties to learn from decentralized data: i) massive scale of end devices; ii) limited communication bandwidth at the network edge; and iii) uncertain data distribution and data quality.
As an promising solution, Federated Learning (FL) (McMahan et al., 2017) is a framework for efficient distributed machine learning with privacy protection (i.e., no data exchange).
A typical process of FL is organized in rounds where the devices (clients) download the global model from the server, perform local training on their data and then upload their updated local models to the server for aggregation.
Compared to traditional distributed learning methods, FL is naturally more communication-efficient at scale (Koneˇcn`y et al., 2016; Wang et al., 2019).
Nonetheless, several issues stand out.
1. M OTIVATION
1) FL is susceptible to biased and low-quality local data .
Only a fraction of clients are selected for a round of FL (involving too many clients leads to diminishing gains (Li et al., 2019)).
The standard FL algorithm (McMahan et al., 2017) selects clients randomly, which implies that every client (and its local data) is considered equally important.
This makes the training process susceptible to local data with strong heterogeneity and of low quality (e.g., user-generated texts (Hard et al.,
Communication rounds 0.
A cc u r a cy o f g l oba l m ode l unbiased and noiseless biased local data noisy local data
[Figure]
2018) and noisy photos).
In some scenarios, local data may contain irrelevant or even adversarial samples (Bhagoji et al., 2019; Bagdasaryan et al., 2020) from malicious clients (Fang et al., 2020;
Bagdasaryan et al., 2020;
Tolpegin et al., 2020).
Traditional solutions such as data augmentation (Yoo et al., 2020) and re-sampling (Lin et al., 2017) prove useful for centralised training but applying them to local datasets may introduce extra noise (Cui et al., 2019) and increase the risk of information leakage (Yu et al., 2021).
Another naive solution is to directly exclude those low-value clients with low-quality data, which, however, is often impractical because i) the quality of the data depends on the learning task and is difficult to gauge; ii) some noisy or biased data could be useful to the training at early stages (Feelders, 1996); and iii) sometimes low-quality data are very common across the clients.
In Fig.
1 we demonstrate the impact of involving ”low-value” clients by running FL over 100 clients to learn a CNN model on MNIST using the standard F ED A VG algorithm.
From the traces we can see that training over clients with problematic or strongly biased data can compromise the efficiency and efficacy of FL, resulting in an inferior global model that takes more rounds to converge.
2) Learned representations can reflect data distribution and quality .
Representation learning is vital to the performance of deep models because learned representations can capture the intrinsic structure of data and provide useful information for the downstream machine learning tasks (Bengio et al., 2013).
In ML research, The value of representations lies in the fact that they characterize the domain and learning task and provide task-specific knowledge (Morcos et al., 2018; Kornblith et al., 2019).
In the context of FL, the similarity of representations are used for refining the model update rules (Li et al., 2021;
Feng & Yu, 2020), but the distributional difference between representations of heterogeneous data is not yet explored.
Our study is also motivated by a key observation that representations from neural networks tend to have Gaussian patterns.
As a demonstration we trained two different models (LeNet-5 and ResNet18) on two different datasets (MNIST and CIFAR-100) separately.
Fig.
2a shows the neuron-wise distribution of representations extracted from the first dense layer (FC-1) of LeNet-5.
Fig.
2b shows the distribution of fused representations (in a channel-wise manner) extracted from a plain convolution layer and a residual block of ResNet-18.
These observations motivate us to study the distributional property of data representations and use it as a means to differentiate clients’ value.
[Figure]
[Figure] [Figure]
1. C ONTRIBUTIONS
Our contributions are summarized as the following:
• We first provide theoretical proof for the observation that data representations from neural networks tend to follow Gaussian distribution, based on which we propose a representation profiling and matching scheme for fast, low-cost comparison between different representation profiles.
• We present a novel FL algorithm F ED P ROF that adaptively adjusts clients’ participation probability based on representation profile dissimilarity.
• Results of extensive experiments show that F ED P ROF reduces the number of communication rounds by up to 77%, shortens the overall training time (up to 4.5x speedup) while increasing the accuracy of the global model by up to 2.5%.
R ELATED W ORK
Different from traditional distributed training methods (e.g., Alistarh et al.
; Wu et al.
Zheng et al.
, Federated Learning assumes strict constraints of data locality and limited communication capacity (Kone ˇ cn ` y et al., 2016).
Much effort has been made in optimizing FL and covers a variety of perspectives including communication (Koneˇcn`y et al., 2016; Niknam et al., 2020; Cui et al., 2021), update rules (Li et al., 2020; Wu et al., 2021a; Leroy et al., 2019; Luping et al., 2019), flexible aggregation (Wang et al., 2019; Wu et al., 2021b) and personalization (Fallah et al., 2020;
Tan et al., 2021; Deng et al., 2020).
The control of device participation is imperative in cross-device FL scenarios (Kairouz et al., 2019;
Yang et al., 2020) where the quality of local data is uncontrollable and the clients show varied value for the training task (Tuor et al., 2020).
To this end, the selection of clients is pivotal to the convergence of FL over heterogeneous data and devices (Nishio & Yonetani, 2019;
Wang et al., 2020b; Chai et al., 2019; Acar et al., 2020).
Non-uniform client selection is widely adopted in existing studies (Li et al., 2020; Goetz et al., 2019; Cho et al., 2020;
Li et al., 2019; Chen et al., 2020b; Wang et al., 2020a)
and has been theoretically proven with convergence guarantees (Chen et al., 2020b; Li et al., 2019).
Many approaches sample clients based on their performance (Nishio & Yonetani, 2019; Chai et al., 2020) or aim to jointly optimize the model accuracy and training time (Shi et al., 2020; Chen et al., 2020a; 2021).
A popular strategy is to use loss as the information to guide client selection (Goetz et al., 2019;
Lai et al., 2021; Sarkar et al., 2020).
For example, AFL (Goetz et al., 2019) prioritizes the clients with high loss feedback on local data, but it is potentially susceptible to noisy and unexpected data that yield illusive loss values.
Data representations are useful in the context of FL for information exchange (Feng & Yu, 2020) or objective adaptation.
For example, Li et al.
(2021) introduces representation similarities into local objectives.
This contrastive learning approach guides local training to avoid model divergence.
Nonetheless, the study on the distribution of data representations is still lacking whilst its connection to clients’ training value is hardly explored either.
D ATA R EPRESENTATION P ROFILING AND M ATCHING
In this paper, we consider a typical cross-device FL setting (Kairouz et al., 2019), in which multiple end devices collaboratively perform local training on their own datasets
D i , i = 1 , 2 , ..., n .
The server owns a validation dataset D ∗ for model evaluation.
Every dataset is only accessible to its owner.
Considering the distributional pattern of data representations (Fig. 2) and the role of the global model in FL, we propose to profile the representations of local data using the global model.
In this section, we first provide theoretical proof to support our observation that representations from neural network models tend to follow Gaussian distribution.
Then we present a novel scheme to profile data representations and define profile dissimilarity for fast and secure representation comparison.
3. G AUSSIAN D ISTRIBUTION OF R EPRESENTATIONS
We first make the following definition to facilitate our analysis.
Definition 1 (The Lyapunov’s condition) .
A set of random variables { Z 1 , Z 2 , . . .
, Z v } satisfy the
Lyapunov’s condition if there exists a δ such that
[Equation]
where µ k = E[ Z k ] , σ 2 k = E[( Z k − µ k ) 2 ] and s =
(cid:112)(cid:80) vk =1 σ 2 k .
The Lyapunov’s condition can be intuitively explained as a limit on the overall variation (with | Z k − µ k | 2+ δ being the (2 + δ ) -th moment of Z k ) of a set of random variables.
Now we present Proposition 1 and Proposition 2.
The Propositions provide theoretical support for our representation profiling and matching method to be introduced in Section 3.2.
Proposition 1.
The representations from linear operators (e.g., a pre-activation dense layer or a plain convolutional layer) in a neural network tend to follow the Gaussian distribution if the layer’s weighted inputs satisfy the Lyapunov’s condition.
Proposition 2.
The fused representations 2 from non-linear operators (e.g., a hidden layer of LSTM or a residual block of ResNet) in a neural network tend to follow the Gaussian distribution if the layer’s output elements satisfy the Lyapunov’s condition.
The proofs of Propositions 1 and 2 are provided in Appendices A.1 and A.2, respectively.
We base our proof on the Lyapunov’s CLT which assumes independence between the variables.
The assumption theoretically holds by using the Bayesian network concepts: let X denote the layer’s input and H k denote the k -th component in its output.
The inference through the layer produces dependencies X
→ H k for all k .
According to Local Markov Property, we have H i independent of any H j ( j ̸ = i ) given X .
Also, the Lyapunov’s condition is typically met when the model is properly initialized and batch normalization is applied.
Next, we discuss the proposed representation profiling and matching scheme.
3. D ISTRIBUTIONAL P ROFILING AND M ATCHING
Based on the Gaussian pattern of representations, we compress the data representations statistically into a compact form called representation profiles .
The profile produced by the global model w on a dataset D , denoted by RP ( w, D ) , has the following format:
[Equation]
where q is the profile length determined by the dimensionality of the representations.
For example, q is equal to the number of kernels for channel-wise fused representations from a convolutional layer.
The tuple ( µ i , σ 2 i ) contains the mean and the variance of the i -th representation element.
Local representation profiles are generated by clients and sent to the server for comparison (the cost of transmission is negligible considering each profile is only q × 8 bytes).
Let RP k denote the local profile from client k , and RP ∗ denote the baseline profile (generated in model evaluation) on the server.
The dissimilarity between RP k and RP ∗ , denoted by div ( RP k , RP ∗ ) , is defined as:
[Equation]
where KL( · ) denotes the Kullback–Leibler (KL) divergence.
An advantage of our profiling scheme is that a much simplified KL divergence formula can be adopted because of the Gaussian distribution property (see Roberts & Penny (2002, Appendix B) for details), which yields:
[Equation]
Eq.
(4) computes the KL divergence without calculating any integral, which is computationally cost-efficient.
Besides, the computation of profile dissimilarity can be performed under the Homomorphic Encryption for minimum knowledge disclosure (see Appendix D for details).
T HE T RAINING A LGORITHM F ED P ROF
Our research aims to optimize the global model over a large group of clients (datasets) of disparate training value.
Given the client set U ( | U | = N ) , let D k denote the local dataset on client k and D ∗ the validation set on the server, We formulate the optimization problem in (5) where the coefficient ρ k differentiates the importance of the local objective functions F k ( w ) and depends on the data in
D k .
Our global objective is in a sense similar to the agnostic learning scenario (Mohri et al., 2019) where a non-uniform mixture of local data distributions is implied.
[Equation]
where w is the parameter set of the global model.
The coefficients { ρ k } Nk =1 add up to 1.
F k ( w ) is client k ’s local objective function of training based on the loss function ℓ ( · ) :
[Equation]
Involving the ”right” clients facilitates the convergence.
With this motivation we score each client with λ k each round based on the representation profile dissimilarity:
[Equation]
[Equation]
where v k is the version of RP k (i.e., the version of the global model that client k receives) and α is a preference factor deciding how biased the selection strategy needs to be towards the clients with small profile dissimilarity (i.e., higher training value).
With α = 0 , our strategy is equivalent to random selection.
The scores connect the representation profiling and matching scheme to the design of the selective client participation strategy adopted in our FL training algorithm F ED P ROF , which is outlined in Algorithm 1 (see Appendix B for a detailed version with both client and server processes).
The key steps of our algorithm are local representation profiling (line 5), baseline representation profiling (line 9) and client scoring (line 3).
Fig.
3 illustrates the workflow of the proposed algorithm representation profiling scheduling.
the global model benchmark data local data
D D
D n
D * local proﬁles baseline footprints [ [ 0.
client credits match & rate
Credit-basedclient selec!on
Update footprints and credits Model distribu!on
Local training & model uploading
Model aggrega!on
Round +
RP RP
RP n
RP * ... representa"ons
Input: maximum number of rounds T max , iterations per round E , fraction C 1 Initialize global model w and generate baseline profile RP ∗ 2 Collect initial representation profiles { RP k } k ∈ U from all clients for round T ← 1 to T max do
Update client scores { λ k } k ∈ U and compute Λ = (cid:80) k ∈
U λ k
S ← Choose K = N · C clients by probability distribution { λ k Λ } k ∈ U for client k in S in parallel do
RP k ← updateProfile( k, w, T − 1 ) w ← localTraining( k w, E ) end
Collect profiles clients S
Update aggregation
Evaluate and update RP end 10 return w
The convergence rate of FL algorithms with opportunistic client selection (sampling) has been extensively studied in the literature (Li et al., 2019; Chen et al., 2020b; Wang et al., 2019). Inspired by these studies, we present Theorem 1 to guarantee the global model’s convergence for our algorithm. Similar to Stich et al. ; Zhang et al.
; Li et al. (2019), we assume that { F k } Nk =1 are L -smooth and µ -strongly convex and that in expectation, the variance of local stochastic gradients are bounded by ϵ 2 and their squared norms are bounded by G 2 . For each local update step t we define S ( t ) as the set of selected clients in the associated round.
Theorem 1.
Using partial aggregation and a strategy π that selects clients by the probability distribution { q k } Nk =1 , the global model w ( t ) converges in expectation by having q k = ρ k , an aggregation interval E ≥ 1 and a decreasing step size (learning rate) η t = [Equation]
where t ∈ T A = { nE | n = 1 , 2 , . .
} , γ = max { 8 Lµ , E }− 1 , B = (cid:80) Nk
=
ρ 2 k
ϵ 2 k +6 L Γ+8( E − 1) 2 G 2 ,
C = 4 K E 2 G 2 , Γ = F ∗ − (cid:80) Nk =
[Equation] ρ k F [Equation] , ∆
[Equation] | [Equation] | = N · C .
The proof of Theorem 1 is provided in Appendix C.
E XPERIMENTS
We conducted extensive experiments to evaluate F ED P ROF under various FL settings.
Apart from
F ED A VG (McMahan et al., 2017), we also reproduced several state-of-the-art FL algorithms for comparison.
For fair comparison, the algorithms are grouped by the aggregation method (i [Table] e., full aggregation and partial aggregation) and configured following the hyper-parameter settings in their papers (if any).
Table 1 summarizes these FL algorithms.
Note that our algorithm can adapt to both aggregation methods.
[Table]
5. E XPERIMENT S ETUP
We built our simulated FL system and implemented the algorithms based on the Pytorch framework (Build 1.7.0).
We first set up a Small-scale Task (S-Task) to learn a multi-layer feed-forward network model from decentralized sensor data for predicting carbon monoxide (CO) and nitrogen oxides (NOx) emissions using the GasTurbine 3 dataset.
Next, we set up a Large-scale Task (L-Task) to train a CNN over a large population of user devices for image classification ( EMNIST 4 ).
In both tasks, data sharing is not allowed between any parties.
The data are non-IID across the end devices.
We introduce a diversity of noise into the local datasets on end devices to simulate the discrepancy in data quality.
The S-Task is performed over 50 sensor clients and 50% of them produce noisy data (including 10% invalid).
The population for L-Task contains 1000 end devices across which the data spread with strong class imbalance – roughly 60% of the samples on each device fall into the same class.
15% of the local datasets in the L-Task are irrelevant images whereas another 45% of them are low-quality images (blurred or affected by salt-and-pepper noise).
Considering the population of the clients, the setting of the selection fraction C is based on the scale of the training participants suggested by Kairouz et al.
For both tasks, the clients are heterogeneous in terms of both performance and communication bandwidth.
More experimental settings are listed in Table 4 in
Appendix E where the environment setup is also given in details.
5. E VALUATION R ESULTS
We evaluate the performance of our F ED P ROF algorithm in terms of the efficacy (best accuracy achieved) and efficiency (costs for convergence) in establishing a global model for the two tasks.
Tables 2 and 3 report the average results of multiple runs with standard deviations for our algorithm.
Figs. 4 and 5 plot the accuracy traces from the round-wise evaluations of the global model.
1) Convergence in different aggregation modes: Our results show a great difference in convergence rate under different aggregation modes.
From Figs.
4 and 5, we observe that partial aggregation facilitates faster convergence of the global model than full aggregation, which is consistent with the observations made by Li et al.
The advantage is especially obvious in the L-Task where partial aggregation requires much fewer communication rounds to reach the 90% accuracy.
Our
F ED P ROF algorithm yields the fastest convergence in both groups of comparison for both tasks because selective participation benefits both aggregation methods. [Table]
[Table] 2) Best accuracy of the global model: Through the training process of FL, the global model is evaluated each round on the server.
The best global model obtained thus far is stored on the server.
As shown in the 2nd column of Tables 2 and 3, our F ED P ROF algorithm improves the accuracy achieved by 2.5% when compared against the baselines (F ED A VG and F ED A VG -RP).
The AFL algorithm uses a loss-oriented client selection strategy, which shows the closest performance to our algorithm in the L-Task but the worst accuracy in the S-Task.
3) Total communication rounds for convergence: The number of communication rounds required for reaching convergence is a key indicator to the efficiency of FL.
In the S-Task, our algorithm takes less than half the communication rounds required by other algorithms in most cases.
In the L-Task with C =0.05, our algorithm reaches 90% accuracy within 100 rounds whilst F ED A VG and CFCFM need more than 200.
In this case, F ED P ROF also achieves approximately 7% higher accuracy in 50 rounds.
Partial aggregation turns out to be much more communication-efficient: F ED A VG -RP [Figure] [Figure] needs 8 rounds to reach the accuracy target for the S-Task and 12 rounds for the L-Task, whilst our algorithm reduces the numbers to 5 and 8, respectively.
4) Total time needed for convergence: The overall time consumption is closely related to total communication rounds needed for convergence and the time cost for each round.
Algorithms requiring more rounds to converge typically take longer to reach the accuracy target except the case of CFCFM, which priorities the clients that work faster.
Using F ED A VG as the baseline, CFCFM accelerates the training process by 2.1x whilst our algorithm provides a 4.5x speedup for achieving the same target accuracy (S-Task, C =0.3).
F ED P ROF also has a clear advantage over F ED A VGRP, F ED P ROX and F ED A DAM in the partial aggregation group where it shows a 2.6x speedup over
F ED A VG -RP in the S-Task with C =0.2.
5) Energy consumption of end devices: A main concern for the end devices, as the participants of
FL, is their power usage (Watt hours).
Full aggregation methods experience slower convergence and thus endure higher energy cost on the devices.
For example, with a small selection fraction C =0.
in the L-Task, F ED A VG and CFCFM consume over 60Wh to reach the target accuracy.
In this case, our algorithm reduces the cost by more than a half (28.22Wh).
With the partial aggregation mode, the reduction by our algorithm is up to 62% (S-Task, C = 0 . 2 ).
Considering all the cases, F ED P ROF achieves the target accuracy with the least energy cost, providing an reduction of 29% ∼ 53%.
6) Differentiated participation with F ED P ROF : Fig.
6 reflects the preference of our selection strategy.
In the S-Task we can observe that the clients with useless samples or noisy data get significantly less involved ( < 10 on average) in training.
In the L-Task our algorithm also effectively limits (basically excludes) the clients who owns the image data of poor quality (i.e., irrelevant or severely blurred), whereas the clients with moderately noisy images are selected with reduced frequency as compared to those with normal data.
A potential issue of having the preference towards some of the devices is about fairness.
Nonetheless, one can apply our algorithm together with an incentive mechanism (e.g., Yu et al.
to address it.
[Figure]
C ONCLUSION
Federated learning provides a privacy-preserving approach to decentralized training but is vulnerable to the heterogeneity and uncertain quality of on-device data.
In this paper, we use a novel approach to address the issue without violating the data locality restriction.
We first provide key insights for the distribution of data representations and then develop a dynamic data representation profiling and matching scheme.
Based on the scheme we propose a selective FL training algorithm F ED P ROF that adaptively adjusts clients’ participation chance based on their profile dissimilarity.
We have conducted extensive experiments on public datasets under various environment settings.
Evaluation results show that our algorithm significantly improves the efficiency of FL and reduces the time and energy costs for the global model to converge.
A P ROOF OF P ROPOSITIONS A. P ROOF OF P ROPOSITION
Without loss of generality, we provide the proof of Proposition 1 for the pre-activation representations from dense (fully-connected) layers and standard convolutional layers, respectively.
The results can be easily extended to other linear neural operators.
Dense layers
Proof.
Let Ω = { neu 1 , neu 2 , ..., neu q } denote a dense layer (with q neurons) of any neural network model and H k denote the pre-activation output of neu k in Ω .
We first provide the theoretical proof to support the observation that H k tends to follow the Gaussian distribution.
Let χ = R v denote the input feature space (with v features) and assume the feature X i (which is a random variable) follows a certain distribution ζ i ( µ i , σ 2 i ) (not necessarily Gaussian) with finite mean µ i =
E
[ X i ] and variance σ 2 i =
E
[ X i − µ i ] .
For each neuron neu k , let w k
=
[ w k, 1 w k, 2 . . .
w k,v ] denote the neuron’s weight vector, b k denote the bias, and Z k,i =
X i w k,i denote the i -th weighted input.
Let H k denote the output of neu k .
During the forward propagation, we have:
[Equation]
Apparently Z k,i is a random variable because Z k,i =
X i w k,i (where the weights w k,i are constants during a forward pass), thus H k is also a random variable according to Eq.
In an ideal situation, the inputs variables X 1 , X 2 , . . .
, X v may follow a multivariate Gaussian distribution, in which case Proposition 1 automatically holds due to the property of multivariate normal distribution that every linear combination of the components of the random vector ( X 1 , X 2 , . . .
, X v ) T follows a Gaussian distribution (Lemons, 2003).
In other words, H k =
X 1 w k, 1 + X 2 w k, 2 + . . .
+ X v w k,v + b k is a normally distributed variable since w k,i and b k ( k = 1 , 2 , . .
. , v ) are constants in the forward propagation.
A special case for this condition is that
X 1 , X 2 , . .
, X v are independent on each other and X i follows a Gaussian distribution N ( µ i , σ 2 i ) for all i = 1 , 2 , . .
, v .
In this case, by the definition of Z k,i , we have:
[Equation]
where Z 1 , Z 2 , . .
, Z v are independent on each other.
Combining Eqs.
(9) and (10), we have:
[Equation]
For more general cases where X 1 , X 2 , . . .
, X v are not necessarily normally distributed, we assume the weighted inputs Z k,i of the dense layer satisfy the Lyapunov’s condition (see definition 1).
As a result, we have the following according to the Central Limit Theorem (CLT) (Billingsley, 2008) considering that X i follows ζ i ( µ i , σ 2 i ) :
[Equation]
12) where s k = (cid:113) (cid:80) vi
=1 (cid:0) w k,i σ i (cid:1) 2 and N (0 , 1) denotes the standard normal distribution.
Equivalently, for every neu k we have:
[Equation]
Combining Eqs.
(9) and (13) we can derive that:
[Equation] [Equation]
which means that H k ( k = 1 , 2 , . .
. , v ) tend to follow Gaussian distribution and proves our Proposition 1 for fully-connected layers.
Convolutional layers
Proof.
Standard convolution in CNNs is also a linear transformation of the input feature space and its main difference from dense layers rests on the restricted size of receptive field.
Without loss of generality, we analyze the representation (output) of a single kernel.
To facilitate our analysis for convolutional layers, let C denote the number of input channels and K denote the kernel size.
For ease of presentation, we define a receptive field mapping function Θ( k, i, j ) that maps the positions ( k for channel index, i and j for indices on the same channel) of elements in the feature map (i.e., the representations) to the input features.
For the k -th kernel, let W k denote its weight tensor (with
W k,c being the weight matrix for channel c ) and b k its bias.
Given the corresponding input patch X Θ( k,i,j ) , The element H k,i,j of the representations from a convolutional layer can be formulated as:
[Equation]
(15) where ◦ denotes Hadamard product.
The three summations reduce the results of element-wise product between the input patch and the k -th kernel to the correspond representation element
H k,i,j in the feature map.
For ease of presentation, here we use the notation Z ( k ) c,i ′ ,j ′ to replace (cid:0) X Θ( k,i,j ) ◦ W k,c (cid:1) i ′ ,j ′ and let ζ ( µ c,i ′ ,j ′ , σ 2 c,i ′ ,j ′ ) be the distribution that Z ( k ) c,i ′ ,j ′ follows.
Note that ζ can be any distribution since we do not make any distributional assumption on Z ( k ) c,i ′ ,j ′ .
With the notations, Eq.
(15) can be rewritten in a similar form to Eq.
[Equation]
[Equation]
We use the condition that the random variables Z ( k ) c,i ′ ,j ′ satisfy the Lyapunov’s condition, i.e., there exists a δ such that
[Equation]
=
[Equation] [Equation]
where s = (cid:113)(cid:80)
Cc =1 (cid:80) Ki ′
=1 (cid:80)
Kj ′
=1 σ
[Equation]
i
[Equation] [Equation]
Then according to the Lyapunov CLT, the following holds:
[Equation] [Equation]
which proves our Proposition 1 for standard convolution layers.
A. P ROOF OF P ROPOSITION
Without loss of generality, we prove Proposition 2 for the fused representations from the LSTM layer and the residual block of ResNet models, respectively.
The results can be easily extended to other non-linear neural operators.
Proof.
Long Short-Term Memory (LSTM) models are popular for extracting useful representations from sequence data for tasks such as speech recognition and language modeling.
Each LSTM layer contains multiple neural units.
For the k -th unit, it takes as input the current input feature vector
X t =
( X t, 1 , X t, 2 , . . . ) , hidden state vector H t − 1 and its cell state c t − 1 ,k .
The outputs of the unit are its new hidden state h t,k and cell state c t,k .
In this paper, we study the distribution of h t,k .
Multiple gates are adopted in an LSTM unit: by i t,k , f t,k , g t,k and o t,k we denote the input gate, forget gate, cell gate and output gate of the LSTM unit k at time step t .
The update rules of these gates and the cell state are:
[Equation]
where the W ( i ) k , W ( f ) k , W ( g ) k and W ( o ) k are the weight parameters and b ( i ) k , b ( f ) k , b ( g ) k and b ( o ) k are the bias parameters for the gates.
The output of the LSTM unit h t,k is calculated as the following:
[Equation]
Using the final hidden states h T,k (with T being the length of the sequence) as the elements of the layer-wise representation, we apply the following layer-wise fusion to further produce H over all the h T,k in a single LSTM layer:
[Equation]
where d is the dimension of the LSTM layer.
Again, by ζ ( µ k , σ 2 k ) we denote the distribution of h T,k (where the notation T is dropped here since it is typically a fixed parameter).
With { h T,k | k
= 1 , 2 , . . .
, d } satisfying the Lyapunov’s condition and by Central Limit Theorem H , tends to follow the Gaussian distribution:
[Equation]
which proves the Proposition 2 for layer-wise fused representations from LSTM.
Residual blocks
Proof.
Residual blocks are the basic units in the Residual neural network (ResNet) architecture (He et al., 2016).
A typical residual block contains two convolutional layers with batch normalization (BN) and uses the ReLU activation function.
The input of the whole block is added to the output of the second convolution (after BN) through a skip connection before the final activation.
Since the convolution operators are the same as we formulate in the second part of Section A.1, here we use the notation Ψ( X ) to denote the sequential operations of convolution on X followed by BN
, i.e., Ψ( X ) ≜
BN ( Conv ( X )) .
Again, we reuse the receptive field mapping Θ( k, i, j ) as defined in
Section A.1 to position the inputs of the residual block corresponding to the element Z k,i,j in the output representation of the whole residual block.
Let X denote the input of the residual block and Z k,i,j denote an element in the output tensor of the whole residual block.
Then we have:
[Equation]
where f is the activation function (ReLU).
We perform channel-wise fusion on the representation from the residual block to produce H k for the k -th channel:
[Equation]
where d H and d W are the dimensions of the feature map and k is the channel index.
Let ζ ( µ k,i,j , σ 2 k,i,j ) denote the distribution that Z k,i,j follows.
Then we apply the Lyapunov’s condition to the representation elements layer-wise, i.e.,
[Equation]
where s k = (cid:113)(cid:80)
d H i =1 (cid:80)
[Equation]
With the above condition satisfied, by CLT H k (the fused representation on channel k ) tends to follow the Gaussian distribution:
[Equation]
which proves the Proposition 2 for channel-wise fused representations from any residual block.
B P SEUDO - CODE OF F ED P ROF ( DETAILED VERSION )
See Algorithm 2 for a complete version of the proposed training algorithm.
C C ONVERGENCE A NALYSIS
In this section we provide the proof of the proposed Theorem 1.
The analysis is mainly based on the results provided by Li et al.
We first introduce several notations to facilitate the analysis.
C. N OTATIONS
Let U ( | U | = N ) denote the full set of clients and S ( t ) ( | S ( t ) | = K ) denote the set of clients selected for participating.
By w k ( t ) we denote the local model on client k
at time step t .
We define an auxiliary sequence v k ( t ) for each client to represent the immediate local model after a local SGD update.
Note that v k ( t ) is updated from v k ( t − 1) with learning rate η t − 1 :
[Equation]
where ∇ F k ( w k ( t − 1) , ξ k,t − 1 ) is the stochastic gradient computed over a batch of data ξ k,t − 1 drawn from D k with regard to w k ( t − 1) .
We also define two virtual sequences ¯ v ( t ) =
(cid:80) Nk
=1 ρ k v k
( t ) and ¯ w ( t ) =
Aggregate ( { v k ( t ) } k ∈ S ( t ) ) for every time step t (Note that the actual global model w ( t ) is only updated at the aggregation steps
T A =
{ E, 2 E, 3 E, . . . } ).
Given an aggregation interval E ≥ 1 , we provide the analysis for the partial aggregation rule that yields ¯ w ( t ) as:
[Equation]
[Equation]
where S ( t ) ( | S ( t ) | =
K ) is the selected set of clients for the round ⌈ tE
that contains step t .
At the aggregation steps T A , w ( t ) is equal to ¯ w ( t ) , i.e., w ( t ) =
¯ w ( t ) if t ∈ T A .
To facilitate the analysis, we assume each client always performs model update (and synchronization) to produce v k ( t ) and ¯ v ( t ) (but obviously it does not affect the resulting ¯ w and w for k / ∈ S ( t ) ).
[Equation]
w ( t ) , if t ∈ T A (29) k
6 Generate
∗ 7 K ← | U | · C
T do
S ←
Distribute to
S do v k
S v
w k ← localTraining
E end
Collect
S
Update w aggregation v
Evaluate w and generate
∗ end 19 return w // updateProfile
Generate k k w received
Label profile RP k with version number v
Return RP k returnlocalTraining( k, w, E ): w k ← w for step e ← 1 to E do
Update w k using gradient-based method end
Return w k return
For ease of presentation, we also define two virtual gradient sequences:
[Equation]
and g ( t ) =
(cid:80) Nk
=
ρ k ∇ F k ( w k ( t ) , ξ k,t ) .
Thus we have E[ g ( t )]
= ¯ g ( t ) and ¯ v ( t ) =
¯ w ( t − 1) − η t − 1 g ( t − 1) .
C. A SSUMPTIONS
We formally make four assumptions to support our analysis of convergence.
Assumptions 1 and 2 are standard in the literature (Li et al., 2019; Wang et al., 2019; Cho et al., 2020) defining the convexity and smoothness properties of the objective functions.
Assumptions 3 and 4 bound the variance of the local stochastic gradients and their squared norms in expectation, respectively.
These two assumptions are also made in by Li et al.
Assumption 1.
F 1 , F 2 , . .
. , F N are L -smooth, i.e., for any k ∈ U , x and y : F k ( y ) ≤ F k ( x )
+ ( y − x ) T ∇ F k ( x ) + L 2 ∥ y − x ∥
It is obvious that the global objective F is also L -smooth as a linear combination of F 1 , F 2 , . .
, F N with ρ 1 , ρ 2 , . . .
, ρ N being the weights.
Assumption 2.
F 1 , F 2 , . .
. , F N are µ -strongly convex, i.e., for all k ∈ U and any x , y : F k ( y ) ≥
[Equation]
Assumption 3.
The variance of local stochastic gradients on each device is bounded: For all k ∈ U ,
[Equation]
Assumption 4.
The squared norm of local stochastic gradients on each device is bounded: For all k ∈ U , E ∥∇ F
[Equation]
To facilitate the proof of our main theorem, we first present several key lemmas.
Lemma 1 (Result of one SGD step) .
Under Assumptions 1 and 2 and with η t < 14 L , for any t it holds true that
[Equation]
where Γ = F ∗ − (cid:80) Nk
=
ρ k F ∗ k .
Lemma 2 (Gradient variance bound) .
Under Assumption 3, one can derive that
[Equation]
Lemma 3 (Bounded divergence of w k ( t ) ) .
Assume Assumption 4 holds and a non-increasing step size η
[Equation]
η t ≤ 2 η t + E for all t = 1 , 2 , . .
, it follows that
[Equation]
Lemmas 1, 2 and 3 hold for both full and partial participation and are independent of the client selection strategy.
We refer the readers to Li et al.
for their proofs and focus our analysis on opportunistic selection.
The next two lemmas give important properties of the aggregated model ¯ w as a result of partial participation and non-uniform client selection/sampling.
Lemma 4 (Unbiased aggregation) .
For any aggregation step t ∈ T A and with q k = ρ k in the selection of S ( t ) , it follows that
[Equation]
Proof.
First, we present a key observation given by Li et al.
as an important trick to handle the randomness caused by client selection with probability distribution { q k } Nk =1 .
By taking the expectation over S ( t ) , it follows that
[Equation]
Let q k
= ρ k , take the expectation of ¯ w ( t ) over S ( t ) and notice that ¯ v ( t
[Equation]
= (cid:80)
[Equation]
Lemma 5 (Bounded variance of ¯ w ( t ) ) .
For any aggregation step t ∈ T A and with a non-increasing step size η t s.t.
η t ≤ 2 η t + E − 1 , it follows that
[Equation]
Proof.
First, one can prove that v k ( t ) is an unbiased estimate of ¯ v ( t ) for any k :
[Equation]
Then by the aggregation rule ¯ w ( t ) = 1 K (cid:80) k ∈ S ( t ) v k ( t ) , we have:
[Equation]
where the second term on the RHS of (37) equals zero because { v k ( t ) } k ∈ U are independent and unbiased (see Eq. 36).
Further, by noticing t − E ∈ T A (because t ∈ T A ) which implies that w k ( t − E ) =
¯ w ( t − E ) since the last communication, we have:
[Equation] [Equation]
where the last inequality results from E[ v k ( t ) − ¯ w ( t − E )]
= ¯ v ( t ) − ¯ w ( t − E ) and that E ∥
X − E X ∥ 2 ≤
E ∥
X ∥ 2 .
Further, we have:
[Equation]
Let i m =
arg max i ∥∇ F k (cid:0)
w k
( i ) , ξ k,i (cid:1) ∥ , i ∈
[ t − E, t − 1] .
By using the Cauchy-Schwarz inequality,
Assumption 4 and choosing a non-increasing η t s.t.
η t ≤ 2 η t + E − 1 , we have:
[Equation]
Plug Z 1 back into (39) and notice that (cid:80) Nk
=
1 q k = 1 , we have:
[Equation]
C. P ROOF OF T HEOREM
Proof.
Taking expectation of ∥ ¯ w ( t ) − w ∗ ∥ 2 , we have:
E
[Equation] [Equation]
t
[Equation]
¯
[Equation]
t )
[Equation]
∥
[Equation]
E ∥
¯
[Equation]
( t
[Equation]
¯ v ( t
[Equation]
(cid:123)(cid:122) (cid:125)
[Equation]
E ∥
¯ v ( t )
[Equation]
(cid:123)(cid:122) (cid:125)
[Equation]
+ E ⟨ ¯
[Equation]
( t )
[Equation]
v ( t )
[Equation]
¯ v ( t )
[Equation]
w
[Equation]
⟩ (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
[Equation]
where A 3 vanishes because ¯ w ( t ) is an unbiased estimate of ¯ v ( t ) by first taking expectation over S ( t ) (Lemma 4).
To bound A 2 for t ∈ T A , we apply Lemma 1:
[Equation]
Then we use Lemmas 2 and 3 to bound B 1 and B 2 respectively, which yields:
[Equation]
where B = (cid:80) Nk
=
ρ 2 k
ϵ 2 k + 6 L Γ + 8( E − 1) 2 G 2 .
To bound A 1 , one can first take expectation over S ( t ) and apply Lemma 5 where the upper bound actually eliminates both sources of randomness.
Thus, it follows that
[Equation]
Let C
= 4 K E 2 G 2 and plug A 1 and A 2 back into (41):
[Equation]
[Equation]
Equivalently, let ∆ t = E ∥
¯ w ( t ) − w ∗ ∥ 2 , then we have the following recurrence relation for any t ≥ 1 :
[Equation] [Equation]
Next we prove by induction that ∆ t ≤ νγ + t where ν = max (cid:110) β 2 ( B + C ) βµ − 1 , ( γ + 1)∆ 1 (cid:111) using an aggregation interval E ≥ 1 and a diminishing step size η t = βt + γ for some β > 1 µ and γ > 0 such that η 1 ≤ min { 1 µ , 14 L } and η t ≤ 2 η t + E .
First, for t = 1 the conclusion holds that ∆ 1 ≤ νγ +1 given the conditions.
Then by assuming it holds for some t , one can derive from (46) that
[Equation]
which proves the conclusion ∆ t ≤ νγ + t for any t ≥ 1 .
Then by the smoothness of the objective function F , it follows that
[Equation] [Equation]
Specifically, by choosing β = 2 µ (i.e., η t =
[Equation]
+
[Equation]
), γ =
[Equation]
{ 8 Lµ , E } − 1 , we have
[Equation]
By definition, we have w ( t ) =
¯ w ( t ) at the aggregation steps.
Therefore, for t ∈ T A :
[Equation]
D P ROFILE D ISSIMILARITY UNDER H OMOMORPHIC E NCRYPTION
The proposed representation profiling scheme encodes the representations of data into a list of distribution parameters, namely RP ( w , D ) = { ( µ i , σ 2 i ) | i = 1 , 2 , . . . , q } where q is the length of the profile.
Theoretically, the information leakage (in terms of the data in D ) by exposing RP ( w , D ) is very limited and it is basically impossible to reconstruct the samples in D given RP ( w , D ) .
Nonetheless, Homomorphic Encryption (HE) can be applied to the profiles (both locally and on the server) so as to guarantee zero knowledge disclosure while still allowing profile matching under the encryption.
In the following we give details on how to encrypt a representation profile and compute profile dissimilarity under Homomorphic Encryption (HE).
To calculate (3) and (4) under encryption, a client needs to encrypt (denoted as [[ · ]] ) every single µ i and σ 2 i in its profile RP ( w , D ) locally before upload whereas the server does the same for its
RP ∗ ( w , D ∗ ) .
Therefore, according to Eq.
(4) we have:
[Equation]
where the first two terms on the right-hand side require logarithm operation on the ciphertext.
However, this may not be very practical because most HE schemes are designed for basic arithmetic operations on the ciphertext.
Thus we also consider the situation where HE scheme at hand only provides additive and multiplicative homomorphisms (Gentry, 2009).
In this case, to avoid the logarithm operation, the client needs to keep every σ 2 i in RP ∗ ( w , D i ) as plaintext and only encrypts µ i , likewise for the server.
As a result, the KL divergence can be computed under encryption as:
[Equation] [Equation]
where the first term on the right-hand side is encrypted after calculation with plaintext values ( σ ki )and ( σ ∗ ) 2 whereas the second term requires multiple operations on the ciphertext values [[ µ ki ]] and [[ µ ∗ ]] .
Now, in either case, we can compute profile dissimilarity under encryption by summing up all the
KL divergence values in ciphertext:
[Equation]
KL( N ( k ) i ||N ∗ i )]] (52)
E D ETAILS OF E XPERIMENTAL SETUP E. E NVIRONMENT S ETUP
[Table]
Detailed experimental settings are listed in Table 4.
In the S-Task, the total population is 50 and the data collected by a proportion of the sensors (i.e., end devices of this task) are of low-quality: 10% of the sensors have no valid data and 40% of them produce noisy data.
In the L-Task, we set up a relatively large population (1000 end devices) and spread the data (from EMNIST digits ) across the devices with strong class imbalance – roughly 60% of the samples on each device fall into the same class.
Besides, many local datasets are of low-quality: the images on 15% of the clients are irrelevant (valueless for the training of this task), 20% are (Gaussian) blurred
[Table]
and 25% are affected by the salt-and-pepper noise (random black and white dots on the image, density=0.3).
For the STask, the maximum number of rounds t max is set to 100 for both aggregation modes, whilst it is set to 300 and 50 for the full aggregation and partial aggregation, respectively, for the L-Task.
The preference factor α for our protocol is set to 10.
Considering the population of the clients, the setting of the selection fraction C is based on the scale of the training participants suggested by Kairouz et al.
To simulate a realistic FL system that consists of disparate end devices, the clients are heterogeneous in terms of both performance and communication bandwidth (see Table 4).
A validation set is kept by the server (as the benchmark data) and used for model evaluation.
E. D ETAILS OF C OST F ORMULATION
In each FL round, the server selects a fraction (i.e., C ) of clients, distributes the global model to these clients and waits for them to finish the local training and upload the models.
Given a selected set of clients S , the time cost and energy cost of a communication round can be formulated as:
[Equation] [Equation]
(54) where T commk and T traink are the communication time and local training time, respectively.
The device-side energy consumption E k mainly comes from model transmission (through wireless channels) and local processing (training), corresponding to E commk and E traink , respectively.
T RPk and
E RPk estimate the time and energy costs for generating and uploading local profiles and only apply to F ED P ROF .
Eq.
(53) formulates the length of one communication round of FL, where T commk can be modeled by Eq.
(55) according to Tran et al.
(2019), where bw k is the downlink bandwidth of device
k (in
MHz)
; SNR is the Signal-to-Noise Ratio of the communication channel, which is set to be constant as in general the end devices are coordinated by the base stations for balanced SNR with the fairnessbased policies; msize is the size of the model; the model upload time is twice as much as that for model download since the uplink bandwidth is set as 50% of the downlink bandwidth.
[Equation]
T traink in Eq.
(53) can be modeled by Eq.
(56), where s k is the device performance (in GHz) and the numerator computes the total number of processor cycles required for processing M epochs of local training on D k .
[Equation]
T RPk consists of two parts: T RPgenk for local model evaluation (to generate the profiles of D k ) and
T RPupk for uploading the profile.
T RPk can be modeled as:
[Equation]
where T RPgenk is estimated as the time cost of one epoch of local training; T RPupk is computed in a similar way to the calculation of T commk in Eq.
(55) (where the uplink bandwidth is set as one half of the total bw k ); RPsize is the size of a profile, which is equal to 4 × 2 × q = 8 × q (four bytes for each floating point number) according to our definition of profile in (2).
Using Eq.
(54) we model the energy cost of each end device by mainly considering the energy consumption of the transmitters for communication (Eq. 58) and on-device computation for local training (Eq. 59).
For F ED P ROF , there is an extra energy cost for generating and uploading profiles (Eq. 60).
[Equation] [Equation]
where P f s 3 k is a simplified computation power consumption model (Song et al., 2013) and P f is the power of a baseline processor.
P trans is the transmitter’s power.
We set P trans and P f to 0.5 and 0.7 Watts respectively based on the benchmarking data provided by Carroll et al.