F EDERATED L EARNING VIA P LURALITY V OTE
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning allows collaborative workers to solve a machine learning problem while preserving data privacy.
Recent studies have tackled various challenges in federated learning, but the joint optimization of communication overhead, learning reliability, and deployment efﬁciency is still an open problem.
To this end, we propose a new scheme named federated learning via plurality vote (FedVote).
In each communication round of FedVote, workers transmit binary or ternary weights to the server with low communication overhead.
The model parameters are aggregated via weighted voting to enhance the resilience against Byzantine attacks.
When deployed for inference, the model with binary or ternary weights is resource-friendly to edge devices.
We show that our proposed method can reduce quantization error and converges faster compared to the methods directly quantizing the model updates.
I NTRODUCTION
Federated learning enables multiple workers to solve a machine learning problem under the coordination of a central server (Kairouz et al., 2021).
Throughout the training stage, client data will be kept locally and only model weights or model updates will be shared with the server.
Federated averaging (FedAvg) (McMahan et al., 2017) was proposed as a generic federated learning solution.
Although FedAvg takes advantage of distributed client data while maintaining their privacy, it leaves the following two challenges unsolved.
First, transmitting high-dimensional messages between a client and the server for multiple rounds can incur signiﬁcant communication overhead.
Quantization has been incorporated into federated learning in recent studies (Reisizadeh et al., 2020;
Haddadpour et al., 2021).
However, directly quantizing the gradient vector may not provide the optimal trade-off between communication efﬁciency and model accuracy 1 .
Second, the aggregation rule in FedAvg is vulnerable to Byzantine attacks (Blanchard et al., 2017).
Prior works tackled this issue by using robust statistics such as coordinate-wise median and geometric median in the aggregation step (Blanchard et al., 2017; Yin et al., 2018).
Another strategy is to detect and reject updates from malicious attackers (Mu ˜ noz-Gonz ´ alez et al., 2019; Sattler et al., 2020).
The robustness of the algorithm is enhanced at the cost of additional computation and increased complexity of algorithms.
In this paper, we propose a new method called federated learning via plurality vote (FedVote).
We train a neural network at each worker with a range normalization function applied to model parameters.
After local updating, binary/ternary weight vectors are obtained via stochastic rounding and sent to the server.
The global model is updated by a voting procedure, and the voting results are sent back to each worker for further optimization in the next round.
The contributions of the paper are summarized as follows.
We present FedVote as a novel federated learning solution to jointly optimize the communication overhead, learning reliability, and deployment efﬁciency.
We theoretically and experimentally verify the effectiveness of our FedVote design.
In bandwidth-limited scenarios, FedVote is particularly advantageous in simultaneously achieving a high compression ratio and good test accuracy.
Given a ﬁxed communication cost, FedVote improves model accuracy on the CIFAR-10 dataset by 5 – 10% , 15 – 20% , and 25 – 30% compared with FedPAQ (Reisizadeh et al., 2020), signSGD
(Bernstein et al., 2018), and FedAvg, respectively.
We extend FedVote to incorporate reputation-based voting.
The proposed method, Byzantine-FedVote, exhibits much better resilience to Byzantine attacks in the presence of close to half attackers without incurring excessive computation compared with existing algorithms.
R ELATED W ORK
Communication-Efﬁcient Federated Learning.
In the prior study of federated learning, various strategies have been proposed to reduce communication cost.
One research direction is to reduce the size of messages in each round.
For example, Bernstein et al.
(2018) showed that sign-based gradient descent schemes can converge well in the homogeneous data distribution scenario, while Chen et al.
(2020); Jin et al.
; Safaryan & Richtarik (2021) extended it to the heterogeneous data distribution setting.
In parallel, FedAvg adopts a periodic averaging scheme and targets at reducing the number of communication rounds (McMahan et al., 2017).
Hybrid methods consider simultaneous local updates and accumulative gradient compression (Reisizadeh et al., 2020; Haddadpour et al., 2021).
In this work, we improve communication efﬁciency by employing binary/ternary weights in the neural network.
Quantized Neural Networks.
Quantized neural networks aim to approximate the full-precision networks using quantized weights while keeping their generalizability.
As a special case, binary neural networks (BNNs) are gaining popularity in recent years.
By restricting model weight values to {− 1 , +1 } , BNNs can reduce computational cost, memory requirement, and energy consumption.
Hubara et al.
(2016) introduced real-valued latent weights and used the sign operator for binarization.
In contrast, Shayer et al.
(2018) let the neural network learn the distribution of the binary or ternary weights.
Gong et al.
(2019) added a soft quantization function to the real-valued weights, thus avoiding the gradient mismatch between the forward and the backward passes.
A more thorough survey on BNN optimization can be found in Qin et al.
(2020a).
Distributed Optimization of Quantized Neural Networks
A few recent works have explored
BNN optimization in a distributed setting, which is more relevant to our work.
Lin et al.
(2020) conducted the case study of 1-bit quantized local models aggregated via ensemble distillation.
The aggregation becomes complicated due to the separate optimization stage of knowledge distillation, especially when the distillation algorithm does not converge well in practice.
In addition, their
BNN optimization is not tailored to the federated learning setting.
Hou et al.
(2019) theoretically analyzed the convergence of distributed quantized neural networks, which was later implemented in the application of intrusion detection (Qin et al., 2020b).
In comparison, FedVote is presented as a new algorithm by leveraging client local updates to accelerate the training.
Different from Hou et al.
(2019), we do not assume a convex and twice differentiable objective function and bounded gradients.
Therefore, the analyses in Hou et al.
(2019) cannot be directly applied to our study.
Byzantine Resilience in Federated Learning.
An attack is Byzantine if arbitrary outputs are produced due to the adversary (Kairouz et al., 2021).
Blanchard et al.
(2017) showed that FedAvg cannot tolerate a single Byzantine attacker.
They proposed an aggregation-rule-based remedy using the similarity of local updates.
Similarly, Yin et al.
(2018) took the advantage of coordinatewise median and trimmed-mean to robustify the aggregation.
Mu˜noz-Gonz´alez et al.
(2019) and Sattler et al.
(2020) detected the adversaries and ﬁltered out their model updates.
In this paper, we propose a reputation-based voting strategy for FedVote that is shown to have good convergence performance in the Byzantine setting.
P RELIMINARIES
Symbol conventions are as follows.
Bold lower cases of letters such as v m denote column vectors, and v m,i is used to denote its i th entry.
For a scalar function, it applies elementwise operation when a vector input is given.
1 =
, 1] (cid:62) denotes a vector with all entries equal to 1 .
3. F EDERATED L EARNING
The goal of federated learning is to build a machine learning model based on the training data distributed among multiple workers.
To facilitate the learning procedure, a server will coordinate the training without seeing the raw data (Kairouz et al., 2021).
In a supervised learning scenario, let
D m = { ( x m,j , y m,j ) } n m j =1 denote the training dataset on the m th worker, with the input x m,j ∈ R dand the label y m,j ∈ R d 2 in each training pair.
The local objective function f m with a model weight vector θ ∈ R d is given by
[Equation]
where (cid:96) is a loss function quantifying the error of model θ predicting the label y m,j for an input x m,j .
A global objective function may be formulated as
[Equation]
3. Q UANTIZED N EURAL N ETWORKS
Consider a neural network g with the weight vector θ ∈ R d .
A forward pass for an input x ∈ R dand a prediction ˆ
y ∈ R d 2 can be written as ˆ y =
g ( θ , x ) .
In quantized neural networks, the realvalued θ is replaced by w ∈ D dn , where D n is a discrete set with a number n of quantization levels.
For example, we may have D 2 = {− 1 , 1 } for a binary neural network.
For a given training set { ( x j , y j ) } Nj =1 and the loss function (cid:96) , the goal is to ﬁnd an optimal w ∗ such that the averaged loss is minimized over a search space of quantized weight vectors:
[Equation]
Prior studies tried to solve (3) by optimizing a real-valued latent weight vector h ∈ R d (Hubara et al., 2016; Shayer et al., 2018;
Gong et al., 2019).
The interpretations of the latent weight vary when viewed from different perspectives.
Hubara et al.
(2016) used the sign operation to quantize the latent weight into two levels during the forward pass.
The binary weights can be viewed as an approximation of their latent real-valued counterparts.
Shayer et al.
(2018) trained a stochastic binary neural network, and the normalized latent parameters are interpreted as the Bernoulli distribution parameter ϑ i :
[Equation]
where S : R → (0 , 1) is the sigmoid function.
In the forward pass, instead of using the binary vector w , its expected value,
[Equation]
will participate in the actual convolution or matrix multiplication operations.
In other words, the neural network function becomes ˆ y =
g
( (cid:101) w sto-bnn , x ) .
Likewise, Gong et al.
(2019) normalized the latent weight but interpreted it from a different viewpoint.
They approximated the staircase quantization function with a differentiable soft quantization (DSQ) function, i.e.,
[Equation]
where tanh : R → ( − 1 , 1) is the hyperbolic tangent function, and a controls the shape of the function.
The neural network function thus becomes ˆ y =
g ( (cid:101) w dsq , x ) .
We now summarize the latent-weight-based BNN training methods and depict an example of a single-layer network in Figure 1.
First, a real-valued vector h ∈ R d is introduced and its range is restricted using a differentiable and invertible normalization function ϕ : R → ( − 1 , 1) .
The forward pass is then calculated with the normalized weight vector (cid:101) w .
The procedure is described as:
[Equation]
Second, in the back propagation, the latent weight vector h is updated with its gradient, i.e., h ( t +1) =
h ( t ) − η ∇ h (cid:96) .
Finally, the normalized weight vector (cid:101) w are mapped to the discrete space to approximate w ∗ via thresholding or stochastic rounding.
[Figure]
is normalized to generate (cid:101)
w ( − 1 , 1) d and the output is ˆ y =
σ ( (cid:101) w (cid:62) x ) .
A binary weight w q can be obtained by thresholding or rounding (cid:101) w to the discrete space D d 2 .
[Figure]
P ROPOSED F ED V OTE A LGORITHM
In this section, we present our proposed method with an emphasis on uplink communication efﬁciency and enhanced Byzantine resilience.
We follow the widely adopted analysis framework in wireless communication to investigate only the worker uplink overhead, assuming that the downlink bandwidth is much larger and the server will have enough transmission power (Tran et al., 2019).
To reduce the message size per round, we train a quantized neural network under the federated learning framework.
The goal is to ﬁnd a quantized weight vector w ∗ that minimizes the global objective function f formulated in (2), i.e.,
[Equation] [Equation]
For the simplicity of presentation, we mainly focus on the BNN case with D 2 = {− 1 , 1 } .
We illustrate the procedure in Figure 2 and provide the pseudo code in Appendix B. Below, we explain each step in more detail.
4. L OCAL M ODEL T RAINING AND T RANSMISSION
We optimize a neural network with a learnable latent weight vector h .
In the k th communication round, we assume all workers are identically initialized by the server, namely, ∀ m ∈ { 1 , . . .
, M } , h ( k, 0) m
= h ( k ) .
To reduce the total number of communication rounds, we ﬁrst let each worker conduct local updates to learn the binary weights.
For each local iteration step, the local latent weight vector h ( k,t +1) is updated by the gradient descent:
[Equation]
where ξ ( k,t ) m is a mini-batch randomly drawn from D m at the t th iteration of round k .
After updating for τ steps, we obtain h ( k,τ ) m and the corresponding normalized weight vector (cid:101) w ( k,τ ) m ∈ ( − 1 , 1) d deﬁned as follows:
[Equation]
To reduce the message size, we use the stochastic rounding to draw a randomly quantized version w ( k,τ ) m using (cid:101) w ( k,τ ) m , namely, w ( k,τ ) m,i = (cid:40) +1 , with probability π
( k,τ ) m,i = 1 2 (cid:104) (cid:101) w ( k,τ ) m,i + 1 (cid:105) , − 1 , with probability 1 − π ( k,τ ) m,i .
[Equation]
It can be shown that the stochastic rounding is an unbiased procedure, i.e., E [ w ( k,τ )
m (cid:12)(cid:12)(cid:101) w ( k,τ ) m ] = (cid:101)
w ( k,τ ) m . After quantization, the local worker will send the binary weights w ( k,τ ) m to the server for the global model aggregation.
4. G LOBAL M ODEL A GGREGATION AND B ROADCAST
Once the server gathers the binary weights from all workers, it will perform the aggregation via plurality vote, i.e., w ( k +1)
= sign (cid:16)(cid:80)
Mm =1 w ( k,τ ) m (cid:17) .
A tie in vote will be broken randomly.
In the following lemma, we show that the probability of error reduces exponentially as the number of workers increases.
The proof can be found in Appendix D.1.
Lemma 1 (One-Shot FedVote) Let w ∗ ∈
D
d 2 be the optimal solution deﬁned in (8).
For the m th worker, ε m,i (cid:44) P ( w ( k,τ ) m,i (cid:54) =
w ∗ i ) denotes the error probability of the voting result of the i th coordinate.
Suppose the error events { w ( k,τ ) m,i (cid:54) =
w ∗ i } Mm =1 are mutually independent, and the mean error probability s i = 1 M (cid:80) Mm =
ε m,i is smaller than 12 . For the voted weight w ( k +1) , we have
[Equation]
In practice, the number of available workers may be limited in each round, and the local data distribution is often heterogenous or even time-variant.
Therefore, it is almost always desirable to execute
FedVote in a multiple-round fashion.
In this case, we ﬁrst use the soft voting to build an empirical distribution of global weight w , i.e., [Equation]
[Equation]
where 1 ( · ) ∈ { 0 , 1 } is the indicator function.
Let p ( k +1)
i (cid:44) (cid:98) P ( w ( k +1) i = 1) and p ( k +1) =
[ p ( k +1), . . .
, p ( k +1)
d ] (cid:62) .
The global latent parameters can be constructed by following (10):
[Equation]
(14) where ϕ − 1 : ( − 1 , 1) → R is the inverse of the normalization function ϕ .
We further apply clipping to restrict the range of the probability, namely, clip( p ( k +1) i )
=
max( p min , min( p max , p ( k +1) i )) , where p min , p max ∈ (0 , 1) are predeﬁned thresholds.
To keep the notation consistent, we denote (cid:101) w ( k +1) (cid:44) ϕ
( h ( k +1) )
as the global normalized weight.
After broadcasting the soft voting results p ( k +1) , all workers are synchronized with the same latent weight h ( k +1) and normalized weight (cid:101) w ( k +1) .
The learning procedure will repeat until a termination condition is satisﬁed.
We relate
FedVote to FedAvg in the following lemma.
The detailed proof can be found in Appendix D.2.
Lemma 2 (Relationship with FedAvg) For the normalized weights, FedVote recovers FedAvg in expectation: E (cid:2)(cid:101) w ( k +1) (cid:3)
= 1 M M (cid:80) m
=1 (cid:101) w ( k,τ ) m , where (cid:101) w ( k +1) =
ϕ ( h ( k +1) ) and (cid:101) w ( k,τ ) m =
ϕ ( h ( k,τ ) m ) .
4. R EPUTATION -B ASED B YZANTINE -F ED V OTE
Lemma 2 shows that FedVote is related to FedAvg in expectation.
As we have reviewed in Section 2, FedAvg cannot tolerate a single Byzantine attacker.
It indicates that FedVote will exhibit similar poor performance in the presence of multiple adversaries (see Appendix C.2).
We improve the design of FedVote based on a reputation voting mechanism, which in essence is a variant of the weighted soft voting method.
Reputation-based voting was presented in failure-robust large scale grids (Bendahmane et al., 2014).
In our design, we modify (13) to (cid:98) P ( w ( k +1) i = 1) = (cid:80) Mm
[Equation] 1 λ ( [Equation] ) m 1 (cid:16) w ( k,τ ) m,i = 1 (cid:17) , where λ ( k ) m is proportional to a credibility score.
In Byzantine-resilient FedVote (Byzantine-FedVote), we assume that at least 50% of the workers behave normally and treat the plurality vote result as the correct decision.
The credibility score of the m th worker is calculated by counting the number of correct votes it makes: CR ( k +1) m
[Equation]
(cid:17) .
Through multiple rounds, we track the credibility of a local worker by taking an exponential moving average over the communication rounds, namely, ν ( k +1) m
[Equation]
, where β ∈ (0 , 1) is a predeﬁned coefﬁcient.
The weight λ ( k ) m is designed as λ ( k ) m =
ν ( k ) m / (cid:80)
Mm =1 ν ( k ) m .
Even though the presentation in this section focuses on the binary weights, the scheme can be naturally extended to quantized neural networks of more discrete levels.
We will brieﬂy discuss the implementation for ternary weights in Section 6.
A NALYSIS OF A LGORITHM
In this section, we present the theoretical analysis of FedVote when local data are independent and identically distributed (i.i.d.).
The empirical results in the non-i.i.d.
setting will be discussed in Section 6, and the corresponding analyses are left for future work.
We use the gradient norm expectation as an indicator of convergence, which is commonly adopted in nonconvex optimization literature (Reisizadeh et al., 2020; Haddadpour et al., 2021).
To simplify the notation, we ﬁrst denote the stochastic local gradient by ˜ g
( k,t ) m (cid:44) ∇ h f m ( ϕ ( h ( k,t ) m ); ξ ( k,t ) m ) .
The local true gradient and global true gradient will be denoted by g ( k,t ) m
[Equation]
(cid:44) ∇ h f ( ϕ ( h ( k ) )) , respectively.
In addition, let ζ ( k ) m
(cid:44) (cid:101) w ( k,τ ) m − w ( k,τ ) m denote the error introduced by stochastic rounding.
According to the unbiased property of stochastic rounding, we have E π
[ ζ ( k ) m ]
= 0 .
With the aforementioned notations, we state ﬁve assumptions for the convergence analysis.
5. A SSUMPTIONS
Assumption 1 (Lower bound) ∀ h ∈ R d , (cid:101) w ∈ ( − 1 , 1) d , the objective function is lower bounded by a constant f ∗ , i.e., f ( (cid:101) w ) (cid:62) f ∗ = min h ∈ R d f ( ϕ ( h )) .
Assumption 2 ( L -smoothness) ∀ (cid:101) w 1 , (cid:101) w 2 ∈ ( − 1 , 1) d , m ∈ { 1 , . .
, M } , there exists some nonnegative L
such that
(cid:107)∇
[Equation]
Assumptions 1 to 2 are common for necessary analyses (Wang & Joshi, 2018).
We limit the range of the normalized weight (cid:101) w while in a typical setting there is no restriction to the model weight.
Assumption 3 The normalization function ϕ : R → ( − 1 , 1) is strictly increasing.
In particular, we assume its ﬁrst derivative is bounded for all h ( k,t ) m,i , i.e., dd h ϕ ( h ( k,t ) m,i ) ∈
[ c 1 , c 2 ] , where c 1 , c 2 are positive parameters independent of k, t, m, and i .
Assumption 3 is not difﬁcult to satisfy in practice.
For example, let ϕ ( h ) =
tanh( ah ) , we have ϕ (cid:48)
( h ) =
a (cid:2) 1 − tanh 2 ( ah ) (cid:3) , with c 2 = a .
Note that ϕ quickly saturates with a large h in the local updating.
On the other hand, the empirical Bernoulli parameter p i will be clipped for stability, which indicates that h ( k,t ) m,i will be upper bounded by certain h B .
In this sense, we have c 1 = a (cid:2) 1 − tanh 2 ( ah B ) (cid:3) .
The next two assumptions bound the variance of the stochastic gradient and quantization noise.
Assumption 4 The stochastic gradient has bounded variance, i.e., E (cid:13)(cid:13) g ( k,t ) m − ˜ g
( k,t ) m
(cid:13)(cid:13) 2 2 (cid:54) σ 2 ε , where σ 2 ε is a ﬁxed variance independent of k , t and m .
Assumption 5 The quantization error ζ ( k ) m has bounded variance, i.e., E (cid:13)(cid:13) ζ ( k ) m (cid:13) (cid:13) 2 2 (cid:54) σ 2 k , where σ 2 k is a ﬁxed variance independent of m .
Note that quantization error is affected by the quantizer type and the corresponding input.
For example, if the normalization function ϕ approximates the sign function very well, the stochastic quantization error will be close to zero.
Formally, the upper bound σ 2 ζ can be viewed as a function of input dimension d , which we formulate in the following lemma.
The proof is in Appendix D.3.
Lemma 3 Suppose we have an input a ∈ ( − 1 , 1) d for the quantizer Q sr deﬁned in (11), then the quantization error satisﬁes
[Equation]
For existing algorithms quantizing the model update δ ( k ) m
(cid:44) θ ( k ) − θ ( k,τ ) m , the quantizer has the property E (cid:104)(cid:13)(cid:13) Q ( x ) − x (cid:13) (cid:13) 2 2 (cid:12)(cid:12) x
(cid:105) (cid:54) q (cid:13) (cid:13) x
(cid:13)(cid:13) 2 2 .
With a ﬁxed quantization step, q increases when the input dimension d increases (Basu et al., 2020).
We state the result for a widely-used quantizer, QSGD (Alistarh et al., 2017), which has been adopted in FedPAQ, in the following lemma.
Lemma 4 Suppose we have an input x ∈ R d for the QSGD quantizer Q .
In the coarse quantization scenario, the quantization error satisﬁes E (cid:104)(cid:13)(cid:13) Q ( x ) − x (cid:13)(cid:13) 2 2 (cid:12)(cid:12) x (cid:105) =
O (cid:16) d 1 2 (cid:17) (cid:13)(cid:13) x (cid:13)(cid:13) 2 2 .
5. C ONVERGENCE A NALYSIS
We state the convergence results in the following theorem.
The proof can be found in Appendix D.5.
Theorem 1 For FedVote under Assumptions 1 to 5, let the learning rate
η = O (cid:16) ( c 1 c 2 )1 Lτ √ K (cid:17) , then after K rounds of communication, we have
[Equation]
where R ( k ) m
(cid:44) − τ − 1 (cid:80) t
= (cid:80) i/ ∈I ( k
[Equation]
Remark 1 When there is no normalization function and quantization, i.e., ϕ ( x ) =
x with c 1 = c 2 = 1 , σ 2 k = 0 , Theorem 1 recovers the result obtained in Wang & Joshi (2018).
[Equation]
[Figure]
To discuss the impact of quantization error, consider the distribution of different inputs.
For the model update δ ( k ) m , we expect the central limit theorem to render its distribution shape, where each entry δ ( k ) m,i follows the Gaussian distribution.
For the Bernoulli probability π ( k,τ ) m , we expect the Beta distribution as the conjugate prior to render its distribution shape, where each entry π ( k,τ ) m,i follows the symmetric Beta distribution.
See Figure 3 for the empirical results.
Remark {+
 Following+} the analysis framework in Theorem 1, the order-wise convergence rate is [Equation]
= O [Equation] 1 √ K ) + E ( d ) , where E ( d ) is the error introduced by the quantization.
For FedVote with the normalized weight (cid:101) w ( k,τ ) m as the input, when π ( k,τ ) m,i ’s follow the symmetric
Beta distribution, it can be shown that E (cid:107) ζ ( k ) m (cid:107)
= O ( d ) based on Lemma 3.
For algorithms such as FedPAQ with the model update δ ( k ) m as the input, when δ ( k ) m,
i ’s follow the Gaussian distribution, it can be shown that E (cid:13)(cid:13) Q ( δ ( k ) m ) − δ
( k ) m (cid:13)(cid:13) 2
= O (cid:0) d 3 / 2 (cid:1)
based on Lemma 4.
When the weight dimension d is sufﬁciently large, FedVote converges faster.
Remark
The value of the positive scalar error term R ( k ) m in (15) depends on the gradient dissimilarity.
If the angle between the local gradient ∇ f m ( (cid:101) w ( k,t ) m ) and the global gradient ∇ f ( (cid:101) w ( k,t ) ) is not large, R ( k ) m can be treated as a bounded variable.
Remark 4 The choice of nonlinear function ϕ : R d → ( − 1 , 1) d will affect the convergence.
If ϕ behaves more like the sign( · ) function, e.g., when a increases in tanh( ax ) , the quantization error will be reduced.
In other words, we expect a smaller σ 2 k according to Lemma 3, which leads to a tighter bound in (15).
Meanwhile, a larger c 2 will negatively inﬂuence the convergence.
E XPERIMENTAL R ESULTS
Data and Models.
We choose image classiﬁcation datasets Fashion-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky, 2009).
Both of them have a total of C = 10 classes.
We consider two data partition strategies: (i) i.i.d.
setting where the whole dataset is randomly shufﬂed and assigned to each worker without overlap; (ii) non-i.i.d.
setting where we follow Hsu et al.
and use the
Dirichlet distribution to simulate the heterogeneity.
In particular, for the m th worker we draw a random vector q m ∼ Dir ( α ) , where q m =
[ q m, 1 , · · · , q m,C ] (cid:62) belongs to the standard ( C − 1) -simplex.
We then assign data samples from different classes to the m th worker following the distribution of [Figure] [Table] [Table]
q m .
We set α = 0 .
5 unless noted otherwise.
We use a LeNet-5 architecture for Fashion-MNIST and a VGG-7 architecture for CIFAR-10.
Results are obtained over three repetitions.
Implementation Details
We provide implementation details in the proposed FedVote design.
First, following prior works (Shayer et al., 2018; Gong et al., 2019), we keep the weights of the
BNN ﬁnal layer as ﬂoating-point values for the sake of the model performance.
The weights of the ﬁnal layer are randomly initialized with a shared seed and will be ﬁxed during the training process.
Second, we notice that for quantized neural networks, the batch normalization (BN) (Ioffe & Szegedy, 2015) after the convolutional layer is necessary to scale the activation.
We use the static BN without learnable parameters and local statistics (Diao et al., 2021) to ensure the voting aggregation of binary weights.
For the normalization function, we choose ϕ ( x ) =
tanh(3 x/ 2) unless noted otherwise.
More details of the experimental setup can be found in Appendix C.1.
Communication Efﬁciency and Convergence Rate.
In this experiment, we consider M =workers with full participation and compare FedVote to different methods within N = 100 communication rounds.
The results of partial participation can be found in Appendix C.3.
The communication cost is calculated as the accumulative uplink message size from all workers.
Figurereveals that FedVote outperforms the gradient-quantization-based methods such as signSGD (Bernstein et al., 2018) that quantizes gradients to 1 bit signs, and FedPAQ (Reisizadeh et al., 2020) that quantizes the updates to 2 bits integers.
Compared with FedPAQ, signSGD, and FedAvg, FedVote improves the test accuracy by 5 – 10% , 15 – 20% , and 25 – 30% , respectively, given the ﬁxed communication costs of 1 . 5 – 4 . 7 GB.
Byzantine Resilience.
This experiment validates the effectiveness of Byzantine-FedVote.
We consider omniscient attackers who can access the datasets of normal workers and send the opposite of the aggregated results to the server.
The number of attackers is 15 , and the remaining 16 clients are normal workers.
We compare the proposed method with coordinate-wise median based (CoMed) gradient descent (Yin et al., 2018) and signSGD
(Bernstein et al., 2018), and report the testing accuracy after 100 communication rounds.
We do not compare with FedAvg and FedPAQ, as they are fragile to Byzantine failure.
The results are shown in Table 1.
It can be observed that ByzantineFedVote exhibits much better resilience to Byzantine attacks with close to half adversaries. [Table] [Table]
Normalization Function.
From Remark [Table] we know that the normalization function can inﬂuence the model convergence.
We empirically examine the impact in this experiment.
For normalization function ϕ ( x ) =
tanh( ax ) [Table] we choose a from { 0 . 5 , 1 . 5 , 2 . 5 , 10 } .
We test the model accuracy after 20 communication rounds on Fashion-MNIST.
The results are shown in Table 2.
As a increases, the linear region of the normalization function shrinks
[Table]
and the algorithm converges slower due to a larger c 2 .
On the other hand, the gap between the model with normalized weight (cid:101) w and the one with binary weight w also decreases due to smaller quantization errors.
A good choice of the normalization function should take this trade-off into consideration.
Ternary Neural Network Extension.
In the previous sections
[Table]
we focus on the BNNs.
We extend FedVote to ternary neural networks (TNNs) and empirically verify its performance.
Training and transmitting the categorical distribution parameters of the ternary weight may bring additional communication and computation cost to edge devices, we therefore simplify the procedure as follows.
For each ternary weight w ( k,t ) m,i , we still keep a latent parameter h
( k,t ) m,i to optimize.
After τ local steps, we use the stochastic rounding to the normalized weight (cid:101) w ( k,τ ) m =
ϕ ( h ( k,τ ) m ) and obtain quantized weight w ( k,τ ) m .
At the aggregation stage on the server, instead of calculating the vote distribution, we directly compute the global normalized weight as (cid:101) w ( k +1) =
1 M (cid:80) Mm =1 w ( k,τ ) m .
More details can be found in Appendix C.4.
The training results are shown in Table 3.
As TNNs can further reduce the quantization error, their performance is better than the BNNs at the cost of additional 1 bit/dimension communication overhead.
Deployment Efﬁciency.
We highlight the advantages of BNNs during deployment on edge devices.
In FedVote, we intend to deploy lightweight quantized neural networks on the workers after the training procedure.
BNNs require 32 × smaller memory size, which can save storage and energy consumption for memory access (Hubara et al., 2016).
As we do not quantize the activations, the advantage of BNNs inference mainly lies in the replacement of multiplications by summations.
Consider the matrix multiplication in a neural network with an input x ∈ R d 1 and output y ∈ R d 2 : y
= W (cid:62) x .
For a ﬂoating-point weight matrix W ∈ R d 1 × d 2 , the number of multiplications is d 1 d 2 , whereas for a binary matrix W b ∈
D
d 1 × d 2all multiplication operations can be replaced by additions.
We investigate the number of real multiplications and additions in the forward pass of different models and present the results in Table 4.
We use the CIFAR-10 dataset and set the batch size to 100 .
As to the energy consumption calculation, we use 3 . 7 pJ and 0 . 9 pJ as in Hubara et al.
(2016) for each ﬂoating-point multiplication and addition, respectively.
C ONCLUSION
In this work, we have proposed FedVote to jointly optimize communication overhead, learning reliability, and deployment efﬁciency.
In FedVote, the server aggregates neural networks with binary/ternary weights via voting.
We have veriﬁed that FedVote can achieve good model accuracy even in coarse quantization settings.
Compared with gradient quantization, model quantization is a more effective design that achieves better trade-offs between communication efﬁciency and model accuracy.
With the voting-based aggregation mechanism, FedVote enjoys the ﬂexibility to incorporate various voting protocols to increase the resilience against Byzantine attacks.
We have demonstrated that Byzantine-FedVote exhibits much better Byzantine resilience in the presence of close to half attackers compared to the existing algorithms.
A V OTING M ETHODS
We review the deﬁnitions of several voting methods (Zhou, 2019).
Suppose we have a set of M individual voters and C candidate output labels { ψ 1 , . . .
, ψ C } .
For the m th voter, it will output result v m =
[ v m, 1 , . . .
, v m,C ] (cid:62) .
Here, the i th entry v m,i ∈ { 0 , 1 } , which takes the value one if the voter chooses the candidate ψ i and zero otherwise.
Majority vote requires the winner to receive more than half of the votes; if none of the candidates receives more than half of the votes, a rejection option will be given.
The majority vote result can be written as
[Equation]
Plurality vote takes the class label that receives the largest number of votes as the ﬁnal winner, i.e.,
[Equation]
In the binary case with C = 2 , plurality vote resembles majority vote except that it does not have a rejection option.
If more than two candidates receive the same number of votes, we randomly select one of them as the ﬁnal results.
Weighted vote assigns different weights for voters and the result can be written as
[Equation]
[Equation]
where λ m is the weight assigned to the m th voter.
In contrast to the aforementioned vote methods, soft vote produces the probability output.
The probability that the i th candidate wins is calculated as
[Equation]
B F ED V OTE AL GORITHM
We summarize the proposed FedVote method and its Byzantine-resilient variant in Algorithm 1.
When there are no attackers involved, we use Option I .
When we require the resilience against
Byzantine failure, we choose Option II .
C S ETUP AND A DDITIONAL E XPERIMENTS C. H YPERPARAMETERS
For the clipping thresholds, we set p min = 0 . 001 and p max = 1 − p min . The thresholds are introduced for numerical stability and have little impact on performance.
We use β = 0 . 5 in Byzantine-FedVote. The choice of the smoothing factor β has little impact on the ﬁnal test accuracy, as the credibility score decays exponentially for adversaries over multiple communication rounds. We use the Adam optimizer and search using the learning rate η over the set { 10 − 4 , 3 × 10 − 4 , 10 − 3 , 3 × 10 − 3 , 10 − 2 , 3 × 10 − 2 , 10 − 1 , 3 × 10 − 1 } .
We set the number of local iterations τ to 40 and the local batch size to 100 . C. C OMPARISON OF V ANILLA F ED V OTE AND B YZANTINE -F ED V OTE
Lemma 2 shows that FedVote is related to FedAvg in expectation. Adversaries sending the opposite results will negatively affect the estimation of the weight distribution and impede the convergence in multiple rounds.
We compare the test accuracy of Byzantine-FedVote, Vanilla FedVote, and signSGD on the non-i.i.d.
CIFAR-10 dataset ( α = 0 . 5 ) with various numbers of omniscient attackers. Figure 6 reveals that the test accuracy of Vanilla FedVote drops severely when the number of adversaries increases, which is consistent with our analysis. In contrast, the drop of accuracy in Byzantine-FedVote is negligible, conﬁrming its resilience to omniscient attackers.
1 initialize p (0) and broadcast 2 for k = 0 , 1 , . . .
, N − 1 do on m th worker: receive p ( k ) from the server initialize latent weight
h
( k, 0) m =
ϕ − 1 (2 p ( k ) − 1) for t = 0 :
τ − 1 do ˜ g
( k,t ) m
= ∇ h f m ( ϕ ( h ( k,t ) m ); ξ ( t,r ) m ) h ( k,t +1) m
= h ( k,t ) m − η ( k,t ) ˜
g ( k,t ) m (cid:101) w ( k,τ ) m =
ϕ ( h ( k,τ ) )
w ( k,τ ) m = sto rounding( (cid:101) w ( k,τ ) m ) (cid:46) Eq.
(11) send w ( k,τ ) m to server on server: { w ( k +1) , p ( k +1) }
= vote( { w ( k,τ ) m } Mm =1 ) broadcast p ( k +1) to workers 15 function vote ( { w ( k,τ ) m } Mm =1 ) for i
= 1 : d do w ( k +1)
i = sign (cid:18)
M (cid:80) m
=1 w (
[Table]
,τ ) m,i (cid:19) p (
[Table]
1) i =
[Table]
M M (cid:80) m =
[Table]
(cid:16) (cid:98) w (
[Table] [Table]
) m,
[Table]
=
[Table]
(cid:17) (cid:46) Option I p (
[Table]
1) i =
M
[Table]
m
[Table]
λ (
[Table]
) m
[Table]
(cid:16)
[Table]
[Table] [Table]
) m,
[Table]
[Figure] return { w ( k +1) , p ( k +1) } to the server [Figure] [Figure]
C. C ONVERGENCE OF F ED V OTE WITH P ARTIAL P ARTICIPATION
We increase the number of workers to 100 and sample 20 of them in each communication round. We compare FedVote with FedAvg McMahan et al. (2017), FedPAQ (Reisizadeh et al., 2020), and signSGD (Bernstein et al., 2018) in Figure 5 on the non-i.i.d.
CIFAR-10 dataset ( α = 0 . 5 ) without changing other experimental setups.
The observation is consistent with the results in Figure 4.
FedVote outperforms gradient quantization methods such as FedPAQ and signSGD.
C. E XTENSION TO T ERNARY N EURAL N ETWORKS
The stochastic rounding used in the ternary neural networks, w i =
Q sr ( (cid:101) w ) , is an extension of (11): w i = (cid:40)
+1 , with probability π 1 = ˜
w i 1 ( (cid:101) w i > 0) , − 1 , with probability π 2 = − ˜ w i 1 ( (cid:101) w i < 0) , 0 , with probability 1 − ( π 1 + π 2 ) .
One can modify the normalization function to optimize neural networks with multiple quantization levels.
For example, consider quaternary weight w i ∈ {− 2 , − 1 , 1 , 2 } , the normalization function ϕ ( x ) can be modiﬁed to ϕ ( x ) = 2 tanh( ax ) .
C. B ATCH N ORMALIZATION IN F ED V OTE
Below we review the commonly-adopted BN function for convenience of presentation.
For a onedimensional input x ( j ) from the current batch B = { x (1) , · · · , x ( n b ) } , the output of BN layer is formulated as
[Equation]
where γ, b are learnable afﬁne transformation parameters, and µ, σ 2 are the mean and variance calculated over the batch samples.
Note that the normal BN layer will introduce the real-valued parameters and track the statistics of the input, all of which may cause problems when being binarized in FedVote.
Therefore, we choose to set the parameter-free static BN, i.e.,
[Equation]
[Equation]
D M ISSING P ROOFS D. P ROOF OF L EMMA
Lemma 1 (One-Shot FedVote) Let w ∗ ∈
D
d 2 be the optimal solution deﬁned in (8).
For the m th worker, ε m,i (cid:44) P ( w ( k,τ ) m,i (cid:54) =
w ∗ i ) denotes the error probability of the voting result of the i th coordinate.
Suppose the error events { w ( k,τ ) m,i (cid:54) =
w ∗ i } Mm =1 are mutually independent, and the mean error probability s i = 1 M (cid:80) Mm =
ε m,i is smaller than 1 / 2 . For the voted weight w ( k +1) , we have
[Equation]
Proof.
Let X m,i (cid:44) 1 (cid:16) w ( k,τ ) m,i (cid:54) =
w ∗ i (cid:17) , following Bernoulli distribution with parameter ε m,i . Let
Y i
=
(cid:80) Mm =
1 X m,i , we have
[Equation]
With independent vote results from workers, Y i follows Poisson binomial distribution with mean µ Y
i = (cid:80)
Mm =
ε m,i .
∀ a > 0 , the Chernoff bound can be derived as
[Equation]
where (cid:172) is based on Markov’s inequality.
(cid:173) holds due to ln(1 + x ) (cid:54) x for all x ∈ ( − 1 , ∞ ) .
By assumption we have µ Y i < M 2 .
Let a = ln M 2 µ Yi , we have
[Equation]
(26b)
Let s i = µ Yi M = 1 M (cid:80) Mm {+=
+} ε m,i and substitute it into (26b), the proof is complete.
D. P ROOF OF L EMMA
Lemma 2 (Relationship with FedAvg) For the normalized weights, FedVote recovers FedAvg in expectation: E (cid:2)(cid:101) w ( k +1) (cid:3)
= 1 M M (cid:80) m
=1 (cid:101) w ( k,τ ) m , where (cid:101) w ( k +1) =
ϕ ( h ( k +1) ) and (cid:101) w ( k,τ ) m =
ϕ ( h ( k,τ ) m ) .
Proof.
From the inverse normalization in (14), we have (cid:101) w ( k +1)
=
2 p ( k +1) − 1 .
Recall the deﬁnition of the empirical Bernoulli parameter p
( k +1)
given by (13), we have the elementwise expectation
[Equation]
where (cid:172) follows from stochastic rounding deﬁned in (11).
Based on the deﬁnition of range normalization in (10), for local normalized weight we have w ( k,τ ) m,i = ϕ
( h ( k,τ ) m,i ) .
Substituting the result into (27b) we have
[Equation] [Equation]
which completes the proof.
D. P ROOF OF L EMMA
Lemma 3 Suppose we have an input a ∈ ( − 1 , 1) d for the quantizer Q sr deﬁned in (11), then the quantization error satisﬁes E (cid:104)(cid:13)(cid:13) Q sr ( a ) − a (cid:13)(cid:13) 2 2 (cid:12)(cid:12) a (cid:105) = d − (cid:13)(cid:13) a (cid:13)(cid:13) 2 2 .
Proof.
Let ˆ a (cid:44) Q sr ( a ) , we have
[Equation]
The proof is complete.
D. P ROOF OF L EMMA
Lemma 4 Suppose we have an input x ∈ R d for the QSGD quantizer Q .
In the coarse quantization scenario, the quantization error satisﬁes E (cid:104)(cid:13)(cid:13) Q [Equation] x [Equation]
x [Equation] 2 (cid:12)(cid:12) x [Equation]
[Equation]
(cid:13)(cid:13) x
(cid:13)(cid:13) 2 2 .
Proof.
Consider a QSGD quantizer (Alistarh et al., 2017) with s = 1 .
For detailed results of other quantizers, we refer readers to Basu et al.
In particular, we have
[Equation]
where
[Equation]
The variance of quantization error is
[Equation]
which completes the proof.
D. P ROOF OF T HEOREM
We ﬁrst introduce some notations for simplicity.
Let ∆ ( k ) denote the difference between two successive global latent weights, i.e., ∆ ( k ) (cid:44) (cid:101) w ( k ) − (cid:101) w ( k +
[Equation]
We use ε ( k,t ) m to denote the stochastic gradient noise, i [Equation] e.,
[Equation]
Finally, we let ∇ f ( (cid:101) w ) denote the gradient with respect to (cid:101) w .
The following ﬁve lemmas are presented to facilitate the proof.
Lemma
The global normalized weight (cid:101) w ( k ) can be reconstructed as the average of local binary weight, i.e.,
[Equation]
Proof.
See Appendix D.6.
Lemma 6 (Lipschitz continuity) Under Assumption 3, ∀ x 1 , x 2 ∈ R , we have
[Equation]
Proof.
Without loss of generality, suppose x 1 < x 2 .
Based on the mean value theorem, there exists some c ∈ ( x 1 , x 2 ) such that
[Equation]
For the monotonically increasing function ϕ , we have
[Equation]
where (cid:172) holds due to Assumption 3.
The similar result can be obtained by assuming x 2 < x 1 , which completes the proof.
Lemma 7 (Bounded weight divergence)
Under Assumption 4, we have
[Equation]
Proof.
See Appendix D.7.
Lemma 8 Under Assumptions 2 to 5, we have
[Equation]
) , ∆ ( t ) (cid:69) (cid:62) c 21 ητ 2 E (cid:13)(cid:13) ∇ f ( (cid:101) w ( k ) ) (cid:13)(cid:13) 2(40) + c 21 η 4 M (cid:0) 2 − ( c 2 L ) 2 η 2 τ ( τ − 1) (cid:1)
M (cid:88) m
= τ − 1 (cid:88) t = E (cid:13)(cid:13) g ( k,t ) m (cid:13)(cid:13) 2− ( c 1 c 2 L ) 2 η 3 τ ( τ − 1) σ 2 ε − η ( c 22 − c 21 ) M M (cid:88) m =
R ( k ) m , (41)
where R ( k ) m
(cid:44) − τ − 1 (cid:80) t
= (cid:80) i/ ∈I ( k,t ) m
E (cid:104) ( ∇ f ( (cid:101) w ( k ) )) i ( ∇ f ( (cid:101) w ( k,t ) m )) i (cid:105) and I ( k,t ) m (cid:44) (cid:110) i | g ( k ) i g ( k,t ) m,i (cid:62) 0 (cid:111) .
Proof.
See Appendix D.8.
Lemma 9 (Bounded global weight difference) Under Assumptions 2 to 5, we have
[Equation]
Proof.
See Appendix D.9.
Theorem 1 For FedVote under Assumptions 1 to 5, if the learning let the learning rate η =
O (cid:16) ( c 1 c 2 )1 Lτ √ K (cid:17) , then after K rounds of communication, we have
[Equation]
where R ( k ) m
(cid:44) − τ − 1 (cid:80) t =
[Equation] k [Equation] ) i (
[Equation]
and I
[Equation]
Proof.
Consider the difference vector ∆ ( k ) deﬁned in (33), we expand it as
[Equation]
44b)
[Equation]
where (cid:172) follows from (35), and (cid:173) holds by substituting the deﬁnition of quantization error.
From Assumption 2, we have
[Equation]
Let ( Lη ) 2 τ ( τ − 1) [Equation]
[Equation]
1 c 22 , take the expectation on both sides, and use Lemmas 8–9: [Equation]
where (cid:172) follows from the restrictions on learning rate.
We rewrite the restriction as
[Equation]
47c
[Equation]
where in (cid:172) we use τ ( τ − 1) (cid:54) τ 2 , and (cid:173) holds due to the Bernoulli inequality (1 + x ) 12 (cid:54) 1 + 1 2 x, ∀ x ∈ [ − 1 , ∞ ) .
Based on (47c), we set the learning rate η = O (cid:16) ( c 1 c 2 )1 Lτ √ K (cid:17) . Summing
[Equation]
D. P ROOF OF L EMMA
Proof.
From the reconstruction rule (14), we have
[Equation]
[Equation]
The i th entry of p ( k ) is deﬁned in (13) as:
[Equation]
[Equation]
Substituting (50) into (49) yields
[Equation]
Note that
[Equation]
[Equation]
we have
[Equation]
53) which completes the proof.
D. P ROOF OF L EMMA
Proof.
With the local initialization and update method described in Algorithm 1, we have
[Equation]
(cid:13)(cid:13)
[Equation]
(54b)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] (cid:88) [Equation]
−
[Equation] g ( k [Equation] t )
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  (54c) =
( c
[Equation]
η ) [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation]
− [Equation] (cid:88) t
[Equation]
(cid:16) g ( k
[Equation] t )
[Equation] + ε ( k,t ) m (cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  (54d) =
( c 2 η )(cid:32)
E (cid:13)(cid:13) (cid:13) (cid:13) (cid:13) τ − 1 (cid:88) t = g ( k [Equation]
t ) m (cid:13)(cid:13) (cid:13) (cid:13)[-2 
-]
[Equation] where (cid:172) comes from the Lipschitz condition in Lemma 6.
D. P ROOF OF L EMMA
Proof.
We have
[Equation]
where (cid:172) follows from Assumption 5.
Based on the mean value theorem, for h ( k ) i , h ( t,τ ) m,i ∈ R , there exists some c ( k ) m,i ∈ H ( r ) m,i such that
[Equation]
56) where H ( r ) m,i is an open interval with endpoints h ( k ) i and h ( k,τ ) m,i :
[Equation]
m
[Equation]
, otherwise.
[Equation]
Let C ( k ) m = diag (cid:16) ϕ (cid:48)
( c ( k ) m, 1 ) , . . .
, ϕ (cid:48)
( c ( k ) m,d ) (cid:17) , we have
[Equation]
Substituting (58) into (55c) yields
[Equation]
[Equation]
According to the chain rule, g ( k,t ) m can be written as
[Equation]
where D ( k,t ) m = diag
[Equation]
(cid:19) .
To simplify the notations, let B ( k,t ) m
=
C ( k ) m D ( k,t ) m .
Note that B ( k,t ) m is still a diagonal matrix, where the i th diagonal element b ( k,t ) m,i is
[Equation]
Substituting (60) into (59) we have
[Equation]
We ﬁrst focus on the following inner product:
[Equation]
where ( ∇ f ) i denotes the i th entry of the gradient vector.
Consider an index set I ( k,t ) m deﬁned as
[Equation]
Since the sign of ( ∇ f ( (cid:101) w ( k ) ))
i ( ∇ f m ( (cid:101) w ( k,t ) m ))
i is equal to g ( k ) i g
( k,t ) m,
i
[Equation]
the index set I ( k,t ) m can also be written as
[Equation]
[Equation]
The result in (63) can be bounded as
[Equation]
where (cid:172) follows from Assumption 3.
To simplify the notation
[Equation]
let R ( k ) m denote the accumulative gradient divergence, namely,
[Equation]
The expected inner product in (62) can be bounded as
[Equation]
68b
[Equation]
(cid:173) [Equation] ητ [Equation]
[Equation]
[Equation]
M
M (cid:88) m
[Equation]
R ( k ) m , (68c) where (cid:172) follows from 2 (cid:10) a , b (cid:11) =
{+(cid:13)(cid:13) a (cid:13)(cid:13) 2 
+}
+ (cid:13)(cid:13)
b (cid:13)(cid:13) 2 2 − (cid:13)(cid:13) a − b (cid:13)(cid:13) 2 2 , and (cid:173) holds due to Assumption 2.
From Lemma 7, we can show that
[Equation]
[Equation]
where t = 1 , . .
, τ −
Substituting (69) in (68c) yields
[Equation]
where (cid:172) follows from τ ( τ − 1) − n ( n + 1) (cid:54)
τ ( τ − 1) , and (cid:173) holds due to the chain rule in (60) and Assumption 3. D. P ROOF OF L EMMA Proof. We have
[Equation] =
E
[Equation]
1 M
M (cid:88) m = ϕ ( h ( k ) ) − ϕ ( h ( k,τ ) m )
+ 1 M M (cid:88) m = ζ ( k ) m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  (71b) = E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 M
M (cid:88) m =
ϕ ( h ( k ) ) − ϕ ( h ( k,τ ) m ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 M
M (cid:88) m = ζ ( k ) m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + 2 E (cid:42) M M (cid:88) m =
ϕ ( h ( k ) )
− ϕ ( h ( k,τ ) m ) [-(cid:13)(cid:13) 
-], 1 M M (cid:88) m
= ζ ( t ) m (cid:43) (71c) (cid:172) =
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
M
M (cid:88) m = ϕ ( h ( k ) )
− ϕ ( h ( k,τ ) m ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 M
M (cid:88) m
= ζ ( k ) m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  , (71d)
[Equation]
) (cid:13)(cid:13)(cid:13) (72a) (cid:172) (cid:54)
[Equation]
m
[Equation]
(cid:13)(cid:13)
[Equation]
τ ) m (cid:13)(cid:13) 2(72b) (cid:173)
[Equation]
[Equation]
m = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
τ
[Equation]
(cid:88) t =
[Equation]
k
[Equation]
t )
[Equation]
[-(cid:13)(cid:13) 
-](72c)
[Equation]
[Equation] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  (72d)
[Equation]
[Equation]
[Equation]
τ
M
[Equation]
(cid:88)
[Equation]
where (cid:172) is due to Lemma 6 and (cid:173) comes from (9).
For the second term in (71d) we have
[Equation]
Combing the results of (72e) and (73) completes the proof.