C OMMUNICATION -C OMPUTATION E FFICIENT S ECURE A GGREGATION FOR F EDERATED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning has been spotlighted as a way to train neural network models using data distributed over multiple clients without a need to share private data.
Unfortunately, however, it has been shown that data privacy could not be fully guaranteed as adversaries may be able to extract certain information on local data from the model parameters transmitted during federated learning.
A recent solution based on the secure aggregation primitive enables privacy-preserving federated learning, but at the expense of signiﬁcant extra communication/computational resources.
In this paper, we propose communication-computation efﬁcient secure aggregation which reduces the amount of communication/computational resources at least by a factor of (cid:112) n/ log n relative to the existing secure solution without sacriﬁcing data privacy, where n is the number of clients.
The key idea behind the suggested scheme is to design the topology of the secret-sharing nodes (denoted by the assignment graph G ) as sparse random graphs instead of the complete graph corresponding to the existing solution.
We ﬁrst obtain a sufﬁcient condition on G to guarantee reliable and private federated learning.
Afterwards, we suggest using the Erd˝os-R´enyi graph as G , and provide theoretical guarantees on the reliability/privacy of the proposed scheme.
Through extensive real-world experiments, we demonstrate that our scheme, using only 50% of the resources required in the conventional scheme, maintains virtually the same levels of reliability and data privacy in practical federated learning systems.
I NTRODUCTION
Federated learning (McMahan et al., 2017) has been considered as a promising framework for training models in a decentralized manner without explicitly sharing the local private data.
This framework is especially useful in various predictive models which learn from private distributed data, e.g. , healthcare services based on medical data distributed over multiple organizations (Brisimi et al., 2018; Xu & Wang, 2019) and text prediction based on the messages of distributed clients (Yang et al., 2018; Ramaswamy et al., 2019).
In the federated learning (FL) setup, each device contributes to the global model update by transmitting its local model only; the private data is not shared across the network, which makes FL highly attractive (Kairouz et al., 2019; Yang et al., 2019).
Unfortunately, however, FL could still be vulnerable against the adversarial attacks on the data leakage.
Speciﬁcally, the local model transmitted from a device contains extensive information on the training data, and an eavesdropper can estimate the data owned by the target device (Fredrikson et al., 2015; Shokri et al., 2017; Melis et al., 2019).
Motivated by this issue, the authors of (Bonawitz et al., 2017) suggested secure aggregation (SA), which integrates cryptographic primitives into the
FL framework to protect data privacy.
However, SA requires signiﬁcant amounts of additional resources on communication and computing for guaranteeing privacy.
Especially, the communication and computation burden of SA increases as a quadratic function of the number of clients, which limits the scalability of SA.
Contributions We propose communication-computation efﬁcient secure aggregation (CCESA), which maintains the reliability and data privacy in federated learning, with reduced resources on communication and computation compared to conventional SA.
Our basic idea is illustrated in Fig.
[Figure]
𝐺 𝑆𝐴 𝐺 𝐶𝐸𝑆𝐴
[Figure]
[-𝐺 𝑆𝐴 𝐺 𝐶𝐸𝑆𝐴  5   
-]
[Figure]
[Figure]
[Figure]
𝐺 𝑆𝐴 𝐺 𝐶𝐸𝑆𝐴
[Figure]
[Figure]
for n = 5 clients.
Compared to the existing scheme (Bonawitz et al., 2017), which applies secret sharing for all client pairs, we suggest sharing secrets for a subset of pairs in a way that data privacy is preserved.
Using theoretic analysis, we provide a sufﬁcient condition on the graph topology for private and reliable federated learning.
As summarized in Table 1, the proposed CCESA algorithm maintains both reliability and privacy against an eavesdropper who can access to any information transmitted between every client and the server, while the required amount of resources reduces by a factor of at least O ( (cid:112) n/ log n ) compared to SA.
Notably, the reduction factor gets bigger with increasing n , suggesting that the proposed scheme is a scalable solution for privacy-preserving federated learning.
Our mathematical results are also conﬁrmed in experiments on two real datasets of AT&T face database and CIFAR-10.
Especially, under the model inversion attack on the face dataset, the results in Fig.
2 show that the suggested scheme achieves perfect data privacy by using less amount of resources than SA, while federated averaging without security measures (McMahan et al., 2017) signiﬁcantly compromises the privacy of the data.
Related work Focusing on the collaborative learning setup with multiple clients and a server, previous works have suggested solutions to prevent the information leakage in the communication links between the server and clients.
One major approach utilizes the concept of differential privacy (DP) (Dwork et al., 2014) by adding artiﬁcial random noise to the transmitted models (Wei et al., 2020;
Geyer et al., 2017; Truex et al., 2020) or gradients (Shokri & Shmatikov, 2015; Abadi et al., 2016; Balcan et al., 2012).
Depending on the noise distribution, DP-based collaborative learning generally exhibits a trade-off between the privacy level and the convergence of the global model.
Another popular approach is deploying secure multiparty computation (MPC)
(Ben-Or et al., 1988;
Damg˚ard et al., 2012;
Aono et al., 2017;
Lindell et al., 2015; Bonawitz et al., 2017; Zhang et al., 2018; Tjell & Wisniewski, 2019;
Shen et al., 2020; So et al., 2020) based on the cryptographic primitives including secret sharing and the homomorphic encryption (Leontiadis et al., 2014; 2015; Shi et al., 2011; Halevi et al., 2011).
Although these schemes guarantee privacy, they suffer from high communication burdens while reconstructing the secret information distributed over multiple clients.
A notable work (Bonawitz et al., 2017) suggested secure aggregation (SA), which tolerates multiple client failures by applying pairwise additive masking ( ´Acs & Castelluccia, 2011; Elahi et al., 2014; Jansen & Johnson, 2016; Goryczka & Xiong, 2015).
A recent work (So et al., 2020) suggested Turbo-aggregate, which partitions n computing devices into L groups and updates the global model by utilizing the circular aggregation topology.
However, each client in Turbo-aggregate requires a communication cost of at least 4 mnR/L bits, which is much larger than that of our scheme (CCESA) requiring a communication cost of √ n log n (2 a K + 5 a S ) + mR 1 .
For example,
CCESA
SA
FedAvg
Communicationcost
Client
O ( √ n log n + m )
O ( n + m )
O ( m ) Server
O ( n √ n log n + mn )
O ( n 2 + mn )
O ( mn )
Computationcost
Client
O ( n log n + m √ n log n )
O ( n 2 + mn )
Server
O ( mn log n )
O ( mn 2 )
O ( mn )
Reliability ≥ 1 − O ( ne −√ n log n ) ≥ 1 − O ( ne − n ) 1 Privacy
[Table]
in a practical scenario with m = 10 6 , R = 32 , n = 100 , L = 10 and a K = a S = 256 , our scheme requires only 3% of the communication bandwidth used in Turbo-aggregate.
The idea of replacing the complete graph with a low-degree graph for communication efﬁciency has been studied in the areas of distributed learning (Charles et al., 2017; Sohn et al., 2020) and multi-party computation (Fitzi et al., 2007; Harnik et al., 2007).
A very recent paper (Bell et al., 2020) proposed new protocols using k -regular graphs for communication in the secure aggregation (Bonawitz et al., 2017) framework, which are robust against semi-honest and semi-malicious threat models, respectively.
However, the results in (Bell et al., 2020) is based on a strong assumption on the number of clients dropped out of the protocol, while our work does not assume anything on the number of dropouts.
B ACKGROUND
Federated learning Consider a scenario with one server and n clients.
Each client i has its local training dataset D i
= { ( x i,k , y i,k ) } N i k
=1 where x i,k and y i,k are the feature vector and the label of the k -th training sample, respectively.
For each round t , the server ﬁrst selects a set S t of cn clients (0 < c ≤ 1) and sends the current global model θ ( t ) global to those selected clients.
Then, each client i ∈ S t updates the received model by using the local data D i and sends the updated model θ ( t +1)
i to the server.
Finally, the server updates the global model by aggregating local updates from the selected clients, i.e. , θ ( t +1) global ← (cid:80) i ∈
S t N i N θ ( t +1) i where N = (cid:80) i ∈
S t N i .
Cryptographic primitives for preserving privacy Here we review three cryptographic tools used in SA (Bonawitz et al., 2017).
First, t -out-of- n secret sharing (Shamir, 1979) is splitting a secret s into n shares, in a way that any t shares can reconstruct s , while any t − 1 shares provides absolutely no information on s .
We denote t -out-of- n secret sharing by s ( t,n ) −−−→ ( s k ) k ∈ [ n ] , where s k indicates the k th share of secret s and [ n ] represents the index set { 1 , 2 , · · · , n } .
Second, the Difﬁe-Hellman key agreement is used to generate a secret s i,j that is only shared by two target clients i, j ∈ [ n ] .
The key agreement scheme designs public-private key pairs ( s PKu , s SKu ) for clients
u ∈ [ n ] in a way that s i,j = f ( s PKi , s SKj )
=
f ( s PKj , s SKi ) holds for all i, j ∈ [ n ] for some key agreement function f .
The secret s i,j is unknown when neither s SKi nor s SKj is provided.
Third, symmetric authenticated encryption is encrypting/decrypting message m using a key k shared by two target clients.
This guarantees the integrity of the messages communicated by the two clients.
Secure aggregation (Bonawitz et al., 2017)
For privacy-preserving federated learning, SA has been proposed based on the cryptographic primitives of Shamir’s secret sharing, key agreement and symmetric authenticated encryption.
The protocol consists of four steps: Step
Advertise Keys,
Step 1 Share Keys, Step 2 Masked Input Collection, and Step 3 Unmasking.
Consider a server with n clients where client i ∈ [ n ] has its private local model θ i .
Denote the client index set as V 0 =
[ n ] .
The objective of the server is to obtain the sum of models (cid:80)
i θ i without getting any other information on private local models.
In Step 0 , client i ∈ V 0 generates key pairs ( s PKi , s SKi ) and ( c PKi , c SKi ) by using a key agreement scheme.
Then, client i advertises its public keys ( s PKi , c PKi ) to the server.
The server collects the public keys from a client set V 1 ⊂ V 0 , and broadcasts { ( i, s PKi , c PKi ) } i ∈ V 1 to all clients in V 1 .
In Step 1 , client i generates a random element b i and applies t -out-of- n secret sharing to generate n shares of b i and s SKi , i.e. , b i ( t,n ) −−−→ ( b i,j ) j ∈ [ n ] and s SKi ( t,n ) −−−→ ( s SKi,j ) j ∈ [ n ] .
By using the symmetric authenticated encryption, client i computes the ciphertext e i,j for all j ∈ V 1 \{ i } , by taking b
i,j and s SKi,j as messages and c i,j = f ( c PKj , c SKi ) as a key, where f is the key agreement function.
Finally, client i sends { ( i, j, e i,j ) } j ∈ V 1 \{ i } to the server.
The server collects the message from at least t clients (denote this set of client as V 2 ⊂ V 1 ) and sends { ( i, j, e i,j ) } i ∈ V 2 to each client j ∈ V 2 .
In Step 2 , client i computes the shared secret s i,j = f
( s PKj , s SKi ) for all j ∈ V 2 \{ i } .
Then, client i computes the masked private vector ˜ θ i
= θ i + PRG ( b i ) + (cid:88) j ∈ V 2 ; i<j PRG ( s i,j ) − (cid:88) j ∈ V 2 ; i>j
PRG ( s i,j ) , (1) and sends ˜ θ i to the server, where PRG ( x ) indicates a pseudorandom generator with seed x outputting a vector having the dimension identical to θ i .
Note that the masked vector ˜ θ i gives no information on private vector θ i unless both s SKi and b i are revealed.
The server collects ˜ θ i from at least t clients (denote this set as V 3 ⊂ V 2 ), and sends V 3 to each client i ∈ V 3 .
In Step 3 , client j decrypts the ciphertext { e i,j } i ∈
V 2 \{ j } by using the key c i,j = f ( c PKi , c SKj ) to obtain { b i,j } i ∈ V 2 \{ j } and { s SKi,j } i ∈
V
2 \{ j } .
Each client j sends a set of shares { b i,j } i ∈
V 3 ∪ { s SKi,j } i ∈ V
2 \ V 3 to the server.
The server collects the responds from at least t clients (denote this set of clients as V 4 ⊂ V 3 ).
For each client i ∈ V 3 , the server reconstructs b i from { b i,j } i ∈ V 4 and computes PRG ( b i ) .
Similarly, for each client i ∈ V 2 \ V 3 , the server reconstructs s SKi from { s SKi,j } i ∈ V 4 and computes PRG ( s i,j ) for all j ∈ V 2 .
Using this information, the server obtains the sum of private local models by computing (cid:88) i ∈
Vθ i = (cid:88) i ∈
V˜ θ i − (cid:88) i ∈ V
PRG ( b i ) − (cid:88) j ∈ V 2 ,i ∈ V 2 \ V 3 ; i<j
PRG ( s i,j ) + (cid:88) j ∈ V 2 ,i ∈ V 2 \ V 3 ; i>j
PRG ( s i,j ) .
S UGGESTED ALGORITHM
In the secure aggregation (Bonawitz et al., 2017), the public keys ( c PKi , s PKi ) and the shares of secrets ( s SKi , b i ) are transmitted between clients and the server, which requires additional communication/computational resources compared with the vanilla federated learning.
Speciﬁcally, since each client i needs to receive information from all other clients j (cid:54)
=
i , the required amount of resources increases as a quadratic function of the number of clients n .
In this paper, we suggest a variant of secure aggregation, dubbed as communication-computation efﬁcient secure aggregation (CCESA), which enables to provide a more scalable solution for privacypreserving federated learning by improving the communication/computational efﬁciency.
The basic idea of the proposed algorithm is to allow each client to share its public keys and secret shares to a subset of other clients, instead of sharing them with all other clients.
By doing so, compared with SA, the suggested scheme achieves two advantages in resource efﬁciency, without losing the reliability of learning algorithms and the data privacy.
The ﬁrst advantage is the reduction of the communication cost, since each node shares its public keys and secrets with less clients.
The second advantage is a reduction of the computational cost of each client, since a smaller number of masks are used while computing its masked private vector.
The proposed algorithm is speciﬁed by the assignment graph which represents how public keys and secret shares are assigned to the other clients.
Given n clients, the assignment graph G =
( V, E ) consists of n vertices where the vertex and the edge set of G are represented by V and E , respectively.
We set V =
[ n ] where each index i ∈ V represents client i , and the edge { i, j } ∈ E connecting vertices i and j indicates that client i and j exchange their public keys and secret shares.
For vertex i ∈ [ n ] , we deﬁne Adj ( i ) :=
(cid:8) j ; { i, j } ∈ E (cid:9) as the index set of vertices adjacent to vertex i .
In our algorithm, public keys and secrets of client i are shared with clients j ∈ Adj ( i ) .
Now, using the assignment graph notation, we formally deﬁne the suggested algorithm.
Due to the space limitation, we put the algorithm in Supplementary Materials A; here we only describe what differs from SA.
In Step 0 , instead of broadcasting the public keys ( c PKj , s PKj ) for client j to all other clients, the server sends the public keys only to the client i satisfying j ∈ Adj ( i ) ∩ V 1 .
In
Step 1 , each client i ∈ V 1 uses t i -out-of-
( | Adj ( i ) | + 1) secret sharing scheme to generate shares of s SKi and b i , i.e. , s SKi ( t i , | Adj ( i ) | +1)
−−−−−−−−−→ ( s SKi,j )
j ∈ Adj ( i ) ∪{ i } and b i ( t i , | Adj ( i ) | +1)
−−−−−−−−−→ ( b i,j )
j ∈ Adj ( i ) ∪{ i } , and sends the encrypted s SKi,j and b i,j to client j through the server.
In Step 2 , client i computes the masked private model
[Equation]
and transmits ˜ θ i to the server.
In Step 3 , client i sends b j,i to the server for all j ∈ V
∩ Adj ( i ) , and sends s SKj,i to the server for all j ∈ ( V 2 \ V 3 )
∩ Adj ( i ) .
After reconstructing secrets from shares, the server obtains the sum of the local models θ i as
[Equation]
[Equation]
Note that the suggested protocol with n -complete assignment graph G reduces to SA.
In Fig.
1, we illustrate an example of the suggested algorithm with n = 5 clients.
Fig.
1a corresponds to SA (Bonawitz et al., 2017), while Fig.
1b depicts the proposed scheme.
Here, we focus on the required communication/computational resources of client 1.
Note that each client exchanges public keys and secret shares with its adjacent clients.
For example, client 1 exchanges the data with four other clients in the conventional scheme, while client 1 exchanges the data with clients 3 and 5 in the suggested scheme.
Thus, the proposed CCESA requires only half of the bandwidth compared to the conventional scheme.
In addition, CCESA requires less computational resources than conventional scheme, since each client generates less secret shares and pseudo-random values, and performs less key agreements.
T HEORETICAL ANALYSIS 4. P ERFORMANCE METRICS
The proposed CCESA algorithm aims at developing private, reliable and resource-efﬁcient solutions for federated learning.
Here, we deﬁne key performance metrics for federated learning systems including federated averaging (McMahan et al., 2017), SA (Bonawitz et al., 2017) and CCESA.
Recall that the server receives (masked) model parameters ˜
θ
i from clients i ∈ V 3 , and wants to update the global model as the sum of the unmasked model parameters, i.e., θ global ← (cid:80) i ∈ V 3 θ i .
The condition for successful (or reliable) global model update is stated as follows.
Deﬁnition 1.
A system is called reliable if the server successfully obtains the sum of the model parameters (cid:80) i ∈ V
3 θ i aggregated over the distributed clients.
Now, we deﬁne private federated learning.
In our analysis, we focus on a passive eavesdropper who can access to any information transmitted between any client and the server throughout running
CCESA algorithm, namely, public keys of clients, secret shares, masked local models and the indices of survived clients V 3 2 .
Once an eavesdropper gets the local model θ i of client i , it can reconstruct the private data ( e.g. , face image or medical records) of the client or can identify the client which contains the target data.
In general, if an eavesdropper obtains the sum of local models occupied by a subset T of clients, a similar privacy attack is possible for the subset of clients.
Thus, to preserve the data privacy, it is safe to protect the information on the partial sum of model parameters against the eavesdropper; we formalize this.
Deﬁnition 2.
A system is called private if H ( (cid:80) i ∈T θ i ) = H ( (cid:80) i ∈T θ
i | E ) holds for all T satisfying T ⊂ V 3 and T / ∈ { ∅ , V 3 } .
Here, H is the entropy function, and E is the information accessible to the eavesdropper.
When both reliability and privacy conditions hold, the server successfully updates the global model, while an eavesdropper cannot extract data in the information-theoretic sense.
We deﬁne P ( r ) e and
P ( p ) e as the probabilities that reliability and privacy conditions do not hold, respectively.
4. R ESULTS FOR GENERAL ASSIGNMENT GRAPH G
Recall that our proposed scheme is speciﬁed by the assignment graph G .
We here provide mathematical analysis on the performance metrics of reliability and privacy, in terms of the graph G .
To be speciﬁc, the theorems below provide the sufﬁcient conditions on the assignment graph G to enable reliable/private federated learning, where the reliability and privacy are deﬁned in Deﬁnitionsand 2.
Theorem 1.
The system is reliable if and only if | ( Adj ( i ) ∪ { i } ) ∩ V 4 | ≥ t i for all i ∈ V +3 , where
V [-+ 
-]+3 =
V 3 ∪ { i ∈ V
Adj ( i ) ∩ V 3 (cid:54) = ∅ } is the union of V 3 and the neighborhoods of V 3 within V 2 .
Proof.
The full proof is given in Supplementary Materials; here we provide a sketch for the proof.
Recall that the server receives the sum of masked models (cid:80) i ∈ V 3 ˜ θ i , while the system is said to be reliable if the server obtains the sum of unmasked models (cid:80) i ∈ V 3 θ i .
Thus, the reliability condition holds if and only if the server can cancel out the random terms in (4), which is possible when either s SKi or b i is recovered for all i ∈ V +3 .
Since a secret is recovered if and only if at least t i shares are gathered from adjacent nodes, we need | ( Adj ( i ) ∪ { i } )
∩ V 4 | ≥ t i , which completes the proof.
Theorem 2.
The system is private if G 3 :
= G − ( V \ V 3 ) is connected.
Proof.
Again we just provide a sketch of the proof here; the full proof is in Supplementary Materials.
Note that G 3 is the induced subgraph of G whose vertex set is V 3 .
Suppose an eavesdropper has access to the masked local models { ˜ θ i } i ∈T of a subset T ⊂ V 3 of nodes.
Now, the question is whether this eavesdropper can recover the sum of the unmasked models (cid:80) i ∈T θ i .
If G 3 is connected, there exists an edge e = { p, q } such that p ∈ T and q ∈ V 3 \T .
Note that (cid:80) i ∈T ˜ θ
i contains the PRG ( s p,q ) term, while s p,q is not accessible by the eavesdropper since p, q ∈ V 3 .
Thus, from (4), the eavesdropper cannot obtain (cid:80) i ∈T θ i , which completes the proof.
The theorems above provide guidelines on how to construct the assignment graph G to enable reliable and private federated learning.
These guidelines can be further speciﬁed when we use the
Erd ˝ os-R ´ enyi graph as the assignment graph G .
In the next section, we explore how the Erd ˝ os-R ´ enyi graph can be used for reliable and private federated learning.
4. R ESULTS FOR E RD ˝ OS -R´ ENYI ASSIGNMENT GRAPH G
The Erd ˝ os-R ´ enyi graph G ∈ G ( n, p ) is a random graph of n nodes where each edge connecting two arbitrary nodes is connected with probability p .
Deﬁne CCESA( n, p ) as the proposed scheme using the assignment graph of G ∈ G ( n, p ) .
According to the analysis provided in this section,
CCESA( n, p ) almost surely achieves both reliability and privacy conditions, provided that the connection probability p is chosen appropriately.
Throughout the analysis below, we assume that each client independently drops out with probability q at each step (from Step 0 to Step 3), and the secret sharing parameter t i is set to t for all i ∈ [ n ] .
4.3. F OR ASYMPTOTICALLY LARGE n
We start with the analysis on CCESA( n, p ) when n is asymptotically large.
The following two theorems provide lower bounds on p to satisfy reliability/privacy conditions.
The proofs are provided in Supplementary Materials.
Theorem 3.
CCESA( n, p ) is asymptotically almost surely reliable if p >
[Equation]
Theorem 4.
CCESA( n, p ) is asymptotically almost surely private if p > log
[Equation]
From these theorems, the condition for achieving both reliability and privacy is obtained as follows.
Remark 1.
Let
[Equation]
If p > p (cid:63) , then CCESA( n, p ) is asymptotically almost surely (a.a.s.) reliable and private.
Note that the threshold connection probability p (cid:63) is a decreasing function of n .
Thus, the proposed algorithm is getting more resource efﬁcient than SA as n grows, improving the scalability of the system.
In the remarks below, we compare SA and the proposed CCESA, in terms of the required amount of communication/computational resources to achieve both reliability and privacy.
These results are summarized in Table 1.
Remark 2.
Let B be the amount of additional communication bandwidth used at each client, compared to that of federated averaging (McMahan et al., 2017).
Since the bandwidth is proportional to np , we have B CCESA ( n,p ) ∼ O ( √ n log n ) and B SA ∼ O ( n ) .
Thus, the suggested CCESA protocol utilizes a much smaller bandwidth compared to SA in (Bonawitz et al., 2017).
The detailed comparison is given in Section D.1 of the Supplementary Materials.
Remark 3.
Compared to SA, the proposed CCESA algorithm generates a smaller number of secret shares and pseudo-random values, and performs less key agreements.
Thus, the computational burden at the server and the clients reduces by a factor of at least O ( (cid:112) n/ log n ) .
The detailed comparison is given in Section D.2 of the Supplementary Materials.
4.3. F OR FINITE n
We now discuss the performance of the suggested scheme for ﬁnite n .
Let P ( p ) e be the error probability that CCESA( n, p ) does not satisfy the the privacy condition, and deﬁne P ( p ) e as the error probability that CCESA( n, p ) is not reliable.
Below we provide upper bounds on P ( p ) e and P ( r ) e .
Theorem 5.
For arbitrary n, p, q and t , the error probability for reliability P ( r ) e is bounded by
P ( r ) e
[Equation]
, where D KL is the Kullback-Leibler (KL) divergence.
Theorem 6.
For arbitrary n, p and q , the error probability for privacy P ( p ) e is bounded by
[Equation]
Fig.
3 illustrates the upper bounds on P ( p ) e and P ( r ) e obtained in Theorems 5 and 6, when p = p (cid:63) .
Here, q total := 1 − (1 − q ) 4 is deﬁned as the dropout probability of the entire protocol (from Step 0 to Step 3).
Note that the upper bounds in Theorems 5 and 6 are decreasing functions of p .
Therefore, the plotted values in Fig.
3 are indeed upper bounds on the error probabilities for arbitrary p > p (cid:63) .
It is shown that a system with the suggested algorithm is private and reliable with high probability for an arbitrary chosen p > p (cid:63) .
The error probability for the privacy P ( p ) e is below 10 − 40 , which is negligible even for small n .
The error probability for the reliability P ( r ) e is below 10 − 2 , which means that in at most one round out of 100 federated learning rounds, the (masked) models { ˜ θ i } i ∈ Vreceived by the server cannot be converted to the sum of (unmasked) local models (cid:80) i ∈
V 3 θ i .
Even in this round when the server cannot obtain the sum of (unmasked) local models, the server is aware of the fact that the current round is not reliable, and may maintain the global model used in the previous round.
This does not harm the accuracy of our scheme, as shown in the experimental results of Section 5.
E XPERIMENTS
Here we provide experimental results on the proposed CCESA algorithm.
We compare CCESA and secure aggregation (SA) of (Bonawitz et al., 2017) in terms of time complexity (running time), reliability, and privacy.
We tested both schemes on two real datasets, AT&T Laboratories Cambridge database of faces (https://www.kaggle.com/kasikrit/att-database-of-faces) and CIFAR-10.
For the
AT&T face dataset containing images of 40 individuals, we considered a federated learning setup where each of n = 40 clients uses its own images for local training.
All algorithms are implemented in python and PyTorch (Paszke et al., 2017).
Codes will be made available to the public.
n q total t p Client Server Step Step
Step
Step
SA        130.
CCESA   0.
[Figure] [Figure] 5. R UNNING TIME In Table 2, we tested the running time of our CCESA and existing SA for various n and q total .
Similar to the setup used in (Bonawitz et al., 2017), we assumed that each node has its local model θ with dimension m = 10000 , where each element of the model is chosen from the ﬁeld F 2 16 .
Here, t is selected by following the guideline in Supplementary Materials, and p is chosen as p (cid:63) ∼
O ( (cid:112) log n/n ) deﬁned in (5) which is proven to meet both reliability and privacy conditions.
For every n, q total setup, the proposed CCESA( n, p ) requires p times less running time compared with the conventional SA of (Bonawitz et al., 2017).
This is because each client generates p times less number of secret shares and pseudo-random values, and performs p times less number of key agreements.
This result is consistent with our analysis on the computational complexity in Table 1.
5. R ELIABILITY
Recall that a system is reliable if the server obtains the sum of the local models (cid:80) i ∈ V 3 θ i .
Fig.shows the reliability of CCESA in CIFAR-10 dataset.
We plotted the test accuracies of SA and the suggested CCESA( n, p ) for various p .
Here, we included the result when p = p (cid:63) = 0 .
3106 , where p (cid:63) is the provably minimum connection probability for achieving both the reliability and privacy according to Remark 1.
One can conﬁrm that CCESA with p = p (cid:63) achieves the performance of
SA in both i.i.d.
and non-i.i.d.
data settings, coinciding with our theoretical result in Theorem 3.
Moreover, in both settings, selecting p = 0 . 25 is sufﬁcient to achieve the test accuracy performance of SA when the system is trained for 200 rounds.
Thus, the required communication/computational resources for guaranteeing the reliability, which is proportional to np , can be reduced to 50% of the conventional wisdom in federated learning.
Similar behaviors are observed from the experiments on
AT&T Face dataset, as in Fig.
B.1 of the Supplementary Materials.
5. P RIVACY
We ﬁrst consider a privacy threat called model inversion attack (Fredrikson et al., 2015).
The basic setup is as follows: the attacker eavesdrops the masked model ˜ θ i sent from client i to the server, and reconstructs the face image of a target client.
Under this setting, we compared how the eavesdropped model reveals the information on the raw data for various schemes.
As in Fig. 2 and Fig.
B.2 in the
Supplementary Materials, the vanilla federated averaging (McMahan et al., 2017) with no privacypreserving techniques reveals the characteristics of individual’s face, compromising the privacy of clients.
On the other hand, both SA and CCESA do not allow any clue on the client’s face; these schemes are resilient to the model inversion attack and preserve the privacy of clients.
This observation is consistent with our theoretical results that the proposed CCESA guarantees data privacy.
We also considered another privacy threat called the membership inference attack (Shokri et al., 2017).
Here, the attacker eavesdrops the masked model ˜ θ i and guesses whether a target data is a member of [Table] the training dataset.
Table 3 summarizes the accuracy of the inference attack for CIFAR-10 dataset, under the federated learning setup where n train training data is equally distributed into n = 10 clients.
The attack accuracy reaches near 70% for federated averaging, while SA and CCESA have the attack accuracy of near 50% , similar to the performance of the random guess.
This shows that both
SA and CCESA do not reveal any clue on the local training set, which secures data privacy.
C ONCLUSION
We devised communication-computation efﬁcient secure aggregation (CCESA) which successfully preserves the data privacy of federated learning in a highly resource-efﬁcient manner.
Based on graph-theoretic analysis, we showed that using O ( n log n ) resources is sufﬁcient for guaranteeing reliability and privacy of the proposed system with n clients, which is much smaller than O ( n 2 ) resources used in the conventional wisdom.
Our experiments on real datasets, measuring the test accuracy and the privacy leakage, show that CCESA requires only 50% of resources than the conventional wisdom, to achieve the same level of reliability and privacy.
A D ETAILED DESCRIPTION OF THE CCESA PROTOCOL
Input: Number of clients n , assignment graph G , privacy thresholds t i of all clients i ∈ [ n ] , local models θ i of all clients i ∈ [ n ] , Difﬁe-Hellman key pairs ( c PKi , c SKi ) , ( s PKi , s SKi ) of all clients i ∈ [ n ] and corresponding key agreement function f , pseudo-random generator PRG
Step 0.
Advertise Keys
Client i :
Sends ( i, c PKi , s PKi ) to the server
Server:
Collects the messages from clients (denote this set of clients as V 1 )
Sends { ( i, c PKi , s PKi ) } i ∈
Adj ( j ) ∩ V 1 to all clients j ∈ V 1 ;
Step 1.
Share Keys
Client i :
Generates a random element b i
Applies t i -out-of- ( | Adj ( i ) | + 1) secret sharing schemes to b i and s SK
i b i ( t i ,
| Adj ( i ) | +1)
−−−−−−−−−→ ( b i,j ) j ∈
( Adj ( i )) ∪{ i } , s SKi ( t i ,
| Adj ( i ) | +1)
−−−−−−−−−→ ( s SKi,j )
j ∈ Adj ( i ) ∪{ i }
Encrypts [ b i,j , s SKi,j ] to [ ¯ b i,j , ¯ s SKi,j ] using the authenticated encryption with key f ( c PKj , c SKi )
Sends { ( i, j, ¯ b i,j , ¯ s SKi,j ) } j ∈ Adj ( i ) ∩ V 1 to the server
Server:
Collects the messages from clients (denote this set of clients as V 2 )
Sends { ( i, j, ¯ b i,j , ¯ s SKi,j ) } i ∈
Adj ( j ) ∩ V 2 to all clients j ∈ V
Step 2.
Masked Input Collection
Client i :
Computes s i,j = f ( s PKj , s SKi ) and ˜
[Equation]
Sends ( i, ˜ θ i ) to the server
Server:
Collects the messages from clients (denote this set of clients as V 3 )
Sends V 3 to all clients j in V
Step 3. Unmasking
Client i :
Decrypts ¯ b i,j with key f ( c PKj , c SKi ) to obtain b i,j for all j ∈ Adj ( i ) ∩ V
Decrypts ¯ s SKi,j with key f ( c PKj , c SKi ) to obtain s SKi,j for all j ∈ Adj ( i )
∩
( V 2 \ V 3 )
Sends { b i,j } j ∈ Adj ( i ) ∩ V 3 , { s SKi,j } j ∈ Adj ( i ) ∩
( V 2 \ V 3 ) to the server
Server:
Collects the messages from clients
Reconstructs b i from { b i,j } j ∈ Adj ( i ) ∩ V 3 for all i ∈ V
Reconstructs s SKi from { s SKi,j } j ∈ Adj ( i ) ∩
( V 2 \ V 3 ) for all i ∈ V 2 \ V
Computes s i,j = f ( s PKj , s SKi ) for all j ∈ Adj ( i ) ∩ V
Computes the aggregated sum of local models (cid:80)
[Equation]
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
B A DDITIONAL EXPERIMENTAL RESULTS B. R ELIABILITY
In Fig.
4 of the main paper we provided the experimental results on the reliaiblity of CCESA on
CIFAR-10 dataset.
Similarly, Fig.
B.1 shows the reliability of CCESA in AT&T Face dataset, where the model is trained over n = 40 clients.
We plotted the test accuracies of SA and the suggested
CCESA( n, p ) for various p .
In both settings of q total , selecting p = 0 .
7 is sufﬁcient to achieve the test accuracy performance of SA when the system is trained for 50 rounds.
Thus, the required communication/computational resources for guaranteeing the reliability, which is proportional to np , can be reduced to 70% of the conventional wisdom in federated learning.
B. P RIVACY
In Section 5.3 and Fig. 2 of the main paper we provided the experimental results on the AT&T Face dataset, under the model inversion attack.
In Fig.
B.2, we provide additional experimental results on the same dataset for different participants.
Similar to the result in Fig.
2, the model inversion attack successfully reveals the individuals identity in federated averaging (McMahan et al., 2017), while the privacy attack is not effective in both SA and the suggested CCESA.
In the main manuscript, we have also considered another type of privacy threat called membership inference attack (Shokri et al., 2017), where the attacker observes masked local model ˜ θ
i sent from client i to the server, and guesses whether a particular data is the member of the training set.
We measured three types of performance metrics of the attacker: accuracy (the fraction of the records correctly estimated the membership), precision (the fraction of the responses inferred as members of the training dataset that are indeed members) and recall (the fraction of the training data that the attacker can correctly infer as members).
Table 3 in the main manuscript summarizes the attack accuracy result, while Table B.1 shows the attack precision for CIFAR-10 dataset.
We also observed that recall is close to 1 for all schemes.
Similar to the results on the attack accuracy, Table B.1 shows that the attack precision of federated averaging reaches near 70% , while that of SA and CCESA remain around the baseline performance of random guess.
This shows that both SA and CCESA do not reveal any clue on the training set.
[Figure]
C
P ROOFS
C
P ROOF OF T HEOREM
Proof.
Note that the sum of masked local models obtained by the server is expressed as
[Equation]
where
[Equation]
Here
[Equation]
z can be rewritten as
[Equation]
where ( a ) and ( b ) come from s i,j = s j,i .
In order to obtain the sum of unmasked local models (cid:80) i ∈ V 3 θ i from the sum of masked local models (cid:80) i ∈
V 3 ˜
θ i , the server should cancel out all the random terms in (cid:80) i ∈ V 3 PRG ( b i )
+ z .
In other words, the server should reconstruct b i for all i ∈ Vand s SKj for all j ∈ V + 3 \ V 3 .
Since the server can obtain | ( Adj ( i ) ∪ { i } ) ∩ V 4 | secret shares of client i in Step 3 , | ( Adj ( i ) ∪ { i } ) ∩ V 4 | ≥ t i for all i ∈ V + 3 is a sufﬁcient condition for reliability.
Now we prove the converse part by contrapositive.
Suppose there exists i ∈ V +3 such that | ( Adj ( i ) ∪ { i } ) ∩ V 4 | < t i .
In this case, note that the server cannot reconstruct both s SKi and
b i from the shares.
If i ∈ V 3 , the server cannot subtract PRG ( b i ) from (cid:80) i ∈ V 3 ˜ θ i .
As a result, the server cannot obtain (cid:80) i ∈
V 3 θ i .
If i ∈ V +3 \ V 3 , the server cannot subtract PRG ( s i,j ) for all j ∈ V 3 since the server does not have any knowledge of neither s SKi nor s SKj .
Therefore, the server cannot compute (cid:80) i ∈ V 3 θ i , which completes the proof.
C. P ROOF OF T HEOREM
Proof.
Let T ⊂ V 3 be an arbitrary set of clients satisfying T / ∈ { ∅ , V 3 } .
It is sufﬁcient to prove the following statement: given a connected graph G 3 , an eavesdropper cannot obtain the partial sum of local models (cid:80) i ∈T θ i from the sum of masked models (cid:80) i ∈T ˜ θ i . More formally, we need to prove
[Equation]
Note that the sum of masked local models (cid:80) i [Equation] ˜ θ i accessible to the eavesdropper is expressed as
[Equation]
where
[Equation]
Here, ( a ) and ( b ) come from s i,j = s j,i .
If G 3 = ( V 3 , E 3 ) is connected, there exists an edge e =
{ p, q } such that p ∈ T and q ∈ ( V 3 \T )
As a consequence, pseudorandom term PRG ( s p,q ) is included in z , and its coefﬁcient c p,q is determined as 1 (if p < q ), or − 1 (if p > q ). Note that equation (6) can be rewritten as [Equation] where r is the sum of pseudorandom terms which do not include PRG ( s p,q ).
In order to unmask PRG ( s p,q ) , the eavesdropper need to know at least one of the secret keys of clients p and q .
However, the eavesdropper cannot obtain any shares of the secret keys since the server do not request the shares of s SKp and s SKq in step 3. Therefore, due to the randomness of pseudorandom generator,
H ( (cid:80) i ∈T θ i | (cid:80) i ∈T ˜ θ i ) = H ( (cid:80) i ∈T θ i ) holds, which completes the proof.
C. P ROOF OF T HEOREM
Proof.
Consider Erd˝os-R´enyi assignment graph G ∈ G ( n, p ) .
Let N i :=
| Adj ( i ) | be the degree of node i , and X i :=
| Adj ( i ) ∩ V 4 | be the number of clients (except client i ) that successfully send the shares of client i to the server in Step 3 .
Then, N i and X i follow the binomial distributions
N i ∼ B ( n − 1 , p ) , X i ∼ B ( N i , (1 − q ) 4 ) =
B ( n − 1 , p (1 − q ) 4 ) , respectively.
By applying Hoeffding’s inequality on random variable
X
i , we obtain
[Equation]
Let E be the event that the system is not reliable, i.e. , the sum of local models (cid:80) i ∈ V 3 θ i is not reconstructed by the server, and E i be the event {| ( Adj ( i ) ∪ { i } )
∩ V 4 | < t } , i.e. , a secret of client i is not reconstructed by the server.
For a given p > t +
[Equation]
, we obtain
[Equation]
where ( a ) comes from Theorem 1.
Therefore, we conclude that CCESA( n, p ) is asymptotically almost surely (a.a.s.) reliable if p > t +
[Equation]
Furthermore, based on the parameter selection rule of t in Section F, we obtain a lower bound on p as
[Equation]
Rearranging the above inequality with respect to p yields
[Equation]
C. P ROOF OF T HEOREM
Proof.
Let X := | V 3 | be the number of clients sending its masked local model in Step 2 .
Then, X follows Binomial random variable B ( n, (1 − q ) 3 ) .
Given assignment graph G of CCESA( n, p ), note that the induced subgraph G 3 =
G − V \ V 3 is an Erd ˝ os-R ´ enyi graph G ( X, p ) .
First, we prove
[Equation]
if p > p (cid:63) = ln
[Equation] [Equation]
q
[Equation]
3 −√ n ln n
[Equation]
) (cid:100) n (
[Equation]
q [Equation] [Equation] n ln n
[Equation]
(1 + (cid:15) ) . The left hand side of (7) can be rewritten as
[Equation]
Here, we use a well-known property of Erd ˝ os-R ´ enyi graph: G ( l, p ) is asymptotically almost surely (a.a.s.)
connected if p > (1+ (cid:15) )
ln l l for some (cid:15) > 0 .
Since ln ll is a decreasing function, G ( l, p ) is a.a.s.
connected for all l ∈ [ n (1 − q ) 3 − √ n ln n, n (1 − q ) 3 + √ n ln n ] when p > ln( (cid:100) n (1 − q ) 3 −√ n ln n (cid:101) ) (cid:100) n (1 − q ) 3 −√ n ln n (cid:101)
[Equation]
Thus, for given p > p (cid:63) , we can conclude
[Equation]
is
[Equation]
Now, we will prove that CCESA( n, p ) is a.a.s.
private when p > p (cid:63) .
The probability that
CCESA( n, p ) is private is lower bounded by
[Equation]
where ( a ) comes from Theorem 2 and ( b ) comes from Hoeffding’s inequality
[Equation]
which completes the proof.
C. P ROOF OF T HEOREM
Proof.
Consider an Erd˝os-R´enyi assignment graph G ∈ G ( n, p ) .
Let N i :=
| Adj ( i ) | be the degree of node i , and X i :=
| Adj ( i ) ∩ V 4 | be the number of clients (except client i ) that successfully send the shares of client i to the server in Step 3 .
Then, N i and X i follow the binomial distributions
[Equation] [Equation]
respectively.
Let E i be an event {| ( Adj ( i ) ∪ { i } )
∩ V 4 | < t } , i.e. , a secret of client i is not reconstructed by the server.
We obtain an upper bound on P ( E i ) as
[Equation]
where ( a ) comes from Chernoff bound on the binomial distribution.
Thus, P ( r ) e is upper bounded by
P
[Equation]
where ( b ) comes from Theorem 1.
C. P ROOF OF T HEOREM
Proof.
Let P dc ( n, p ) be the probability of an event that Erd ˝ os-R ´ enyi graph G ∈ G ( n, p ) is disconnected.
Then, P dc ( n, p ) is upper bounded as follows.
[Equation]
there exists a subset of k nodes that is disconnected }
[Equation]
there exists a subset of k nodes that is disconnected )
[Equation]
Therefore, P ( p ) e is upper bounded by
P ( p ) e ( a ) ≤ P ( G 3 =
G − V \ V 3 is disconnected )
=
n (cid:88) m =
P ( G 3 has m vertices
[Equation]
P dc ( m
[Equation]
p )
=
n (cid:88) m
[Equation]
(cid:18) n m (cid:19) (
[Equation]
q )
[Equation]
m (
[Equation] (1 − q ) 3 ) ( n − m ) · P dc ( m, p ) = n (cid:88) m = (cid:18) n m (cid:19) (1 − q
[Equation]
m (1 − (1 − q ) 3 ) ( n − m )
[Equation]
[Equation]
k
[Equation]
m k
[Equation]
(1 − p ) k
( m
[Equation]
k
[Equation]
where ( a ) comes from Theorem 2.
D R EQUIRED RESOURCES OF CCESA D. C OMMUNICATION COST
Here, we derive the additional communication bandwidth B CCESA used at each client for running
CCESA, compared to the bandwidth used for running federated averaging (McMahan et al., 2017).
We consider the worst-case scenario having the maximum additional bandwidth, where no client fails during the operation.
The required communication bandwidth of each client is composed of four parts.
First, in Step 0 , each client i sends two public keys to the server, and receives 2 | Adj ( i ) | public keys from other clients.
Second, in Step 1 , each client i sends encrypted 2 | Adj ( i ) | shares to other nodes, and receives 2 | Adj ( i ) | shares from other nodes through the server.
Third, in Step 2 , each client i sends a masked data
y i of mR bits.
Here, m is the dimension of model parameters where each parameter is represented in R bits.
Fourth, in Step 3 , each client i sends | Adj ( i ) | + 1 shares to the server.
Therefore, total communication bandwidth of client i can be expressed as
[Equation]
where a K and a S are the number of bits required for exchanging public keys and the number of bits in a secret share.
Since each client i requires mR bits to send the private vector θ i in the federated averaging (McMahan et al., 2017), we have
[Equation]
If we choose the connection probability p
= (1 + (cid:15) ) p (cid:63) for a small (cid:15) > 0 , we have B CCESA =
O ( √ n log n ) , where p (cid:63) is deﬁned in (5).
Note that the additional bandwidth B SA required for SA can be similarly obtained as B SA = 2 na K + (5 n − 4) a S having B SA = O ( n ) .
Thus, we have
B CCESA
B SA
→as n increases, showing the scalability of CCESA.
These results are summarized in Table 1 in the main manuscript.
D. C OMPUTATIONAL COST
We evaluate the computational cost of CCESA.
Here we do not count the cost for computing the signatures since it is negligible.
First, we derive the computational cost of each client.
Given the number of model parameters m and the number of clients n , the computational cost of client i is composed of three parts: (1) computing 2 | Adj ( i ) | key agreements, which takes O ( | Adj ( i ) | ) time, (2) generating shares of t i -out-of-
| Adj ( i ) | secret shares of s SKi and b i , which takes O ( | Adj ( i )
| 2 ) time, and (3) generating masked local model ˜ θ i , which requires O ( m | Adj ( i ) | ) time.
Thus, the total computational cost of each client is obtained as O ( | Adj ( i ) |
+ m | Adj ( i ) | ) .
Second, the server’s computational cost is composed of two parts: (1) reconstructing t
i -out-of-
| Adj ( i ) | secrets from shares for all clients i ∈
[ n ] , which requires O ( | Adj ( i ) | 2 ) time, and (2) removing masks from masked sum of local models (cid:80) ni
=1 ˜ θ i , which requires O ( m | Adj ( i ) | 2 ) time in the worst case.
As a result, the total computational cost of the server is O ( m | Adj ( i ) | 2 ) .
If we choose p = (1 + (cid:15) ) p (cid:63) for small (cid:15) > 0 , the total computational cost per each client is O ( n log n + m √ n log n ) , while the total computation cost of the server is O ( mn log n ) .
The computational cost of SA can be obtained in a similar manner, by setting Adj ( i ) = n − 1 ; each client requires O ( n 2 + mn ) time while the server requires O ( mn 2 ) time.
These results are summarized in Table 1 in the main manuscript.
E R ELIABILITY AND PRIVACY OF CCESA
Here, we analyze the asymptotic behavior of probability that a system is reliable/private.
In our analysis, we assume that the connection probability is set to p (cid:63) and the parameter t used in the secret sharing is selected based on the rule in Section F. First, we prove that a system is reliable with probability ≥ 1 − O ( ne − n log n ) .
Using Theorem 5, the probability that a system is reliable can be directly derived as
[Equation]
Using the fact that Kullback-Leibler divergence term satisﬁes
[Equation]
we conclude that CCESA( n, p (cid:63) ) is reliable with probablilty ≥ 1 − O ( ne −√ n log n ) .
Now we prove that a system is private with probability ≥ 1 − O ( n − C ) for an arbitrary C > 0 .
Using
Theorem 6, the probability that a system is private can be obtained as
[Equation]
where a m = (cid:0) nm (cid:1) (1 − q ) 3 m (1 − (1 − q ) 3 ) ( n − m ) and b m = (cid:80) (cid:98) m/ 2 (cid:99) k
=
[Equation]
Note that the summation term (cid:80) nm =0 a m b m can be broken up into two parts: (cid:80) m th m =
a m b m and (cid:80) nm =
m th +1 a m b m , where m th = (cid:98) n (1 − q ) 3 / 2 (cid:99) .
In the rest of the proof, we will prove two lemmas, showing that (cid:80) m th m =
a m b m =
O ( e − n ) and (cid:80)
[Equation]
respectively
[Equation]
Lemma 1.
[Equation]
Proof.
Since b m ≤ 1 for all m , we have
[Equation]
Note that a m = P ( X = m ) holds for binomial random variable X
= B ( n, (1 − q ) 3 ) .
By utilizing
Hoeffding’s inequality, we have
[Equation]
Therefore, we conclude that (cid:80) m th m =
a m b m =
O ( e − n ) .
Lemma 2.
[Equation]
Proof.
Since a m ≤ 1 for all m , we have
[Equation]
Let C > 0 be given.
Then, the upper bound on b m can be obtained as
[Equation]
where λ = p (cid:63) n/ log n , c m = (cid:0) mk (cid:1) (cid:80) k
(cid:63) k
=1 m − λk ( m − k ) /m ,
d m = (cid:0) mk (cid:1) (cid:80) (cid:98) m/ 2 (cid:99) k
= k (cid:63) +1 m − λk ( m − k ) /m , and k (cid:63) = (cid:98) m (1 − C +2 λ ) (cid:99) for some C > 0 .
The ﬁrst part of summation is upper bounded by c m =
[Equation]
(cid:88)
[Equation]
The second part of summation, we will use the bound (cid:0) nk (cid:1) ≤ ( enk ) k .
Using this bound, d m is upper bounded by
[Equation]
For sufﬁciently large λ , we have em − λ/ 2 / (1 − λ − 1 ( C + 2)) <
δ for some δ < 1 .
Therefore, d m is upper bounded by
[Equation]
where C (cid:48)
= (1 − λ − 1 ( C + 2)) > 0 .
Combining upper bounds on c m and d m , we obtain b m =
O ( m − ( C +1) ) .
Since b m is a decreasing function of m , n (cid:88)
[Equation]
holds where ( a ) comes from m th = (cid:98) n (1 − q ) 3 / 2 (cid:99) .
Combining the above two lemmas, we conclude that CCESA( n, p (cid:63) ) is private with probability ≥ 1 − O ( n − C ) for arbitrary C > 0 .
These results on the reliability and the privacy are summarized in
Table 1 of the main manuscript.
F D ESIGNING THE PARAMETER t FOR THE SECRET SHARING
Here we provide a rule for selecting parameter t used in the secret sharing.
In general, setting t to a smaller number is better for tolerating dropout scenarios.
However, when t is excessively small, the system is vulnerable to the unmasking attack of adversarial server; the server may request shares of b i and s SKi to disjoint sets of remaining clients simultaneously, which reveals the local model θ i to the server.
The following proposition provides a rule of designing parameter t to avoid such unmasking attack.
Proposition 1 (Lower bound on t ) .
For CCESA( n, p ), let t > ( n − 1) p + √
[Equation]
be given.
Then, the system is asymptotically almost surely secure against the unmasking attack.
Proof.
Let E be the event that at least one of local models are revealed to the server, and E i be the event that i th local model θ i is revealed to the server.
Note that θ i is revealed to the server if t clients send the shares of b
i and other t clients send the shares of s SKi in Step 3 .
Therefore,
[Equation]
where (a) comes from Hoeffding’s inequality of binomial random variable.
As a result, we obtain
P ( E ) =
P ( ∪ i ∈ [ n ] E i ) ≤ (cid:88) i ∈ [ n ]
P ( E i )
=
nP ( E 1 ) =
n ( n − 1)n →∞ −−−−→ 0 , which completes the proof.
As stated above, setting t to a smaller number is better to tolerate the dropout of multiple clients.
Thus, as in the following remark, we set t to be the minimum value avoiding the unmasking attack.
Remark 4 (Design rule for t ) .
Throughout the paper, we set t = (cid:100) ( n − 1) p + √ ( n − 1)log( n − 1)+1(cid:101) for
CCESA( n, p ), in order to secure a system against the unmasking attack and provide the maximum tolerance against dropout scenarios.
G D ETAILED EXPERIMENTAL SETUP G. AT & T FACE DATASET
AT&T Face dataset contains images of 40 members.
We allocated the data to n = 40 clients participating in the federated learning, where each client contains the images of a speciﬁc member.
This experimental setup is suitable for the practical federated learning scenario where each client has its own image and the central server aggregates the local models for face recognition.
Following the previous work (Fredrikson et al., 2015) on the model inversion, we used softmax regression for the classiﬁcation.
Both the number of local training epochs and the number of global aggregation rounds are set to E local =
E global = 50 , and we used the SGD optimizer with learning rate γ = 0 .
G. CIFAR-10 DATASET G.2. R ELIABILITY EXPERIMENT IN F IG .
We ran experiments under the federated learning setup where 50000 training images are allocated to n = 1000 clients.
Here, we considered two scenarios for data allocation: one is partitioning the data in the i.i.d.
manner (i.e., each client randomly obtains 50 images), while the other is non-i.i.d.
allocation scenario.
For the non-i.i.d.
scenario, we followed the procedure of (McMahan et al., 2017).
Speciﬁcally, the data is ﬁrst sorted by its category, and then the sorted data is divided into 2000 shards.
Each client randomly chooses 2 shards for its local training data.
Since each client has access to at most 2 classes, the test accuracy performance is degraded compared with the i.i.d. setup.
For training the classiﬁer, we used VGG-11 network and the SGD optimizer with learning rate γ = 0 .
1 and momentum β = 0 .
The local training epoch is set to E local = 3 .
G.2. P RIVACY EXPERIMENTS IN T ABLE 3 AND T ABLE B.
We conducted experiments under the federated learning setup where n train training images are assigned to n = 10 clients.
We considered i.i.d.
data allocation setup where each client randomly obtains n train / 10 training images.
The network architecture, the optimizer, and the number of local training epochs are set to the options used in Sec.
G.2.1.
G. C ONNECTION PROBABILITY SETUP IN F IG .
In Fig.
3, we select different connection probabilities p = p (cid:63) ( n, q total ) for various n and q total , where p (cid:63) is deﬁned in (5). The detailed values of connection probability p are provided in Table G.2. G. R UNNING TIME EXPERIMENT IN T ABLE
We implemented CCESA algorithm in python. For symmetric authenticated encryption, we use AES-GCM with 128-bit keys in Crypto.
Cipher package.
For the pseudorandom generator, we use randint function (input: random seed, output: random integer in the ﬁeld of size 2 16 ) in numpy.random package. For key agreement, we use Elliptic-Curve Difﬁe-Hellman over the NIST SP800-56 curve composed with a SHA-256 hash function. For secret sharing, we use standard t -out-of- n secret sharing (Shamir, 1979).
[Table]