Original: Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning
Revised: Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning

Original: Abstract
Revised: Abstract

Original: Secure aggregation is a critical component in federated learning (FL), whichenables the server to learn the aggregate model of the users without observingtheir local models.
Revised: Secure aggregation is a critical component in federated learning (FL), whichenables the server to learn the aggregate model of the users without observingtheir local models.

Original: Conventionally, secure aggregation algorithms focus only onensuring the privacy of individual users in a single training round.
Revised: Conventionally, secure aggregation algorithms focus only onensuring the privacy of individual users in a single training round.

Original: We contend thatsuch designs can lead to signiﬁcant privacy leakages over multiple training rounds,due to partial user selection/participation at each round of FL.
Revised: We contend thatsuch designs can lead to signiﬁcant privacy leakages over multiple training rounds,due to partial user selection/participation at each round of FL.

Original: In fact, we show thatthe conventional random user selection strategies in FL may lead to leaking users’individual models within a number of rounds that is linear in the number of users.
Revised: In fact, we show thatthe conventional random user selection strategies in FL may lead to leaking users’individual models within a number of rounds that is linear in the number of users.

Original: To address this challenge, we introduce a secure aggregation framework, MultiRoundSecAgg, with multi-round privacy guarantees.
Revised: To address this challenge, we introduce a secure aggregation framework, MultiRoundSecAgg, with multi-round privacy guarantees.

Original: In particular, we introduce anew metric to quantify the privacy guarantees of FL over multiple training rounds,and develop a structured user selection strategy that guarantees the long-termprivacy of each user (over any number of training rounds).
Revised: In particular, we introduce anew metric to quantify the privacy guarantees of FL over multiple training rounds,and develop a structured user selection strategy that guarantees the long-termprivacy of each user (over any number of training rounds).

Original: Our framework alsocarefully accounts for the fairness and the average number of participating users ateach round.
Revised: Our framework alsocarefully accounts for the fairness and the average number of participating users ateach round.

Original: Our experiments on MNIST, CIFAR- 10 and CIFAR- 100 datasets inthe IID and the non-IID settings demonstrate the performance improvement overthe baselines, both in terms of privacy protection and test accuracy.
Revised: Our experiments on MNIST, CIFAR- 10 and CIFAR- 100 datasets inthe IID and the non-IID settings demonstrate the performance improvement overthe baselines, both in terms of privacy protection and test accuracy.

Original:  Introduction
Revised:  Introduction

Original:  Federated learning (FL) enables collaborativetraining of machine learning models over thedata collected and stored locally by multipledata-owners.
Revised: [Figure] Federated learning (FL) enables collaborativetraining of machine learning models over thedata collected and stored locally by multipledata-owners.

Original: The training in FL is typicallycoordinated by a central server who maintains aglobal model that is updated locally by the users.
Revised: The training in FL is typicallycoordinated by a central server who maintains aglobal model that is updated locally by the users.

Original: The local updates are then aggregated by theserver to update the global model.
Revised: The local updates are then aggregated by theserver to update the global model.

Original: Throughoutthe training process, the users never share theirdata with the server, i.e., the data is always kepton device, rather, they only share their localupdates.
Revised: Throughoutthe training process, the users never share theirdata with the server, i.e., the data is always kepton device, rather, they only share their localupdates.

Original: However, as has been shown recently,the local models may still reveal substantialinformation about the local datasets, and theprivate training data can be reconstructed fromthe local models through inference or inversionattacks (see e.g., [11, 26, 42, 12]).
Revised: However, as has been shown recently,the local models may still reveal substantialinformation about the local datasets, and theprivate training data can be reconstructed fromthe local models through inference or inversionattacks (see e.g., [8, 22, 36, 9]).

Original: To prevent such information leakage, secure aggregation protocols are proposed (e.g., [4, 31, 15,40, 2, 38, 30]) to protect the privacy of the local models, both from the server and the other users,while still allowing the server to learn their aggregate.
Revised: To prevent such information leakage, secure aggregation protocols are proposed (e.g., [2, 27, 12,34, 1, 32, 26]) to protect the privacy of the local models, both from the server and the other users,while still allowing the server to learn their aggregate.

Original: More speciﬁcally, the secure aggregationprotocols ensure that, at any given round, the server can only learn the aggregate model of the users,and beyond that no further information is revealed about the individual model.
Revised: More speciﬁcally, the secure aggregationprotocols ensure that, at any given round, the server can only learn the aggregate model of the users,and beyond that no further information is revealed about the individual model.

Original: Secure aggregation protocols, however, only ensure the privacy of the individual users in a singletraining round , and do not consider their privacy over multiple training rounds [4, 2, 31, 32].
Revised: Secure aggregation protocols, however, only ensure the privacy of the individual users in a singletraining round , and do not consider their privacy over multiple training rounds.

Original: Onthe other hand, due to partial user selection [7, 5, 6, 28], the server may be able to reconstruct theindividual models of some users using the aggregated models from the previous rounds.
Revised: On the other hand,due to partial user selection [5, 3, 4, 24], the server may be able to reconstruct the individual modelsof some users using the aggregated models from the previous rounds.

Original: In fact, weshow that after a sufﬁcient number of rounds, all local models can be recovered with a high accuracyif the server uniformly chooses a random subset of the users to participate at every round.
Revised: In fact, we show that aftera sufﬁcient number of rounds, all local models can be recovered with a high accuracy if the server uniformly chooses a random subset of the users to participate at every round.

Original: As shown in Fig.1, performing model inversion attack [12] with the recovered local models yields reconstructedimages with a similar quality as the original images.
Revised: As shown in Fig.1,performing model inversion attack [9] with the recovered local models yields reconstructed imageswith a similar quality as the original images.

Original: Contributions .
Revised: Contributions .

Original: As such motivated, we study long-term user privacy in FL.
Revised: As such motivated, we study long-term user privacy in FL.

Original: Speciﬁcally, ourcontributions are as follows.
Revised: Speciﬁcally, ourcontributions are as follows

Original: We introduce a new metric to capture long-term privacy guaranteesfor secure aggregation protocols in FL for the ﬁrst time.
Revised: We introduce a new metric to capture long-term privacy guarantees in FL for the ﬁrst time.

Original: This long-term privacy  requires that the server cannot reconstruct anyindividual model using the aggregated models from any number of training rounds.
Revised: This long-term privacy condition requires that the server cannot reconstruct any individual model using the aggregated models from any number of training rounds.

Original: Using this 
Revised: Using this metric, we show that the

Original: metric, we show that the conventional random  selection schemes can result in leaking the local
Revised: conventional random user selection schemes can result in leaking the individual user models after

Original: models after a sufﬁcient number of rounds, even if secure aggregation is employed at each round.
Revised: a sufﬁcient number of rounds, even if secure aggregation is employed at each round.

Original: We propose Multi-RoundSecAgg, a privacy-preserving structured user selection strategy thatensures the long-term privacy of the individual users over any number of training rounds.
Revised: We propose Multi-RoundSecAgg, a privacy-preserving structured user selection strategy that ensures the long-term privacy of the individual users over any number of training rounds.

Original: Thisstrategy also takes into account the fairness of the selection process and the average number of
Revised: This strategy also takes into account the fairness of the selection process and the average number of

Original: participating users at each round.
Revised: participating users at each round.

Original: We demonstrate that Multi-RoundSecAgg creates a trade-off between the long-term privacyguarantee and the average number of participating users.
Revised: We demonstrate that Multi-RoundSecAgg creates a trade-off between the long-term privacy guarantee and the average number of participating users.

Original: In particular, as the average number ofparticipating users increases, the long-term privacy guarantee becomes weaker.4.
Revised: In particular, as the average number ofparticipating users increases, the long-term privacy guarantee becomes weaker.4.

Original: We provide the convergence analysis of Multi-RoundSecAgg, which shows that the long-termprivacy guarantee and the average number of participating users control the convergence rate.
Revised: We provide the convergence analysis of Multi-RoundSecAgg, which shows that the long-termprivacy guarantee and the average number of participating users control the convergence rate.

Original: Theconvergence rate is maximized when the average number of participating users is maximized.(e.g.
Revised: Theconvergence rate is maximized when the average number of participating users is maximized.(e.g.

Original: , the random user selection strategy maximizes the average number of participating users atthe expense of not providing long-term privacy guarantees).
Revised: , the random user selection strategy maximizes the average number of participating users at the expense of not providing long-term privacy guarantees).

Original: As we require stronger long-termprivacy guarantees, the average number of participating users decreases and a larger number oftraining rounds is required to achieve the same level of accuracy as the random selection strategy.
Revised: As we require stronger long-term privacy guarantees, the average number of participating users decreases and a larger number of training rounds is required to achieve the same level of accuracy as the random selection strategy.

Original: Finally, our experiments in both IID and non-IID settings on MNIST, CIFAR-
Revised: Finally, our experiments in both IID and non-IID settings on MNIST, CIFAR-

Original: 10 and CIFAR-
Revised: 10 and CIFAR-

Original: 100datasets demonstrate that Multi-RoundSecAgg achieves almost the same test accuracy comparedto the random selection scheme while providing better long-term privacy guarantees.
Revised: 100datasets demonstrate that Multi-RoundSecAgg achieves almost the same test accuracy comparedto the random selection scheme while providing better long-term privacy guarantees

Original:  Related Work
Revised:  Related Work

Original: The underlying principle of the secure aggregation protocol in [4] is that each pair of users exchange apairwise secret key which they can use to mask their local models before sharing them with the server.
Revised: The underlying principle of the secure aggregation protocol in [2] is that each pair of users exchange apairwise secret key which they can use to mask their local models before sharing them with the server.

Original: The pairwise masks cancel out when the server aggregates the masked models, allowing the server toaggregate the local models.
Revised: The pairwise masks cancel out when the server aggregates the masked models, allowing the server toaggregate the local models.

Original: These masks also ensure that the local models are kept private, i.e., nofurther information is revealed beyond the aggregate of the local models.
Revised: These masks also ensure that the local models are kept private, i.e., nofurther information is revealed beyond the aggregate of the local models.

Original: This protocol, however,incurs a signiﬁcant communication cost due to exchanging and reconstructing the pairwise keys.
Revised: This protocol, however,incurs a signiﬁcant communication cost due to exchanging and reconstructing the pairwise keys.

Original: Recently, several works have developed computation and communication-efﬁcient protocols [31, 15,2 , 35, 8, 10, 38], which are complementary to and can be combined with our work.
Revised: Recently, several works have developed computation and communication-efﬁcient protocols [27, 12,1, 30, 6, 7, 32], which are complementary to and can be combined with our work.

Original: Another line ofwork focused on designing partial user selection strategies to overcome the communication bottleneckin FL while speeding up the convergence by selecting the users based on their local loss [7, 5, 6, 28].
Revised: Another line ofwork focused on designing partial user selection strategies to overcome the communication bottleneckin FL while speeding up the convergence by selecting the users based on their local loss [5, 3, 4, 24].

Original: Previous works, either on secure aggregation or on partial user selection, however, do not considermitigating the potential privacy leakage as a result of partial user participation and the server observingthe aggregated models across multiple training rounds.
Revised: Previous works, either on secure aggregation or on partial user selection, however, do not considermitigating the potential privacy leakage as a result of partial user participation and the server observingthe aggregated models across multiple training rounds.

Original: While [27] pointed out to the privacy leakageof secure aggregation, mitigating this leakage has not been considered and our work is the ﬁrst secureaggregation protocol to address this challenge.
Revised: While [23] pointed out to the privacy leakageof secure aggregation, mitigating this leakage has not been considered and our work is the ﬁrst toaddress this challenge.

Original: Differential privacy (DP), in which each user adds artiﬁcial noises to the local models, can be one ofthe potential solution to protect the privacy leakage over the multiple rounds [9, 1, 37, 3, 16].

Original: In DP,however, the privacy guarantee sacriﬁces the model performance, which is known as a privacy-utilitytrade-off.

Original: It is worth noting that secure aggregation and DP are complementary, i.e., all the beneﬁtsof DP can be applied to our approach by adding noise to the local models [3].

Original: In this paper, ourobjective is to understand the secure aggregation itself.

Original:  System Model
Revised:  System Model

Original: In this section, we ﬁrst describe the basic federated learning model in Section 3.1.
Revised: In this section, we ﬁrst describe the basic federated learning model in Section 3.1.

Original: Next, we introducethe multi-round secure aggregation problem for federated learning and deﬁne the key multi
Revised: Next, we introducethe multi-round secure aggregation problem for federated learning and deﬁne the key metrics toevaluate the performance of a multi-round secure aggregation protocol in Section 3.2.

Original: 3.1 Basic Federated Learning Model
Revised: 3.1 Basic Federated Learning Model

Original: We consider a cross-device federated learning setup consisting of a server and 𝑁 users.
Revised: We consider a cross-device federated learning setup consisting of a server and 𝑁 users.

Original: User 𝑖 ∈ [ 𝑁 ]has a local dataset D 𝑖 consisting of 𝑚 𝑖 =
Revised: User 𝑖 ∈ [ 𝑁 ]has a local dataset D 𝑖 consisting of 𝑚 𝑖 =

Original: |D 𝑖 | data samples.
Revised: |D 𝑖 | data samples.

Original: The users are connected to each otherthrough the server, i.e., all communications between the users goes through the server [24, 4, 17].
Revised: The users are connected to each other through the server, i.e., all communications between the users goes through the server [20, 2, 13].

Original: The goal is to collaboratively learn a global model x with dimension 𝑑 , using the local datasets thatare generated, stored, and processed locally by the users.
Revised: The goal is to collaboratively learn a global model x with dimension 𝑑 , using the local datasets thatare generated, stored, and processed locally by the users.

Original: The training task can be represented byminimizing a global loss function,
Revised: The training task can be represented byminimizing a global loss function,

Original: [Equation]
Revised: [Equation]

Original: where 𝐿 𝑖 is the loss function of user 𝑖 and 𝑤 𝑖 ≥ 0 is a weight parameter assigned to user 𝑖 to specifythe relative impact of that user.
Revised: where 𝐿 𝑖 is the loss function of user 𝑖 and 𝑤 𝑖 ≥ 0 is a weight parameter assigned to user 𝑖 to specifythe relative impact of that user.

Original: A common choice for the weight parameters is 𝑤 𝑖 = 𝑚 𝑖
Revised: A common choice for the weight parameters is 𝑤 𝑖 = 𝑚 𝑖

Original: Wedeﬁne the optimal model parameters x ∗ and x ∗ 𝑖 as x ∗ =
Revised: Wedeﬁne the optimal model parameters x ∗ and x ∗ 𝑖 as x ∗ =

Original: arg min x ∈ R 𝑑 𝐿 ( x ) and x ∗ 𝑖 = arg min x ∈ R 𝑑 𝐿 𝑖 ( x ) .
Revised: arg min x ∈ R 𝑑 𝐿 ( x ) and x ∗ 𝑖 = arg min x ∈ R 𝑑 𝐿 𝑖 ( x ) .

Original: Federated Averaging with Partial User Participation.
Revised: Federated Averaging with Partial User Participation.

Original: To solve (1), the most common algorithmis the FedAvg (federated averaging) algorithm [24].
Revised: To solve (1), the most common algorithmis the FedAvg (federated averaging) algorithm [20].

Original: FedAvg is an iterative algorithm, where the modeltraining is done by repeatedly iterating over individual local updates.
Revised: FedAvg is an iterative algorithm, where the modeltraining is done by repeatedly iterating over individual local updates.

Original: At the beginning of traininground 𝑡 , the server sends the current state of the global model, denoted by x ( 𝑡 ) , to the users.
Revised: At the beginning of traininground 𝑡 , the server sends the current state of the global model, denoted by x ( 𝑡 ) , to the users.

Original: Eachround consists of two phases, local training and aggregation.
Revised: Eachround consists of two phases, local training and aggregation.

Original: In the local training phase, user 𝑖 ∈ [ 𝑁 ]updates the global model by carrying out 𝐸 ( ≥ 1 ) local stochastic gradient descent (SGD) steps andsends the updated local model x ( 𝑡 ) 𝑖 to the server.
Revised: In the local training phase, user 𝑖 ∈ [ 𝑁 ]updates the global model by carrying out 𝐸 ( ≥ 1 ) local stochastic gradient descent (SGD) steps andsends the updated local model x ( 𝑡 ) 𝑖 to the server.

Original: One of key features of cross-device FL is partialdevice participation.
Revised: One of key features of cross-device FL is partialdevice participation.

Original: Due to various reasons such as unreliable wireless connectivity, or battery issues,at any given round, only a fraction of the users are available to participate in the protocol.
Revised: Due to various reasons such as unreliable wireless connectivity, or battery issues,at any given round, only a fraction of the users are available to participate in the protocol.

Original: We referto such users as available users throughout the paper.
Revised: We referto such users as available users throughout the paper.

Original: In the aggregation phase, the server selects𝐾 ≤ 𝑁 users among the available users if this is possible and aggregates their local updates.
Revised: In the aggregation phase, the server selects𝐾 ≤ 𝑁 users among the available users if this is possible and aggregates their.

Original: Theserver updates the global model as follows
Revised: After receiving thelocal updates, the server updates the global model as follows

Original: [Equation]
Revised: [Equation]

Original: where S ( 𝑡 ) is the set of participating users at round 𝑡 , 𝑤 (cid:48)
Revised: where S ( 𝑡 ) is the set of participating users at round 𝑡 , 𝑤 (cid:48)

Original: 𝑖
Revised: 𝑖

Original: = 𝑤 𝑖 (cid:205) 𝑖 ∈S( 𝑡 ) 𝑤 𝑖 , and p ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is thecorresponding characteristic vector.
Revised: = 𝑤 𝑖 (cid:205) 𝑖 ∈S( 𝑡 ) 𝑤 𝑖 , and p ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is thecorresponding characteristic vector.

Original: That is, p ( 𝑡 ) denotes a participation vector at round 𝑡 whose 𝑖 -thentry is 0 when user 𝑖 is not selected and 1 otherwise.
Revised: That is, p ( 𝑡 ) denotes a participation vector at round 𝑡 whose 𝑖 -thentry is 0 when user 𝑖 is not selected and 1 otherwise.

Original: X ( 𝑡 ) denotes the concatenation of the weightedlocal models at round 𝑡 , i.e., X ( 𝑡 ) = (cid:2)
Revised: X ( 𝑡 ) denotes the concatenation of the weightedlocal models at round 𝑡 , i.e., X ( 𝑡 ) = (cid:2)

Original: 𝑤 (cid:48) 1 x ( 𝑡 ) 1 , . .
Revised: 𝑤 (cid:48) 1 x ( 𝑡 ) 1 , . .

Original: , 𝑤 (cid:48)
Revised: , 𝑤 (cid:48)

Original: 𝑁 x ( 𝑡 ) 𝑁 (cid:3) (cid:62) ∈ R
Revised: 𝑁 x ( 𝑡 ) 𝑁 (cid:3) (cid:62) ∈ R

Original: 𝑁 × 𝑑 .
Revised: 𝑁 × 𝑑 .

Original: Finally, the server broadcaststhe updated global model x ( 𝑡 + 1 ) to the users for the next round.
Revised: Finally, the server broadcaststhe updated global model x ( 𝑡 + 1 ) to the users for the next round.

Original: Threat Model.
Revised: Threat Model.

Original: Similar to the prior works on secure aggregation as [4, 15, 31], we consider thehonest-but-curious  model.
Revised: Similar to the prior works on secure aggregation as [2, 12, 27], we consider thehonest-but-curious server’s model.

Original: All participants follow the protocol honestly in this model, but try to learnas much as possible about the users.
Revised: The server follows the protocol honestly in this model, but triesto learn as much as possible about the users.

Original: At each round, the privacy of individual model x ( 𝑡 ) 𝑖 in (2) isprotected by secure aggregation such that the server only learns the aggregated model (cid:205)
Revised: At each round, the privacy of individual model x ( 𝑡 ) 𝑖 in(2) is protected by secure aggregation such that the server only learns the aggregate of the selectedmodels (cid:205)

Original: 𝑖 ∈S ( 𝑡 ) 𝑤 (cid:48)
Revised: 𝑖 ∈S ( 𝑡 ) 𝑤 (cid:48)

Original: 𝑖 x ( 𝑡 ) 𝑖 .
Revised: 𝑖 x ( 𝑡 ) 𝑖 .

Original: 3.2 Multi-round Secure Aggregation
Revised: 3.2 Multi-round Secure Aggregation

Original: Conventional secure aggregation protocols only consider the privacy guarantees over a single traininground.
Revised: Conventional secure aggregation protocols only consider the privacy guarantees over a single traininground.

Original: While secure aggregation protocols have provable privacy guarantees at any single round,in the sense that no information is leaked beyond the aggregate model at each round, the privacyguarantees do not extend to attacks that span multiple training rounds .
Revised: While secure aggregation protocols have provable privacy guarantees at any single round,in the sense that no information is leaked beyond the aggregate model at each round, the privacyguarantees do not extend to attacks that span multiple training rounds .

Original: Speciﬁcally, by using theaggregate models and participation information across multiple rounds, an individual model may bereconstructed.
Revised: Speciﬁcally, by using theaggregate models and participation information across multiple rounds, an individual model may bereconstructed.

Original: For instance, consider the following user participation strategy across three trainingrounds, p ( 1 ) =
Revised: For instance, consider the following user participation strategy across three trainingrounds, p ( 1 ) =

Original: [ 1 , 1 , 0 ] (cid:62) , p ( 2 ) =
Revised: [ 1 , 1 , 0 ] (cid:62) , p ( 2 ) =

Original: [ 0 , 1 , 1 ] (cid:62) , and p ( 3 ) =
Revised: [ 0 , 1 , 1 ] (cid:62) , and p ( 3 ) =

Original: [ 1 , 0 , 1 ] (cid:62) .
Revised: [ 1 , 0 , 1 ] (cid:62) .

Original: Assume a scenario where the localupdates do not change signiﬁcantly over time (e.g., models start to converge, or the server ﬁxes theglobal model over consecutive rounds), i.e., x 𝑖 = x ( 𝑡 ) 𝑖 for all 𝑖 ∈ [ 3 ] and 𝑡 ∈ [ 3 ] .
Revised: Assume a scenario where the localupdates do not change signiﬁcantly over time (e.g., models start to converge, or the server ﬁxes theglobal model over consecutive rounds), i.e., x 𝑖 = x ( 𝑡 ) 𝑖 for all 𝑖 ∈ [ 3 ] and 𝑡 ∈ [ 3 ] .

Original: Then, the server cansingle out individual model, e.g., x
Revised: Then, the server cansingle out individual model, e.g., x

Original: 1 = ( x ( 1 ) + x ( 3 ) − x ( 2 ) )/ 2 .
Revised: 1 = ( x ( 1 ) + x ( 3 ) − x ( 2 ) )/ 2 .

Original: Similarly, the server can single out allindividual models x 𝑖 , even if a secure aggregation protocol is employed at each round.
Revised: Similarly, the server can single out allindividual models x 𝑖 , even if a secure aggregation protocol is employed at each round.

Original: In this paper, we study secure aggregation protocols with long-term privacy guarantees (which weterm multi-round secure aggregation ) for the cross-device FL setup which has not been studied before.
Revised: In this paper, we study secure aggregation protocols with long-term privacy guarantees (which weterm multi-round secure aggregation ) for the cross-device FL setup which has not been studied before.

Original: We assume that user 𝑖 ∈ [ 𝑁 ] drops from the protocol at each round with probability 𝑝 𝑖 .
Revised: We assume that user 𝑖 ∈ [ 𝑁 ] drops from the protocol at each round with probability 𝑝 𝑖 .

Original: U ( 𝑡 ) denotesthe index set of available users at round 𝑡 and u ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is a vector indicating the available userssuch that { u ( 𝑡 ) } 𝑗 = 1 { 𝑗 ∈ U ( 𝑡 ) } , where { u } 𝑗 is 𝑗 -th entry of u and 1 {·} is the indicator function.
Revised: U ( 𝑡 ) denotesthe index set of available users at round 𝑡 and u ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is a vector indicating the available userssuch that { u ( 𝑡 ) } 𝑗 = 1 { 𝑗 ∈ U ( 𝑡 ) } , where { u } 𝑗 is 𝑗 -th entry of u and 1 {·} is the indicator function.

Original: The server selects 𝐾 users from U ( 𝑡 ) , if |U ( 𝑡 ) | ≥ 𝐾 , based on the history of selected users in previousrounds.
Revised: The server selects 𝐾 users from U ( 𝑡 ) , if |U ( 𝑡 ) | ≥ 𝐾 , based on the history of selected users in previous rounds.

Original: If |U ( 𝑡 ) | <
Revised: If |U ( 𝑡 ) | <

Original: 𝐾 , the server skips this round.
Revised: 𝐾 , the server skips this round.

Original: The local models of the selected users are thenaggregated via a secure aggregation protocol (i.e., by communicating masked models), at the end ofwhich the server learns the aggregate of the local models of the selected users.
Revised: The local models of the selected users are thenaggregated via a secure aggregation protocol (i.e., by communicating masked models), at the end ofwhich the server learns the aggregate of the local models of the selected users.

Original: Our goal is to design auser selection algorithm A ( 𝑡 ) : { 0 , 1 } 𝑡 [Equation] [Equation] [Equation]
Revised: Our goal is to design auser selection algorithm A ( 𝑡 ) : { 0 , 1 } 𝑡 × 𝑁 × { 0 , 1 } 𝑁 → { 0 , 1 } 𝑁 ,

Original: 𝑁 ,

Original: [Equation] u [Equation]
Revised: A ( 𝑡 ) (cid:0) P ( 𝑡 ) , u ( 𝑡 ) (cid:1) =

Original: = p ( 𝑡 ) such that (cid:107) p ( 𝑡 ) (cid:107) 

Revised: p ( 𝑡 ) such that (cid:107) p ( 𝑡 ) (cid:107)

Original: ∈ { 0 , 𝐾 } , (3) to prevent the potential information leakage over multiple rounds, where p ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is theparticipation vector deﬁned in (2), (cid:107) x (cid:107) 0 denotes the 𝐿 0 -“norm” of a vector x
Revised: ∈ { 0 , 𝐾 } , (3) to prevent the potential information leakage over multiple rounds, where p ( 𝑡 ) ∈ { 0 , 1 } 𝑁 is theparticipation vector deﬁned in (2), (cid:107) x (cid:107) 0 denotes the 𝐿 0 -“norm” of a vector x

Original: and 𝐾 denotes thenumber of selected users.
Revised: and 𝐾 denotes thenumber of selected users.

Original: We note that A ( 𝑡 ) can be a random function.
Revised: We note that A ( 𝑡 ) can be a random function.

Original: P ( 𝑡 ) is a matrix representingthe user participation information up to round 𝑡 , and is termed the participation matrix , given by 
Revised: P ( 𝑡 ) is a matrix representingthe user participation information up to round 𝑡 , and is termed the participation matrix , given by P (

Original: [Equation]
Revised: [Equation]

Revised: = (cid:2)

Original: [Equation]
Revised: [Equation]

Original: Key Metrics.
Revised: Key Metrics.

Original: A multi-round secure aggregation protocol can be represented by A = {A ( 𝑡 ) } 𝑡 ∈
Revised: A multi-round secure aggregation protocol can be represented by A = {A ( 𝑡 ) } 𝑡 ∈

Original: [ 𝐽 ] ,where A ( 𝑡 ) is the user selection algorithm at round 𝑡 deﬁned in (3) and 𝐽 is the total number of rounds.
Revised: [ 𝐽 ] ,where A ( 𝑡 ) is the user selection algorithm at round 𝑡 deﬁned in (3) and 𝐽 is the total number of rounds.

Original: The inputs of A ( 𝑡 ) are a random vector u ( 𝑡 ) , which indicates the available users at round 𝑡 , and theparticipation matrix P ( 𝑡 ) deﬁned in (4) which can be a random matrix.
Revised: The inputs of A ( 𝑡 ) are a random vector u ( 𝑡 ) , which indicates the available users at round 𝑡 , and theparticipation matrix P ( 𝑡 ) deﬁned in (4) which can be a random matrix.

Original: Given the participation matrix
Revised: Given the participation matrix

Original: P ( 𝐽 ) , we evaluate the performance of the corresponding multi-round secure aggregation protocolthrough the following metrics.
Revised: P ( 𝐽 ) , we evaluate the performance of the corresponding multi-round secure aggregation protocolthrough the following metrics.

Original: Multi-round Privacy Guarantee.
Revised: Multi-round Privacy Guarantee.

Original: The secure aggregation protocols ensure that the server can only learn the sum of the local models of some users in each single round, butthey do not consider
Revised: The secure aggregation protocols ensure that the server canonly learn the sum of the local models of some users in each single round , but

Original: what the server can learn over the long run.
Revised: they do not considerwhat the server can learn over the long run.

Original: Our multi-round privacy deﬁnition extends theguarantees of the secure aggregation protocols from one round to all  rounds by requiring that theserver can only learn a sum of the local models even if the server exploits the aggregate models of all rounds.
Revised: Our multi-round privacy deﬁnition extends the guarantees of the secure aggregation protocolsfrom one round to all training rounds by requiring that the server can only learn a sum of thelocal models even if the server exploits the aggregate models of all rounds.

Original: That is, our multi-round privacy guarantee is a natural extension of the privacy guarantee provided by the secure aggregation protocols considering a single training round.
Revised: That is, our multiround privacy guarantee is a natural extension of the privacy guarantee provided by the secureaggregation protocols considering a single training round.

Original: Speciﬁcally, a multi-round privacy guarantee 𝑇 requires that any non-zero partial sum of thelocal models that the server can reconstruct, through any linear combination X (cid:62) P ( 𝐽 )(cid:62) z , wherez ∈ R 𝐽 \ { 0 } , must be of the form {+
 [Equation]+}
Revised: Speciﬁcally, a multi-round privacy guarantee 𝑇 requires that any non-zero partial sum of thelocal models that the server can reconstruct, through any linear combination X (cid:62) P ( 𝐽 )(cid:62) z , wherez ∈ R 𝐽 \ { 0 } , must be of the form {+
 [Equation]+}

Original: [Equation]
Revised: [Equation]

Original: where |S 𝑖 | ≥ 𝑇, 𝑎 𝑖 ≠ 0 , ∀ 𝑖 ∈ [ 𝑛  and 𝑛 ∈ Z + .
Revised: where |S 𝑖 | ≥ 𝑇, 𝑎 𝑖 ≠ 0 , ∀ 𝑖 ∈ [ 𝑛 ] and 𝑛 ∈ Z + .

Original: Here all the sets S 𝑖 , the number of sets 𝑛 , and each 𝑎 𝑖 could all depend on z .
Revised: Here all the sets S 𝑖 , the number of sets 𝑛 , and each𝑎 𝑖 could all depend on z .

Original: In equation (5), we consider the worst-case scenario, where the local models do not change over the rounds.
Revised: In equation (5), we consider the worst-case scenario, where the localmodels do not change over the rounds.

Original: That is, X ( 𝑡 ) =
Revised: That is, X ( 𝑡 ) =

Original: X , ∀ 𝑡 ∈ [ 𝐽 ] .
Revised: X , ∀ 𝑡 ∈ [ 𝐽 ] .

Original: Intuitively, this guaranteeensures that the best that the server can do is to reconstruct a partial sum of 𝑇 local models whichcorresponds to the case where 𝑛 = 1 .
Revised: Intuitively, this guaranteeensures that the best that the server can do is to reconstruct a partial sum of 𝑇 local models whichcorresponds to the case where 𝑛 = 1 .

Original: When 𝑇 ≥ 2 , this condition implies that the server cannotget any user model from the aggregate models of all training rounds (the best it can obtain is thesum of two local models).
Revised: When 𝑇 ≥ 2 , this condition implies that the server cannotget any user model from the aggregate models of all training rounds (the best it can obtain is thesum of two local models).

Original: Remark 1.
Revised: Remark 1.

Original: (Weaker Privacy Notion).
Revised: (Weaker Privacy Notion).

Original: It is worth noting that, a weaker privacy notion wouldrequire that (cid:107) P ( 𝐽 )(cid:62) z (cid:107) 0 ≥ 𝑇 when P ( 𝐽 )(cid:62) z ≠ 0 .
Revised: It is worth noting that, a weaker privacy notion wouldrequire that (cid:107) P ( 𝐽 )(cid:62) z (cid:107) 0 ≥ 𝑇 when P ( 𝐽 )(cid:62) z ≠ 0 .

Original: When 𝑇 = 2 , this deﬁnition requires that the servercannot reconstruct any individual model (the best it can do is to obtain a linear combination oftwo local models).
Revised: When 𝑇 = 2 , this deﬁnition requires that the servercannot reconstruct any individual model (the best it can do is to obtain a linear combination oftwo local models).

Original: This notion, however, allows constructions in the form of 𝑎 x 𝑖 + 𝑏 x 𝑗 for any𝑎, 𝑏 ∈ R \ { 0 } .
Revised: This notion, however, allows constructions in the form of 𝑎 x 𝑖 + 𝑏 x 𝑗 for any𝑎, 𝑏 ∈ R \ { 0 } .

Original: When 𝑎 (cid:29) 𝑏 , however, this is almost the same as recovering x 𝑖 perfectly, hencethis privacy criterion is weaker than that of (5).
Revised: When 𝑎 (cid:29) 𝑏 , however, this is almost the same as recovering x 𝑖 perfectly, hencethis privacy criterion is weaker than that of (5).

Original: Remark 2.
Revised: Remark 2.

Original: (Multi-round Privacy of Random Selection).
Revised: (Multi-round Privacy of Random Selection).

Original: In Section 6, we empirically show that arandom selection strategy in which 𝐾 available users are selected uniformly at random at eachround does not ensure multi-round privacy even with respect to the weaker deﬁnition of Remark1.
Revised: In Section 6, we empirically show that arandom selection strategy in which 𝐾 available users are selected uniformly at random at eachround does not ensure multi-round privacy even with respect to the weaker deﬁnition of Remark1.

Original: Speciﬁcally, the local models can be reconstructed within a number of rounds that is linear in𝑁 .
Revised: Speciﬁcally, the local models can be reconstructed within a number of rounds that is linear in𝑁 .

Original: We also show theoretically in Appendix H that when min ( 𝑁 − 𝐾, 𝐾 ) ≥ 𝑐𝑁 , where 𝑐 > 0 is aconstant, then the probability that the server can reconstruct all local models after 𝑁 rounds is at least 1 − 2 𝑒 − 𝑐 (cid:48) 𝑁 for a constant 𝑐 (cid:48) that depends on 𝑐 .
Revised: We also show theoretically in Appendix H that when min ( 𝑁 − 𝐾, 𝐾 ) ≥ 𝑐𝑁 , where 𝑐 > 0 is aconstant, then the probability that the server can reconstruct all local models after 𝑁 rounds isat least 1 − 2 𝑒 − 𝑐 (cid:48) 𝑁 for a constant 𝑐 (cid:48) that depends on 𝑐 .

Original: Finally, we show that a random selectionscheme in which the users are selected in an i.i.d fashion according to Bern( 𝐾 𝑁 ( 1 − 𝑝 ) ) reveals alllocal models after 𝑁 rounds with probability that converges to 1 exponentially fast.
Revised: Finally, we show that a random selectionscheme in which the users are selected in an i.i.d fashion according to Bern( 𝐾 𝑁 ( 1 − 𝑝 ) ) reveals alllocal models after 𝑁 rounds with probability that converges to 1 exponentially fast

Original: Remark 3.
Revised: Remark 3.

Original: (Worst-Case Assumption).
Revised: (Worst-Case Assumption).

Original: In (5), we considered the worst-case assumption wherethe models do not change over time.
Revised: In (5), we considered the worst-case assumption wherethe models do not change over time.

Original: When the local models change over rounds, the multi-roundprivacy guarantee becomes even stronger as the number of unknowns increases.
Revised: When the local models change over rounds, the multi-roundprivacy guarantee becomes even stronger as the number of unknowns increases.

Original: In Fig.
Revised: In Fig.

Original: 1 and
Revised: 1 and

Original: Appendix H, we empirically show that the conventional secure aggregation schemes leak extensiveinformation of training data even in the realistic settings where the models change over the rounds.2.
Revised: Appendix H, we empirically show that the conventional secure aggregation schemes leak extensiveinformation of training data even in the realistic settings where the models change over the rounds.2.

Original: Aggregation Fairness Gap.
Revised: Aggregation Fairness Gap.

Original: The average aggregation fairness gap quantiﬁes the largest gapbetween any two users in terms of the expected relative number of rounds each user has participatedin training.
Revised: The average aggregation fairness gap quantiﬁes the largest gapbetween any two users in terms of the expected relative number of rounds each user has participatedin training.

Original: Formally, the average aggregation fairness gap is deﬁned as follows
Revised: Formally, the average aggregation fairness gap is deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: where { p ( 𝑡 ) } 𝑖 is 𝑖 -th entry of the vector p ( 𝑡 ) and the expectation is over the randomness of the userselection algorithm A and the user availability.
Revised: where { p ( 𝑡 ) } 𝑖 is 𝑖 -th entry of the vector p ( 𝑡 ) and the expectation is over the randomness of the userselection algorithm A and the user availability.

Original: The main intuition behind this deﬁnition is thatwhen 𝐹 = 0 , all users participate on average on the same number of rounds.
Revised: The main intuition behind this deﬁnition is thatwhen 𝐹 = 0 , all users participate on average on the same number of rounds.

Original: This is important totake the different users into consideration equally and our experiments show that the accuracy ofthe schemes with small 𝐹 are much higher than the schemes with high 𝐹 .
Revised: This is important totake the different users into consideration equally and our experiments show that the accuracy ofthe schemes with small 𝐹 are much higher than the schemes with high 𝐹 .3.

Original: Average Aggregation Cardinality.
Revised: Average Aggregation Cardinality.

Original: The aggregation cardinality quantiﬁes the expected numberof models to be aggregated per round.
Revised: The aggregation cardinality quantiﬁes the expected numberof models to be aggregated per round.

Original: Formally, it is deﬁned as
Revised: Formally, it is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over the randomness in A and the user availability.
Revised: where the expectation is over the randomness in A and the user availability.

Original: Intuitively, lessnumber of rounds are needed to converge as more users participate in the training.
Revised: Intuitively, lessnumber of rounds are needed to converge as more users participate in the training.

Original: In fact, as weshow in Section 5.2, 𝐶 directly controls the convergence rate.
Revised: In fact, as weshow in Section 5.2, 𝐶 directly controls the convergence rate.

Original: 3.3 Baseline Schemes
Revised: 3.3 Baseline Schemes

Original: In this subsection, we introduce three baseline schemes for multi-round secure aggregation.
Revised: In this subsection, we introduce three baseline schemes for multi-round secure aggregation.

Original: Random Selection.
Revised: Random Selection.

Original: In this scheme, at each round, the server selects 𝐾 users at random from the setof available users if this is possible.
Revised: In this scheme, at each round, the server selects 𝐾 users at random from the setof available users if this is possible.

Original: Random Weighted Selection.
Revised: Random Weighted Selection.

Original: This scheme is a modiﬁed version of random selection to reduce 𝐹when the dropout probabilities of the users are not equal.
Revised: This scheme is a modiﬁed version of random selection to reduce 𝐹when the dropout probabilities of the users are not equal.

Original: Speciﬁcally, 𝐾 users are selected at randomfrom the available users with the minimum frequency of participation in the previous rounds.
Revised: Speciﬁcally, 𝐾 users are selected at randomfrom the available users with the minimum frequency of participation in the previous rounds.

Revised: If lessthan 𝐾 users are available, the server skips this round.

Original: User Partitioning (Grouping).
Revised: User Partitioning (Grouping).

Original: In this scheme, the users are partitioned into 𝐺 =
Revised: In this scheme, the users are partitioned into 𝐺 =

Original: 𝑁 / 𝐾 equal-sizedgroups denoted as G 1 , G 2 , · · · , G 𝐺 .
Revised: 𝑁 / 𝐾 equal-sizedgroups denoted as G 1 , G 2 , · · · , G 𝐺 .

Original: At each round, the server selects one of the groups if none ofthe users in this group has dropped out.
Revised: At each round, the server selects one of the groups if none ofthe users in this group has dropped out.

Original: If multiple groups are available, to reduce the aggregationfairness gap, the server selects a group including a user with the minimum frequency of participationin previous rounds.
Revised: If multiple groups are available, to reduce the aggregationfairness gap, the server selects a group including a user with the minimum frequency of participationin previous rounds.

Original: If no group is available, the server skips this round.
Revised: If no group is available, the server skips this round.

Original:  Proposed Scheme: Multi-RoundSecAgg
Revised:  Proposed Scheme: Multi-RoundSecAgg

Original: In this section, we present Multi-RoundSecAgg, which has two components as follows
Revised: In this section, we present Multi-RoundSecAgg, which has two components as follows.

Original: • The ﬁrst component designs a family of sets of users that satisfy the multi-round privacy requirement.
Revised: • The ﬁrst component designs a family of sets of users that satisfy the multi-round privacy requirement.

Original: The inputs of the ﬁrst component are the number of users ( 𝑁 ), the number of selectedusers at eachround ( 𝐾 ), and the desired multi-round privacy guarantee ( 𝑇 ).
Revised: The inputs of the ﬁrst component are the number of users ( 𝑁 ), the number of selected at each round( 𝐾 ), and the desired multi-round privacy guarantee ( 𝑇 ).

Original: The output is a family of sets of 𝐾users satisfying the multi-round privacy guarantee 𝑇 , termed as a privacy-preserving family .
Revised: The output is a family of sets of 𝐾 users satisfying the multi-round privacy guarantee 𝑇 , termed as a privacy-preserving family .

Original: Thisfamily is represented by a matrix B , where the rows are the characteristic vectors of these user sets.
Revised: This family is represented by a matrix B , where the rows are the characteristic vectors of these user sets.

Original: • The second component selects a set from this designed family to satisfy the fairness guarantee.
Revised: • The second component selects a set from this designed family to satisfy the fairness guarantee.

Original: Theinputs to the second component are the  family B , the set of available users at round 𝑡 , U ( 𝑡 ) , and thefrequency of participation of eachuser.
Revised: Theinputs to the second component are the privacy-preserving family B , the set of available users atround 𝑡 , U ( 𝑡 ) , and the frequency of participation of each

Original:  The output is the set of users that will participate at round 𝑡 .
Revised: user. The output is the set of users thatwill participate at round 𝑡 .

Original: We now describe these two components in detail.
Revised: We now describe these two components in detail.

Original: Component 1 (Batch Partitioning (BP) of the users to guarantee multi-round privacy).
Revised: Component 1 (Batch Partitioning (BP) of the users to guarantee multi-round privacy).

Original: Theﬁrst component designs a family of 𝑅 BP sets, where 𝑅 BP is the size of the set, satisfying the multiround privacy requirement 𝑇 .
Revised: Theﬁrst component designs a family of 𝑅 BP sets, where 𝑅 BP is the size of the set, satisfying the multiround privacy requirement 𝑇 .

Original: We denote the 𝑅 BP × 𝑁 binary matrix corresponding to these sets by
Revised: We denote the 𝑅 BP × 𝑁 binary matrix corresponding to these sets by

Original: B = [ b 1 , · · · , b 𝑅 BP ] (cid:62) , where (cid:107) b 𝑖 (cid:107) 

Revised: B = [ b 1 , · · · , b 𝑅 BP ] (cid:62) , where (cid:107) b 𝑖 (cid:107)

Original: = 𝐾, ∀ 𝑖 ∈ [ 𝑅 BP ] .
Revised: = 𝐾, ∀ 𝑖 ∈ [ 𝑅 BP ] .

Original: That is, the rows of B are the characteristicvectors of those sets.
Revised: That is, the rows of B are the characteristicvectors of those sets.

Original: The main idea of our scheme is to restrict certain sets of users of size 𝑇 , denotedas batches, to either participate together or not participate at all.
Revised: The main idea of our scheme is to restrict certain sets of users of size 𝑇 , denotedas batches, to either participate together or not participate at all.

Original: This guarantees a multi-round privacy 241 𝑇 as we show in Section 5.
Revised: This guarantees a multi-round privacy 238 𝑇 as we show in Section 5.

Original: To construct a family of sets with this property, the users are ﬁrst partitioned into 𝑁 / 𝑇batches.
Revised: To construct a family of sets with this property, the users are ﬁrst partitioned into 𝑁 / 𝑇batches.

Original: At any given round, either all or none of the users of a particular batch participatein training.
Revised: At any given round, either all or none of the users of a particular batch participatein training.

Original: The server can choose 𝐾 / 𝑇 batches to participate in training, provided that allusers in any given selected batch are available.
Revised: The server can choose 𝐾 / 𝑇 batches to participate in training, provided that allusers in any given selected batch are available.

Original: Since there are (cid:0) 𝑁 / 𝑇 𝐾 / 𝑇 (cid:1) possible sets withthis property, then the size of this privacy-preserving family of sets is given by 𝑅 [Figure]
Revised: Since there are (cid:0) 𝑁 / 𝑇 𝐾 / 𝑇 (cid:1) possible sets withthis property, then the size of this privacy-preserving family of sets is given by 𝑅 BP def = (cid:0) 𝑁 / 𝑇 𝐾 / 𝑇 (cid:1) 2 .

Original: In the extreme case of 𝑇 = 1 , this strategy specializes to randomselection where the server can choose any  𝐾possible users.
Revised: In the extreme case of 𝑇 = 1 , this strategy specializes to randomselection where the server can choose any set from the (cid:0) 𝑁𝐾 (cid:1)possible sets of 𝐾 users.

Original: Inthe other extreme case of 𝑇 =
Revised: In the other extreme case of 𝑇 =

Original: 𝐾 , this strategy specializes tothe partitioning strategy where there are 𝑁 / 𝐾 possible sets.
Revised: 𝐾 ,this strategy specializes to the partitioning strategy where thereare 𝑁 / 𝐾 possible sets.

Original: Wenext provide an example to illustrate the construction of B .
Revised: We next provide an example to illustratethe construction of B .

Original: Example 1 ( 𝑁 = 8 , 𝐾 = 4 ,𝑇 = 2 ) .
Revised: Example 1 ( 𝑁 = 8 , 𝐾 = 4 ,𝑇 = 2 ) .

Original: In this example, the usersare partitioned into 4 batches as G 1 = { 1 , 2 } , G 2 = { 3 , 4 } , G 3 ={ 5 , 6 } and G 4 = { 7 , 8 } as given in Fig.
Revised: In this example, the usersare partitioned into 4 batches as G 1 = { 1 , 2 } , G 2 = { 3 , 4 } , G 3 ={ 5 , 6 } and G 4 = { 7 , 8 } as given in Fig.

Original: The server can choose any two batches out of these 4batches, hence we have 𝑅 BP = (cid:0) 42 (cid:1)
Revised: The server can choose any two batches out of these 4batches, hence we have 𝑅 BP = (cid:0) 42 (cid:1)

Original: = 6 possible sets.
Revised: = 6 possible sets.

Original: This ensures a multi-round privacy 𝑇 = 2 .
Revised: This ensures a multi-round privacy 𝑇 = 2 .

Original: Component 2 (Available batch selection to guarantee fairness) .
Revised: Component 2 (Available batch selection to guarantee fairness) .

Original: At round 𝑡 , user 𝑖 ∈ [ 𝑁 ] isavailable to participate in the protocol with a probability 1 − 𝑝 𝑖 ∈ ( 0 , 1 ] .
Revised: At round 𝑡 , user 𝑖 ∈ [ 𝑁 ] isavailable to participate in the protocol with a probability 1 − 𝑝 𝑖 ∈ ( 0 , 1 ] .

Original: The frequency of participationof user 𝑖 before round 𝑡 is denoted by 𝑓 ( 𝑡 ) 𝑖 def = (cid:205)
Revised: The frequency of participationof user 𝑖 before round 𝑡 is denoted by 𝑓 ( 𝑡 ) 𝑖 def = (cid:205)

Original: 𝑡 − 1 𝑗 = 0 1 (cid:8) { p ( 𝑗 ) } 𝑖 = 1 (cid:9) .
Revised: 𝑡 − 1 𝑗 = 0 1 (cid:8) { p ( 𝑗 ) } 𝑖 = 1 (cid:9) .

Original: Given the set of available users atround 𝑡 , U ( 𝑡 ) , and the frequencies of participation f ( 𝑡 − 1 ) =
Revised: Given the set of available users atround 𝑡 , U ( 𝑡 ) , and the frequencies of participation f ( 𝑡 − 1 ) =

Original: ( 𝑓 ( 𝑡 − 1 ), · · · , 𝑓 ( 𝑡 − 1 ) 𝑁 )  , the server selects𝐾 users.
Revised: ( 𝑓 ( 𝑡 − 1 ), · · · , 𝑓 ( 𝑡 − 1 ) 𝑁 ) (cid:62) , the server selects𝐾 users.

Original: To do so, the server ﬁrst ﬁnds the submatrix of B denoted by B ( 𝑡 ) corresponding to  U ( 𝑡 ) .
Revised: To do so, the server ﬁrst ﬁnds the submatrix of B denoted by B ( 𝑡 ) corresponding to theset of available users at round 𝑡 , U ( 𝑡 ) .

Original: Speciﬁcally, the 𝑖 -th row of B denoted by b (cid:62) 𝑖 is included in B ( 𝑡 ) provided that supp ( b i ) ⊆
Revised: Speciﬁcally, the 𝑖 -th row of B denoted by b (cid:62) 𝑖 is included in B ( 𝑡 ) provided that supp ( b i ) ⊆

Original: U ( 𝑡 ) .
Revised: U ( 𝑡 ) .

Original: If B ( 𝑡 ) is an empty matrix, then the server skips this round.
Revised: If B ( 𝑡 ) is an empty matrix, then the server skips this round.

Original: Otherwise, the server selects a row from B ( 𝑡 )uniformly at random if 𝑝 𝑖 = 𝑝, ∀ 𝑖 ∈
Revised: Otherwise, the server selects a row from B ( 𝑡 ) uniformly at random if 𝑝 𝑖 = 𝑝, ∀ 𝑖 ∈

Original: [ 𝑁 ] .
Revised: [ 𝑁 ] .

Original: If the users have different 𝑝 𝑖 , the server selects a row from B ( 𝑡 ) that includes the user with the minimum frequency of participation ℓ ( 𝑡 − 1 ) min def = arg min 𝑖 ∈U ( 𝑡 ) 𝑓 ( 𝑡 − 1 ) 𝑖 .
Revised: If the usershave different dropout probabilities, the server selects a row from B ( 𝑡 ) that includes the user with theminimum frequency of participation ℓ ( 𝑡 − 1 ) min def = arg min 𝑖 ∈U ( 𝑡 ) 𝑓 ( 𝑡 − 1 ) 𝑖 .

Original: If there are many such rows, then the server selects one of them uniformly at random.
Revised: If there are many such rows, thenthe server selects one of them uniformly at random.

Original: Remark 4.
Revised: Remark 4.

Original: (Necessity of the Second Component).
Revised: (Necessity of the Second Component).

Original: The second component is necessary to guaranteethat the aggregation fairness gap goes to zero as we show in Theorem 1 and Section 6.
Revised: The second component is necessary to guaranteethat the aggregation fairness gap goes to zero as we show in Theorem 1 and Section 6.

Original: Overall, the algorithm ﬁrst designs a privacy-preserving family of sets to ensure the multi-roundprivacy guarantee 𝑇 .
Revised: Overall, the algorithm ﬁrst designs a privacy-preserving family of sets of 𝐾 users such that any set canbe selected at any given round while ensuring the multi-round privacy guarantee 𝑇 .

Original: Then speciﬁc sets are selected from this family  to ensure fairness.
Revised: Then a speciﬁcset is selected from this family at each round to ensure fairness.

Original: We describethe two components of Multi-RoundSecAgg in detail in Algorithm 1 and Algorithm 2 in Appendix D.
Revised: We describe the two components of Multi-RoundSecAgg in detail in Algorithm 1 and Algorithm 2 in Appendix D.

Original:  Theoretical Results
Revised:  Theoretical Results

Original: In this section, we provide the theoretical guarantees of Multi-RoundSecAgg in Section 5.1 and theconvergence analysis of Multi-RoundSecAgg in Section 5.2.
Revised: In this section, we provide the theoretical guarantees of Multi-RoundSecAgg in Section 5.1 and theconvergence analysis of Multi-RoundSecAgg in Section 5.2.

Original: 5.1 Theoretical Guarantees of Multi-RoundSecAgg
Revised: 5.1 Theoretical Guarantees of Multi-RoundSecAgg

Original: In this subsection, we establish the theoretical guarantees of Multi-RoundSecAgg in terms of themulti-round privacy guarantee, the aggregation fairness gap and the average aggregation cardinality.
Revised: In this subsection, we establish the theoretical guarantees of Multi-RoundSecAgg in terms of themulti-round privacy guarantee, the aggregation fairness gap and the average aggregation cardinality.

Original: Theorem 1.
Revised: Theorem 1.

Original: Multi-RoundSecAgg with parameters 𝑁, 𝐾,𝑇 ensures a multi-round privacy guaranteeof 𝑇 , an aggregation fairness gap 𝐹 = 0 , and an average aggregation cardinality given by
Revised: Multi-RoundSecAgg with parameters 𝑁, 𝐾,𝑇 ensures a multi-round privacy guaranteeof 𝑇 , an aggregation fairness gap 𝐹 = 0 , and an average aggregation cardinality given by

Original: [Equation]
Revised: [Equation]

Original: where 𝑞 = 1 − ( 1 − 𝑝 ) 𝑇 , when all users have the dropout probability 𝑝 .
Revised: where 𝑞 = 1 − ( 1 − 𝑝 ) 𝑇 , when all users have the dropout probability 𝑝 .

Original: We provide the proof of Theorem 1 in Appendix A.
Revised: We provide the proof of Theorem 1 in Appendix A.

Original: Remark 5.
Revised: Remark 5.

Original: (Trade-off between “Multi-round Privacy Guarantee” and “Average Aggregation
Revised: (Trade-off between “Multi-round Privacy Guarantee” and “Average Aggregation

Original: Cardinality”).
Revised: Cardinality”).

Original: Theorem 1 indicates a trade-off between the multi-round privacy and the averageaggregation cardinality since as 𝑇 increases, 𝐶 decreases which slows down the convergence as weshow in Sec.
Revised: Theorem 1 indicates a trade-off between the multi-round privacy and the averageaggregation cardinality since as 𝑇 increases, 𝐶 decreases which slows down the convergence as weshow in Sec.

Original: We show this trade-off in Fig.
Revised: We show this trade-off in Fig.

Original: Remark 6.
Revised: Remark 6.

Original: (Necessity of Batch Partitioning (BP)).
Revised: (Necessity of Batch Partitioning (BP)).

Original: We show that any strategy that satisﬁes the privacyguarantee in Equation (5) must have a batch partitioning structure, and for given 𝑁, 𝐾,𝑇, 𝐾 ≤ 𝑁 / 2 ,the largest number of distinct user sets in any strategy is at most (cid:0) 𝑁 / 𝑇 𝐾 / 𝑇 (cid:1) , which is achieved in ourdesign in Section 4.
Revised: We show that any strategy that satisﬁes the privacyguarantee in Equation (5) must have a batch partitioning structure, and for given 𝑁, 𝐾,𝑇, 𝐾 ≤ 𝑁 / 2 ,the largest number of distinct user sets in any strategy is at most (cid:0) 𝑁 / 𝑇 𝐾 / 𝑇 (cid:1) , which is achieved in ourdesign in Section 4.

Original: We provide the proof in Appendix C.
Revised: We provide the proof in Appendix C.

Original: Remark 7.
Revised: Remark 7.

Original: (Non-linear Reconstructions of Aggregated Models).
Revised: (Non-linear Reconstructions of Aggregated Models).

Original: The privacy criterion in Eq. (5)considers linear reconstructions of the aggregated models.
Revised: The privacy criterion in Equation(5) considers linear reconstructions of the aggregated models.

Original: One may also consider more generalnon-linear reconstructions.
Revised: One may also consider more generalnon-linear reconstructions.

Original: The long-term privacy guarantees of batch partitioning hold even undersuch reconstructions as the users in the same batch always participate together or do not participateat all.
Revised: The long-term privacy guarantees of batch partitioning hold even undersuch reconstructions as the users in the same batch always participate together or do not participateat all.

Original: Hence, the server cannot separate individual models within the same batch even throughnon-linear operations.
Revised: Hence, the server cannot separate individual models within the same batch even throughnon-linear operations.

Original: 5.2 Convergence Analysis of Multi-RoundSecAgg
Revised: 5.2 Convergence Analysis of Multi-RoundSecAgg

Original: For convergence analysis of Multi-RoundSecAgg, weﬁrst introduce a few common assumptions [23, 39].
Revised: For convergence analysis of Multi-RoundSecAgg, weﬁrst introduce a few common assumptions [19, 33].

Original: Assumption 1.
Revised: Assumption 1.

Original: 𝐿 1 , . . .
Revised: 𝐿 1 , . . .

Original: , 𝐿 𝑁 in (1) are all 𝜌 -smooth:for all a , b ∈ R 𝑑 and 𝑖 ∈
Revised: , 𝐿 𝑁 in (1) are all 𝜌 -smooth:for all a , b ∈ R 𝑑 and 𝑖 ∈

Original: [ 𝑁 ] , 𝐿 𝑖 ( a ) ≤ 𝐿 𝑖 ( b ) + ( a −b ) (cid:62) ∇ 𝐿 𝑖 ( b ) + 𝜌 2 (cid:107) a − b (cid:107) 2 .
Revised: [ 𝑁 ] , 𝐿 𝑖 ( a ) ≤ 𝐿 𝑖 ( b ) + ( a −b ) (cid:62) ∇ 𝐿 𝑖 ( b ) + 𝜌 2 (cid:107) a [Equation] b (cid:107) [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: 𝐿 1 , . . .
Revised: 𝐿 1 , . . .

Original: , 𝐿 𝑁 in (1) are all 𝜇 -stronglyconvex: for all a , b ∈ R 𝑑 and 𝑖 ∈
Revised: , 𝐿 𝑁 in (1) are all 𝜇 -stronglyconvex: for all a , b ∈ R 𝑑 and 𝑖 ∈

Original: [ 𝑁 ] , 𝐿 𝑖 ( a ) ≥𝐿 𝑖 ( b ) + ( a − b ) (cid:62) ∇ 𝐿 𝑖 ( b ) + 𝜇 2 (cid:107) a − b (cid:107) 2 .
Revised: [ 𝑁 ] , 𝐿 𝑖 ( a ) ≥𝐿 𝑖 ( [Equation] + ( [Equation]

Original: Assumption 3.
Revised: Assumption 3.

Original: Let 𝜉 ( 𝑡 )
Revised: Let 𝜉 ( 𝑡 )

Original: 𝑖 be a sample uniformly selected from the dataset D 𝑖 .
Revised: 𝑖 be a sample uniformly selected from the dataset D 𝑖 .

Original: The variance of thestochastic gradients at each user is bounded, i.e., E (cid:107)∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 , 𝜉 ( 𝑡 ) 𝑖 ) − ∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 )(cid:107) 2 ≤ 𝜎 2 𝑖 for 𝑖 ∈ [ 𝑁 ] .
Revised: The variance of thestochastic gradients at each user is bounded, i.e., E (cid:107)∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 , 𝜉 ( 𝑡 ) 𝑖 ) − ∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 )(cid:107) 2 ≤ 𝜎 2 𝑖 for 𝑖 ∈ [ 𝑁 ] .

Original: Assumption 4.
Revised: Assumption 4.

Original: The expected squared norm of the stochastic gradients is uniformly bounded, i.e.,
Revised: The expected squared norm of the stochastic gradients is uniformly bounded, i.e.,

Original: E (cid:107)∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 , 𝜉 ( 𝑡 ) 𝑖 )(cid:107) 2 ≤ 𝐺 2 for all 𝑖 ∈ [ 𝑁 ] .
Revised: E (cid:107)∇ 𝐿 𝑖 ( x ( 𝑡 ) 𝑖 , 𝜉 ( 𝑡 ) 𝑖 )(cid:107) 2 ≤ 𝐺 2 for all 𝑖 ∈ [ 𝑁 ] .

Original: We now state the convergence guarantees of Multi-RoundSecAgg.
Revised: We now state the convergence guarantees of Multi-RoundSecAgg.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider a FL setup with 𝑁 users to train a machine learning model from (1) .
Revised: Consider a FL setup with 𝑁 users to train a machine learning model from (1) .

Original: Assume𝐾 users are selected by Multi-RoundSecAgg with average aggregation cardinality 𝐶 deﬁned in (7) toupdate the global model from (2) , and all users have the same dropout rate, hence Multi-RoundSecAggselects a random set of 𝐾 users uniformly from the set of available user sets at each round.
Revised: Assume𝐾 users are selected by Multi-RoundSecAgg with average aggregation cardinality 𝐶 deﬁned in (7) toupdate the global model from (2) , and all users have the same dropout rate, hence Multi-RoundSecAggselects a random set of 𝐾 users uniformly from the set of available user sets at each round.

Original: Then, thefollowing is satisﬁed
Revised: Then, thefollowing is satisﬁed

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where 𝛼 = 1 𝑁 (cid:205)
Revised: where 𝛼 = 1 𝑁 (cid:205)

Original: 𝑁𝑖 = 1 𝜎 2 𝑖+ 6 𝜌
Revised: 𝑁𝑖 = 1 𝜎 2 𝑖

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: ( 𝐸

Original: [Equation][Equation]
Revised: [Equation]

Original: , 𝛽 = 4 ( 𝑁 − 𝐾 ) 𝐸 2 𝐺 2 𝐾 ( 𝑁 − 1 ) , Γ = 𝐿 ∗ − (cid:205)
Revised: = 4 ( 𝑁 − 𝐾 ) 𝐸 2 𝐺 2 𝐾 ( 𝑁 − 1 ) , Γ = 𝐿 ∗ − (cid:205)

Original: 𝑁𝑖 = 1 𝐿 ∗ 𝑖 , and 𝛾 = max (cid:110) 8 𝜌𝜇 , 𝐸 (cid:111) .
Revised: 𝑁𝑖 = 1 𝐿 ∗ 𝑖 , and 𝛾 = max (cid:110) 8 𝜌𝜇 , 𝐸 (cid:111) .

Original: We provide the proof of Theorem 2 in Appendix B.
Revised: We provide the proof of Theorem 2 in Appendix B.

Original: Remark 8.
Revised: Remark 8.

Original: (The average aggregation cardinality controls the convergence rate.)
Revised: (The average aggregation cardinality controls the convergence rate.)

Original: Theorem 2 showshow the average aggregation cardinality affects the convergence.
Revised: Theorem 2 showshow the average aggregation cardinality affects the convergence.

Original: When the average aggregationcardinality is maximized, i.e., 𝐶 =
Revised: When the average aggregationcardinality is maximized, i.e., 𝐶 =

Original: 𝐾 , the convergence rate in Theorem 2 equals that of the randomselection algorithm provided in Theorem 3 of [23].
Revised: 𝐾 , the convergence rate in Theorem 2 equals that of the randomselection algorithm provided in Theorem 3 of [19].

Original: In (8), we have the additional term 𝐸 (number oflocal epochs) in front of 𝐽 compared to Theorem 3 of [23] as we use global round index 𝑡 instead ofusing step index of local SGD.
Revised: In (8), we have the additional term 𝐸 (number oflocal epochs) in front of 𝐽 compared to Theorem 3 of [19] as we use global round index 𝑡 instead ofusing step index of local SGD.

Original: As the average aggregation cardinality decreases, a greater number oftraining rounds is required to achieve the same level of accuracy.
Revised: As the average aggregation cardinality decreases, a greater number oftraining rounds is required to achieve the same level of accuracy.

Original: Remark 9.
Revised: Remark 9.

Original: (General Convex and Non-Convex Convergence Rates).
Revised: (General Convex and Non-Convex Convergence Rates).

Original: Theorem 2 considers thestrongly-convex case, but we consider the general convex and the non-convex cases in Appendix I.
Revised: Theorem 2 considers thestrongly-convex case, but we consider the general convex and the non-convex cases in Appendix I.

Original: Remark 10.
Revised: Remark 10.

Original: (Different Dropout Rates).
Revised: (Different Dropout Rates).

Original: When the dropout probabilities of the users are not the same,characterizing the convergence guarantees of Multi-RoundSecAgg is challenging.
Revised: When the dropout probabilities of the users are not the same,characterizing the convergence guarantees of Multi-RoundSecAgg is challenging.

Original: This is due to thefact that batch selection based on the frequency of participation breaks the conditional unbiasednessof the user selection, which is required for the convergence guarantee.
Revised: This is due to thefact that batch selection based on the frequency of participation breaks the conditional unbiasednessof the user selection, which is required for the convergence guarantee.

Original: In experiments, however, weempirically show that Multi-RoundSecAgg guarantees the convergence with different dropout rates.
Revised: In experiments, however, weempirically show that Multi-RoundSecAgg guarantees the convergence with different dropout rates.

Original:  Experiments
Revised:  Experiments

Original: Our experiments consist of two parts.
Revised: Our experiments consist of two parts.

Original: We ﬁrst numerically demonstrate the performance of MultiRoundSecAgg compared to the baselines of Section 3.3 in terms of the key metrics of Section 3.2.
Revised: We ﬁrst numerically demonstrate the performance of MultiRoundSecAgg compared to the baselines of Section 3.3 in terms of the key metrics of Section 3.2.

Original: Next, we implement convolutional neural networks (CNNs) for image classiﬁcation with MNIST [21],
Revised: Next, we implement convolutional neural networks (CNNs) for image classiﬁcation with MNIST [17],

Original: CIFAR-10, and CIFAR-100 [20] to investigate how the key metrics affect the test accuracy.
Revised: CIFAR-10, and CIFAR-100 [16] to investigate how the key metrics affect the test accuracy.

Original: Setup.
Revised: Setup.

Original: We consider a FL setting with 𝑁 = 120 users, where the server aims to choose 𝐾 = 12 usersat every round.
Revised: We consider a FL setting with 𝑁 = 120 users, where the server aims to choose 𝐾 = 12 usersat every round.

Original: We study two settings for partitioning the CIFAR-100 dataset across the users.
Revised: We study two settings for partitioning the CIFAR-100 dataset across the users.

Original: • IID Setting.
Revised: • IID Setting.

Original: 50000 training samples are shufﬂed and partitioned uniformly across 𝑁 = 120 users.• Non-IID Setting.
Revised: 50000 training samples are shufﬂed and partitioned uniformly across 𝑁 = 120 users.• Non-IID Setting.

Original: We distribute the dataset using a Dirichlet distribution [13], which samplesd 𝑐 ∼ Dir ( 𝛽 = 0 . 5 ) which specifying the prior class distribution over 100 classes, and allocate aportion 𝑑 𝑐,𝑖 of the class 𝑐 to user 𝑖 .
Revised: We distribute the dataset using a Dirichlet distribution [10], which samplesd 𝑐 ∼ Dir ( 𝛽 = 0 . 5 ) which specifying the prior class distribution over 100 classes, and allocate aportion 𝑑 𝑐,𝑖 of the class 𝑐 to user 𝑖 .

Original: The parameter 𝛽 controls the heterogeneity of the distributionsat each user, where 𝛽 → ∞ results in IID setting.
Revised: The parameter 𝛽 controls the heterogeneity of the distributionsat each user, where 𝛽 → ∞ results in IID setting.

Original: We implement a VGG-11 [29], which is sufﬁcient for our needs, as our goal is to evaluate variousschemes, not to achieve the best accuracy.
Revised: We implement a VGG-11 [25], which is sufﬁcient for our needs, as our goal is to evaluate variousschemes, not to achieve the best accuracy.

Original: The hyperparameters are provided in Appendix F.
Revised: The hyperparameters are provided in Appendix G.

Original: Modeling dropouts.
Revised: Modeling dropouts.

Original: To model heterogeneous system, users have different dropout probability 𝑝 𝑖selected from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
Revised: To model heterogeneous system, users have different dropout probability 𝑝 𝑖which is selected from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .

Original: At each round, user 𝑖 ∈ [ 𝑁 ] drops with probability 𝑝 𝑖 .
Revised: At each round, user 𝑖 ∈ [ 𝑁 ] drops from the protocolwith probability 𝑝 𝑖 .

Original: [Table]
Revised: [Table]

Original: Implemented Schemes.
Revised: Implemented Schemes.

Original: For the benchmarks, weimplement the three baselines introduced in Sec.
Revised: For the benchmarks, weimplement the three baselines introduced in Sec.

Original: 3.3,referred to as Random , Weighted Random , and Partition .
Revised: 3.3,referred to as Random , Weighted Random , and Partition .

Original: For Multi-RoundSecAgg, we construct three privacypreserving families with different target multi-roundprivacy guarantees, 𝑇 = 6 , 𝑇 = 4 , and 𝑇 = 3 whichwe refer to as Multi-RoundSecAgg
Revised: For Multi-RoundSecAgg, we construct three privacypreserving families with different target multi-roundprivacy guarantees, 𝑇 = 6 , 𝑇 = 4 , and 𝑇 = 3 whichwe refer to as Multi-RoundSecAgg

Original: ( 𝑇 = 6 ), MultiRoundSecAgg ( 𝑇 = 4 ), and Multi-RoundSecAgg ( 𝑇 =3 ), respectively.
Revised: ( 𝑇 = 6 ), MultiRoundSecAgg ( 𝑇 = 4 ), and Multi-RoundSecAgg ( 𝑇 =3 ), respectively.

Original: One can view the Random and Partition as extreme cases of Multi-RoundSecAggwith 𝑇 = 1 and 𝑇
Revised: One can view the Random and Partition as extreme cases of Multi-RoundSecAggwith 𝑇 = 1 and 𝑇

Original: =
Revised: =

Original: 𝐾 , respectively.
Revised: 𝐾 , respectively.

Original: Table 1 summarizes the family size 𝑅 deﬁned in Section 4.
Revised: Table 1 summarizes the family size 𝑅 deﬁned in Section 4.

Original: Key Metrics.
Revised: Key Metrics.

Original: To numerically demonstrate the performance of the six schemes in terms of the keymetrics deﬁned in Sec. 3.2, at each round, we measure the following metrics.
Revised: To numerically demonstrate the performance of the six schemes in terms of the keymetrics deﬁned in Sec. 3.2, at each round, we measure the following metrics.

Original: • For the multi-round privacy guarantee, we measure the number of models in the partial sum that
Revised: • For the multi-round privacy guarantee, we measure the number of models in the partial sum that

Original: the server can reconstruct, which is given by 𝑇 ( 𝑡 ) (cid:66)
Revised: the server can reconstruct, which is given by 𝑇 ( 𝑡 ) (cid:66)

Original: min z ∈ R 𝐽 } (cid:107) z
Revised: min z ∈ R 𝐽 } (cid:107) z

Original: (cid:62) P ( 𝑡 ) (cid:107) 0 , s.t. P ( 𝑡 )(cid:62) z ≠ 0 .
Revised: (cid:62) P ( 𝑡 ) (cid:107) 0 , s.t. P ( 𝑡 )(cid:62) z ≠ 0 .

Original: Thiscorresponds to the weaker privacy deﬁnition of Remark 1.
Revised: This corresponds to the weaker privacy deﬁnition of Remark 1.

Original: We use this weaker privacy deﬁnition
Revised: We use this weaker privacy deﬁnition

Original: as the random selection and the random weighted selection strategies provide the worst privacy
Revised: as the random selection and the random weighted selection strategies provide the worst privacy

Original: guarantee even with this weaker deﬁnition, as demonstrated later.
Revised: guarantee even with this weaker deﬁnition, as demonstrated later.

Original: On the other hand, Multi-
Revised: On the other hand, Multi-

Original: RoundSecAgg provides better privacy guarantees with both the strong and the weaker deﬁnitions.
Revised: RoundSecAgg provides better privacy guarantees with both the strong and the weaker deﬁnitions.

Original: • For the aggregation fairness gap, we measure the instantaneous fairness gap, 𝐹 ( 𝑡 ) (cid:66)
Revised: • For the aggregation fairness gap, we measure the instantaneous fairness gap, 𝐹 ( 𝑡 ) (cid:66)max 𝑖 ∈ [ 𝑁 ] 𝐹

Original: max 𝑖 ∈[ 𝑁 ] 𝐹 ( 𝑡 ) 𝑖 − min 𝑖 ∈
Revised: [Equation] 𝑡 [Equation] 𝑖 − min 𝑖 ∈

Original: [ 𝑁 ] 𝐹 ( 𝑡 ) 𝑖 where 𝐹 ( 𝑡 ) 𝑖 = 1 𝑡 + 1 (cid:205) 𝑡𝑙 = 0 1 (cid:8) { p ( 𝑙 ) } 𝑖 = 1 (cid:9) .
Revised: [ 𝑁 [Equation] ( 𝑡 ) 𝑖 where 𝐹 ( 𝑡 ) 𝑖 = 1 𝑡 + 1 (cid:205) 𝑡𝑙 = 0 1 (cid:8) { p ( 𝑙 ) } 𝑖 = 1 (cid:9) .

Original: • We measure the instantaneous aggregation cardinality as 𝐶 ( 𝑡 ) (cid:66) 1 𝑡 + 1 (cid:205) 𝑡𝑙 = 0 (cid:107) p ( 𝑙 ) (cid:107) 0 .
Revised: • We measure the instantaneous aggregation cardinality as 𝐶 ( 𝑡 ) (cid:66) 1 𝑡 + 1 (cid:205) 𝑡𝑙 = 0 (cid:107) p ( 𝑙 ) (cid:107) 0 .

Original: We demonstrate these key metrics in Figure 4.
Revised: We demonstrate these key metrics in Figure 4.

Original: We make the following key observations
Revised: We make the following key observations.

Original: • Multi-RoundSecAgg achieves better multi-round privacy guarantee than both the random selectionand random weighted selection strategies, while user partitioning achieves the best multi-roundprivacy guarantee,
Revised: • Multi-RoundSecAgg achieves better multi-round privacy guarantee than both the random selectionand random weighted selection strategies, while user partitioning achieves the best multi-roundprivacy guarantee,

Original: 𝑇 =
Revised: 𝑇 =

Original: 𝐾 = 12 .
Revised: 𝐾 = 12 .

Original: However, the partitioning strategy has the worst aggregationcardinality, which results in the lowest convergence rate as demonstrated later.
Revised: However, the partitioning strategy has the worst aggregationcardinality, which results in the lowest convergence rate as demonstrated later.

Original: • Figure 5 demonstrates the trade-off between the multi-round privacy guarantee 𝑇 and the averageaggregation cardinality 𝐶 .
Revised: • Figure 5 demonstrates the trade-off between the multi-round privacy guarantee 𝑇 and the averageaggregation cardinality 𝐶 .

Original: Interestingly, Multi-RoundSecAgg when
Revised: Interestingly, Multi-RoundSecAgg when

Original: 𝑇 = 3 or 𝑇 = 4 achieves bettermulti-round privacy guarantee than both the random selection and the weighted random selectionstrategies while achieving almost the same average aggregation cardinality.
Revised: 𝑇 = 3 or 𝑇 = 4 achieves bettermulti-round privacy guarantee than both the random selection and the weighted random selectionstrategies while achieving almost the same average aggregation cardinality.

Original: [Figure]
Revised: [Figure]

Original:  d
Revised: o [Figure] C a r d

Original: [Figure]
Revised: [Figure]

Revised: n a li t

Original: [Figure] 
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: d P r i v a c y
Revised: d P r i v a c y

Original: G
Revised: G

Original: [Figure]
Revised: [Figure]

Original: a r
Revised: a r

Original: [Figure]
Revised: [Figure]

Original: n t
Revised: n t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Remark 11.
Revised: Remark 11.

Original: (Multi-round Privacy of Random and Weighted Random).
Revised: (Multi-round Privacy of Random and Weighted Random).

Original: The multi-round privacyguarantees of Random and Weighted Random drop sharply as shown in Fig. 4(a) as the participatingmatrix P ( 𝑡 ) ∈ { 0 , 1 } 𝑡 × 𝑁 becomes full rank with high probability when 𝑡 ≥ 𝑁 , and hence the server canreconstruct the individual models by utilizing a pseudo inversion of the matrix P ( 𝑡 ) .
Revised: The multi-round privacyguarantees of Random and Weighted Random drop sharply as shown in Fig. 4(a) as the participatingmatrix P ( 𝑡 ) ∈ { 0 , 1 } 𝑡 × 𝑁 becomes full rank with high probability when 𝑡 ≥ 𝑁 , and hence the server can [Figure] reconstruct the individual models by utilizing a pseudo inversion of the matrix P ( 𝑡 ) .

Original: More precisely,
Revised: More precisely,

Original: Theorem 3 in Appendix H shows this thresholding phenomenon , where the probability that the servercan reconstruct individual models after certain number of rounds converges to 1 exponentially fast.
Revised: Theorem 3 in Appendix H shows this thresholding phenomenon , where the probability that the servercan reconstruct individual models after certain number of rounds converges to 1 exponentially fast.

Original: Key Metrics versus Test Accuracy.
Revised: Key Metrics versus Test Accuracy.

Original: To investigate how the key metrics affect the test accuracy, wemeasure the test accuracy of the six schemes in the two settings, the IID and the non-IID settings.
Revised: To investigate how the key metrics affect the test accuracy, wemeasure the test accuracy of the six schemes in the two settings, the IID and the non-IID settings.

Original: Our results are demonstrated in Figure 6.
Revised: Our results are demonstrated in Figure 6.

Original: We make the following key observations.
Revised: We make the following key observations.

Original: • In the IID setting, the Multi-RoundSecAgg schemes show test accuracies that are comparable to therandom selection and random weighted selection schemes while the Multi-RoundSecAgg schemesprovide higher levels of privacy.
Revised: • In the IID setting, the Multi-RoundSecAgg schemes show test accuracies that are comparable to therandom selection and random weighted selection schemes while the Multi-RoundSecAgg schemesprovide higher levels of privacy.

Original: Speciﬁcally, the Multi-RoundSecAgg schemes achieve 𝑇 = 3 , 4 , 6based on the privacy-preserving family design while the random selection and random weighted
Revised: Speciﬁcally, the Multi-RoundSecAgg schemes achieve 𝑇 = 3 , 4 , 6based on the privacy-preserving family design while the random selection and random weightedselection schemes have 𝑇

Original: selection schemes have 𝑇 = 1 , i.e., the server can learn an individual local model.
Revised: = 1 , i.e., the server can learn an individual local model.

Original: • In the non-IID setting, Multi-RoundSecAgg not only outperforms the random selection scheme butalso achieves a smaller aggregation fairness gap as demonstrated in Fig. 4(b).•
Revised: • In the non-IID setting, Multi-RoundSecAgg not only outperforms the random selection scheme butalso achieves a smaller aggregation fairness gap as demonstrated in Fig. 4(b).

Original:  In both IID and non-IID settings, the user partitioning scheme has the worst accuracy as its averageaggregation cardinality is much smaller than the other schemes as demonstrated in Fig. 4(c).
Revised: • In both IID and non-IID settings, the user partitioning scheme has the worst accuracy as its averageaggregation cardinality is much smaller than the other schemes as demonstrated in Fig. 4(c).

Original: We also implement additional experiments on MNIST and CIFAR-10 datasets in Appendix E andpresent ablation study for various settings of ( 𝑁, 𝐾,𝑇 ) in Appendix G
Revised: We also implement additional experiments on MNIST and CIFAR-10 datasets in Appendix F andpresent ablation study for various settings of ( 𝑁, 𝐾,𝑇 ) in Appendix E

Original:  Conclusion
Revised:  Conclusion

Original: Partial user participation may breach user privacy in federated learning, even if secure aggregation isemployed at every training round.
Revised: Partial user participation may breach user privacy in federated learning, even if secure aggregation isemployed at every training round.

Original: To address this challenge, we introduced the notion of long-termprivacy, which ensures that the privacy of individual models are protected over all training rounds.
Revised: To address this challenge, we introduced the notion of long-termprivacy, which ensures that the privacy of individual models are protected over all training rounds.

Original: Wedeveloped Multi-RoundSecAgg, a structured user selection strategy that guarantees long-term privacywhile taking into account the fairness in user selection and average number of participating users,and showed that Multi-RoundSecAgg provides a trade-off between long-term privacy and averagenumber of participating users (hence the convergence rate).
Revised: Wedeveloped Multi-RoundSecAgg, a structured user selection strategy that guarantees long-term privacywhile taking into account the fairness in user selection and average number of participating users,and showed that Multi-RoundSecAgg provides a trade-off between long-term privacy and average number of participating users (hence the convergence rate).

Original: Our experiments on the CIFAR-100,
Revised: Our experiments on the CIFAR-100,

Original: CIFAR-10 , and MNIST datasets on both the IID and non-IID settings 
Revised: CIFAR- and MNIST datasets on both the IID and non-IID settings show that Multi-RoundSecAgg achieves comparable accuracy to the random selection strategy (which does not ensure long-term privacy) while ensuring long-term privacy guarantees Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Revised: contributions and scope?

Revised: [Yes]

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] See Section 7.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] SeeSection 7.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] Seeassumptions 1 to 4 in Section 5.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] We provide the proofs

Original: {+of Theorem 1 and 
+}
Revised: {+of Theorem 1 and 
+}

Revised: in Appendix A and B, respectively.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main

Revised: experimental results (either in the supplemental material or as a URL)?

Revised: [Yes]

Revised: Due to

Revised: the "double blind review", we do not include a URL of our code.

Revised: Instead, we attach ourcode in the supplementary materials.

Revised: We will include a URL of our code in Section

Revised: when

Revised: the paper

Revised: is accepted

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Original: show that Multi-RoundSecAggachieves comparable accuracy to the random selection strategy (which does not ensure long-termprivacy), while ensuring long-term privacy guarantees.
Revised: were chosen)? [Yes] We describe two settings for partitioning the dataset across theusers in Section 6, and provide hyperparameters in Appendix G. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] See Appendix G

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Appendix G

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We cite the creatorsof data and models in Section 6.

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]