Original: D 2 P-F ED :
Revised: D 2 P-F ED :

Original: D IFFERENTIALLY
Revised: D IFFERENTIALLY

Original: P RIVATE
Revised: P RIVATE

Original: F EDERATED
Revised: F EDERATED

Original: L EARNING WITH E FFICIENT C OMMUNICATION
Revised: L EARNING WITH E FFICIENT C OMMUNICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we propose the d iscrete Gaussian based d ifferentially p rivate fed erated learning (D 2 P-F ED ), a uniﬁed scheme to achieve both differential privacy (DP) and communication efﬁciency in federated learning (FL).
Revised: In this paper, we propose the d iscrete Gaussian based d ifferentially p rivate fed erated learning (D 2 P-F ED ), a uniﬁed scheme to achieve both differential privacy (DP) and communication efﬁciency in federated learning (FL).

Original: In particular, compared with the only prior work taking care of both aspects, D 2 P-F ED provides stronger privacy guarantee, better composability and smaller communication cost.
Revised: In particular, compared with the only prior work taking care of both aspects, D 2 P-F ED provides stronger privacy guarantee, better composability and smaller communication cost.

Original: The key idea is to apply the discrete Gaussian noise to the private data transmission.
Revised: The key idea is to apply the discrete Gaussian noise to the private data transmission.

Original: We provide complete analysis of the privacy guarantee, communication cost and convergence rate of D 2 P-F ED .
Revised: We provide complete analysis of the privacy guarantee, communication cost and convergence rate of D 2 P-F ED .

Original: We evaluated D 2 P-F ED on INFIMNIST and
Revised: We evaluated D 2 P-F ED on INFIMNIST and

Original: CIFAR10.
Revised: CIFAR10.

Original: The results show that D 2 P-F ED outperforms the-state-of-the-art by 6.7% to 9.75% in terms of model accuracy while saving one third of the communication cost.
Revised: The results show that D 2 P-F ED outperforms the-state-of-the-art by 4.7% to 13.0% in terms of model accuracy while saving one third of the communication cost.

Revised: The results might be surprising at its ﬁrst glance but is reasonable because the quantization level k in D 2 P-F ED is independent of q .

Revised: As long as q is large enough, the probability that the noise exceeds q is small and thus has negligible impact on the model accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) is a popular machine learning paradigm that allows a central server to train models over decentralized data sources.
Revised: Federated learning (FL) is a popular machine learning paradigm that allows a central server to train models over decentralized data sources.

Original: In federated learning, each client performs training locally on their data source and only updates the model change to the server, which then updates the global model based on the aggregated local updates.
Revised: In federated learning, each client performs training locally on their data source and only updates the model change to the server, which then updates the global model based on the aggregated local updates.

Original: Since the data stays locally, FL can provide better privacy protection than traditional centralized learning.
Revised: Since the data stays locally, FL can provide better privacy protection than traditional centralized learning.

Original: However, FL is facing two main challenges: (1) FL lacks a rigorous privacy guarantee (e.g., differential privacy (DP)) and indeed, it has been shown to be vulnerable to various inference attacks (Nasr et al., 2019; Pustozerova & Mayer); (2) FL incurs considerable communication costs.
Revised: However, FL is facing two main challenges: (1) FL lacks a rigorous privacy guarantee (e.g., differential privacy (DP)) and indeed, it has been shown to be vulnerable to various inference attacks (Nasr et al., 2019; Pustozerova & Mayer; Xie et al., 2019); (2) FL incurs considerable communication costs.

Original: In many potential applications of FL such as mobile devices, these two challenges are present simultaneously .
Revised: In many potential applications of FL such as mobile devices, these two challenges are present simultaneously .

Original: However, privacy and communication-efﬁciency have mostly been studied independently in the past.
Revised: However, privacy and communication-efﬁciency have mostly been studied independently in the past.

Original: As regards privacy, existing work has applied a gold-standard privacy notion – differential privacy (DP) – to FL, which ensures that the server could hardly determine the participation of each client by observing their updates (Geyer et al., 2017).
Revised: As regards privacy, existing work has applied a gold-standard privacy notion – differential privacy (DP) – to FL, which ensures that the server could hardly determine the participation of each client by observing their updates (Geyer et al., 2017).

Original: To achieve DP, each client needs to inject noise to their local updates and as a side effect, the performance of the trained model would inevitably degrade.
Revised: To achieve DP, each client needs to inject noise to their local updates and as a side effect, the performance of the trained model would inevitably degrade.

Original: To improve model utility, secure multiparty computation (SMC) has been used in tandem with DP to reduce noise (Jayaraman et al., 2018; Truex et al., 2019).
Revised: To improve model utility, secure multiparty computation (SMC) has been used in tandem with DP to reduce noise (Jayaraman et al., 2018; Truex et al., 2019).

Original: The key idea is to prevent the server from observing the individual updates, make only the aggregate accessible, and thus transform from local DP to central DP.
Revised: The key idea is to prevent the server from observing the individual updates, make only the aggregate accessible, and thus transform from local DP to central DP.

Original: However, SMC introduces extra communication overhead to each client.
Revised: However, SMC introduces extra communication overhead to each client.

Original: There has been extensive research on improving communication efﬁciency of FL while ignoring the privacy aspect (Tsitsiklis & Luo, 1987;
Revised: There has been extensive research on improving communication efﬁciency of FL while ignoring the privacy aspect (Tsitsiklis & Luo, 1987;

Original: Balcan et al., 2012; Zhang et al., 2013; Arjevani &
Revised: Balcan et al., 2012; Zhang et al., 2013; Arjevani &

Original: Shamir, 2015; Chen et al., 2016).
Revised: Shamir, 2015; Chen et al., 2016).

Original: However, these communication reduction methods either have incompatible implementations with the existing DP mechanisms or would break the DP guarantees when combined with SMC.
Revised: However, these communication reduction methods either have incompatible implementations with the existing DP mechanisms or would break the DP guarantees when combined with SMC.

Original: The only existing work that tries to reconcile DP and communication efﬁciency in FL is cpSGD (Agarwal et al., 2018).
Revised: The only existing work that tries to reconcile DP and communication efﬁciency in FL is cpSGD (Agarwal et al., 2018).

Original: The authors leveraged the Binomial mechanism, which adds Binomial noise into local updates to ensure differential privacy.
Revised: The authors leveraged the Binomial mechanism, which adds Binomial noise into local updates to ensure differential privacy.

Original: The discrete nature of Binomial noise allows it to be transmitted efﬁciently.
Revised: The discrete nature of Binomial noise allows it to be transmitted efﬁciently.

Original: However, cpSGD faces several limitations when applied to real-world applications.
Revised: However, cpSGD faces several limitations when applied to real-world applications.

Original: Firstly, with Binomial noise, the output of a learning algorithm would have different supports when any client participates or withdraws from learning; as a result, Binomial noise can only guarantee approximate DP where the participation of the client can be completely exposed with nonzero probability.
Revised: Firstly, with Binomial noise, the output of a learning algorithm would have different supports on different input datasets; as a result, Binomial noise can only guarantee approx- imate DP where the participation of the client can be completely exposed with nonzero probability.

Original: Also, there lacks a tight composition for DP with Binomial noise and the resulting privacy budget skyrockets in a multi-round FL protocol.
Revised: Also, there lacks a tight composition for DP with Binomial noise and the resulting privacy budget skyrockets in a multi-round FL protocol.

Original: Hence, the Binomial mechanism cannot produce a useful model with a reasonable privacy budget on complex tasks.
Revised: Hence, the Binomial mechanism cannot produce a useful model with a reasonable privacy budget on complex tasks.

Original: Last but not least, the Binomial mechanism involves several mutually constrained hyper-parameters and the privacy formula is extremely complicated, which makes hyper-parameter tuning a difﬁcult task.
Revised: Last but not least, the Binomial mechanism involves several mutually constrained hyper-parameters and the privacy formula is extremely complicated, which makes hyper-parameter tuning a difﬁcult task.

Original: In this paper, we propose the d iscrete Gaussian based d ifferential p rivate fed erated learning (D 2 PFed), an alternative technique to reduce communication costs while maintaining differential privacy in FL.
Revised: In this paper, we propose the d iscrete Gaussian based d ifferential p rivate fed erated learning (D 2 PF ED ), an alternative technique to reduce communication costs while maintaining differential privacy in FL.

Original: Our key idea is to leverage the discrete Gaussian mechanism in FL, which adds discrete
Revised: Our key idea is to leverage the discrete Gaussian mechanism in FL, which adds discrete

Original: Gaussian noise into client updates.
Revised: Gaussian noise into client updates.

Original: We show that the discrete Gaussian mechanism satisﬁes R´enyi
Revised: We show that the discrete Gaussian mechanism satisﬁes R´enyi

Original: DP which provides better composability.
Revised: DP which provides better composability.

Original: We employ secure aggregation along with the discrete
Revised: We employ secure aggregation along with the discrete

Original: Gaussian mechanism to lower the noise and exhibit the privacy guarantee for this hybrid privacy protection approach.
Revised: Gaussian mechanism to lower the noise and exhibit the privacy guarantee for this hybrid privacy protection approach.

Original: To save the communication cost, we integrate the stochastic quantization and random rotation into the protocol.
Revised: To save the communication cost, we integrate the stochastic quantization and random rotation into the protocol.

Original: We then cast FL as a general distributed mean estimation problem and provide the analysis of the utility for the overall protocol.
Revised: We then cast FL as a general distributed mean estimation problem and provide the analysis of the utility for the overall protocol.

Original: Our theoretical analysis sheds light on the superiority of D 2 P-Fed to cpSGD.
Revised: Our theoretical analysis sheds light on the superiority of D 2 P-F ED to cpSGD.

Original: Our experiments show that D 2 P-Fed can lead to state-ofthe-art performance in terms of managing the trade-off among privacy, utility, and communication.
Revised: Our experiments show that D 2 P-F ED can lead to state-ofthe-art performance in terms of managing the trade-off among privacy, utility, and communication.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: It is well studied how to improve the communication cost in traditional distributed learning settings (Tsitsiklis & Luo (1987);
Revised: It is well studied how to improve the communication cost in traditional distributed learning settings (Tsitsiklis & Luo (1987);

Original: Balcan et al.
Revised: Balcan et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Arjevani & Shamir (2015);
Revised: ; Arjevani & Shamir (2015);

Original: Chen et al.
Revised: Chen et al.

Original: However, most of the approaches either require communication between the workers or are designed for speciﬁc learning tasks so they cannot be applied directly to generalpurpose FL.
Revised: However, most of the approaches either require communication between the workers or are designed for speciﬁc learning tasks so they cannot be applied directly to generalpurpose FL.

Original: The most relevant work is Suresh et al.
Revised: The most relevant work is Suresh et al.

Original: (2017) which proposed to use stochastic quantization to save the communication cost and random rotation to lower mean squared error of the estimated mean.
Revised: (2017) which proposed to use stochastic quantization to save the communication cost and random rotation to lower mean squared error of the estimated mean.

Original: We follow their approach to improve the communication efﬁciency and model utility of D 2 P-F ED .
Revised: We follow their approach to improve the communication efﬁciency and model utility of D 2 P-F ED .

Original: Nevertheless, our work differs from theirs in that we also study how to ensure DP for rotated and quantized data transmission and prove a convergence result for the learning algorithm with both communication cost reduction and privacy protection steps in place.
Revised: Nevertheless, our work differs from theirs in that we also study how to ensure DP for rotated and quantized data transmission and prove a convergence result for the learning algorithm with both communication cost reduction and privacy protection steps in place.

Original: On the other hand, differentially private FL is undergoing rapid development during the past few years (Geyer et al.
Revised: On the other hand, differentially private FL is undergoing rapid development during the past few years (Geyer et al.

Original: ; McMahan et al.
Revised: ; McMahan et al.

Original: ; Jayaraman et al.
Revised: ; Jayaraman et al.

Original: However, these methods mainly focus on improving utility under a small privacy budget and ignore the issue of communication cost.
Revised: However, these methods mainly focus on improving utility under a small privacy budget and ignore the issue of communication cost.

Original: In particular, we adopt a similar hybrid approach to Truex et al.
Revised: In particular, we adopt a similar hybrid approach to Truex et al.

Original: (2019), which combines SMC with DP for reducing the noise.
Revised: (2019), which combines SMC with DP for reducing the noise.

Original: SMC ensures that the centralized server can only see the aggregated update but not individual ones from clients and as a result, the noise added by each client can be reduced by a factor of thesquare root of the number of clients participating in one round.
Revised: SMC ensures that the centralized server can only see the aggregated update but not individual ones from clients and as a result, the noise added by each client can be reduced by a factor of the number of clients participating in one round.

Original: The difference of our work from theirs is that we inject discrete Gaussian noise to local updates instead of the continuous Gaussian noise.
Revised: The difference of our work from theirs is that we inject discrete Gaussian noise to local updates instead of the continuous Gaussian noise.

Original: This allows us to use secure aggregation (Bonawitz et al., 2017) which is much cheaper than threshold homomorphic encryption used by Truex et al.
Revised: This allows us to use secure aggregation (Bonawitz et al., 2017) which is much cheaper than threshold homomorphic encryption used by Truex et al.

Original: We further study the interaction between discrete Gaussian noise and the secure aggregation as well as their effects on the learning convergence.
Revised: We further study the interaction between discrete Gaussian noise and the secure aggregation as well as their effects on the learning convergence.

Original: We identify cpSGD (Agarwal et al.
Revised: We identify cpSGD (Agarwal et al.

Original: as the most comparable work to D 2 P-F ED .
Revised: as the most comparable work to D 2 P-F ED .

Original: Just like
Revised: Just like

Original: D 2 P-F ED , cpSGD aims to improve both the communication cost and the utility under rigorous privacy guarantee.
Revised: D 2 P-F ED , cpSGD aims to improve both the communication cost and the utility under rigorous privacy guarantee.

Original: However, cpSGD suffers from three main defects discussed in Section 1.
Revised: However, cpSGD suffers from three main defects discussed in Section 1.

Original: This paper proposes to use the discrete Gaussian mechanism to mitigate these issues in cpSGD.
Revised: This paper proposes to use the discrete Gaussian mechanism to mitigate these issues in cpSGD.

Original:  B ACKGROUND AND N OTATION
Revised:  B ACKGROUND AND N OTATION

Original: In this section, we provide an overview of FL and DP and establish the notation system.
Revised: In this section, we provide an overview of FL and DP and establish the notation system.

Original: We use bold lower-case letters ( e.g. a , b , c ) to denote vectors, and bold upper-case letters ( e.g. A , B , C ) for matrices.
Revised: We use bold lower-case letters ( e.g. a , b , c ) to denote vectors, and bold upper-case letters ( e.g. A , B , C ) for matrices.

Original: We denote 1 · · · n by [ n ] .
Revised: We denote 1 · · · n by [ n ] .

Original: FL Overview.
Revised: FL Overview.

Original: In a FL system, there are one server and n clients
Revised: In a FL system, there are one server and n clients

Original: C i , i ∈ [ n ] .
Revised: C i , i ∈ [ n ] .

Original: The server holds a global model of dimension m .
Revised: The server holds a global model of dimension d .

Original: Each client holds (IID or non-IID) samples drawn from some unknown distribution D .
Revised: Each client holds (IID or non-IID) samples drawn from some unknown distribution D .

Original: The goal is to learn the global model w ∈ R d that minimizes some loss function L ( w , D ) .
Revised: The goal is to learn the global model w ∈ R d that minimizes some loss function L ( w , D ) .

Original: To achieve this, the system runs a T -round FL protocol.
Revised: To achieve this, the system runs a T -round FL protocol.

Original: The server initializes the global model with w 0 .
Revised: The server initializes the global model with w 0 .

Original: In round t ∈ [ T ] , the server randomly sub-samples γn clients from [ n ] with sub-sampling rate γ and broadcasts the global model w t − 1 to the chosen clients.
Revised: In round t ∈ [ T ] , the server randomly sub-samples γn clients from [ n ] with sub-sampling rate γ and broadcasts the global model w t − 1 to the chosen clients.

Original: Each chosen client C i then runs the local optimizers ( e.g. SGD, Adam, and RMSprop), computes the difference between the locally optimized model w ( i ) t and the global model w t − 1 : g ( i ) t = w ( i ) t − w t − 1 , and uploads g ( i ) t to the server.
Revised: Each chosen client C i then runs the local optimizers ( e.g. SGD, Adam, and RMSprop), computes the difference between the locally optimized model w ( i ) t and the global model w t − 1 : g ( i ) t = w ( i ) t − w t − 1 , and uploads g ( i ) t to the server.

Original: The server takes the average of the differences and update the global model w t = w t − 1 + 1 k (cid:80) g
Revised: The server takes the average of the differences and update the global model w t = w t − 1 + 1 k (cid:80) g

Original: ( i ) t .
Revised: ( i ) t .

Original: Communication in FL.
Revised: Communication in FL.

Original: The clients in FL are often edge devices, where the upload bandwidth is fairly limited; therefore, communication efﬁciency is of uttermost importance to FL.
Revised: The clients in FL are often edge devices, where the upload bandwidth is fairly limited; therefore, communication efﬁciency is of uttermost importance to FL.

Original: Let π denote a communication protocol.
Revised: Let π denote a communication protocol.

Original: We denote the per-round communication cost as C ( π, g [ n ] ) .
Revised: We denote the per-round communication cost as C ( π, g [ n ] ) .

Original: To lower the communication cost, the difference vectors are typically compressed before sent to the server.
Revised: To lower the communication cost, the difference vectors are typically compressed before sent to the server.

Original: The compression would degrade model performance and we measure the performance loss via the mean squared error.
Revised: The compression would degrade model performance and we measure the performance loss via the mean squared error.

Original: Speciﬁcally, letting ¯ g denote the actual mean of difference vectors 1 n (cid:80) ni
Revised: Speciﬁcally, letting ¯ g denote the actual mean of difference vectors 1 n (cid:80) ni

Original: =1 g ( i ) and ˜ g denote the server’s estimated mean of difference vectors using some protocol such as D 2 PF ED , we could measure the performance loss by E ( π, δ [ n ] ) =
Revised: =1 g ( i ) and ˜ g denote the server’s estimated mean of difference vectors using some protocol such as D 2 PF ED , we could measure the performance loss by E ( π, g [ n ] ) =

Original: E [ (cid:107) ˜ g − ¯ g (cid:107) 2 ] , i.e., the mean squared error between the estimated and the actual mean.
Revised: E [ (cid:107) ˜ g − ¯ g (cid:107) 2 ] , i.e., the mean squared error between the estimated and the actual mean.

Original: This mean squared error is directly related to the convergence rate of FL (Agarwal et al., 2018).
Revised: This mean squared error is directly related to the convergence rate of FL (Agarwal et al., 2018).

Original: Threat Model & Differential Privacy.
Revised: Threat Model & Differential Privacy.

Original: We assume that the server is honest-but-curious.
Revised: We assume that the server is honest-but-curious.

Original: Namely, the server will follow the protocol honestly under the law enforcement or reputation pressure, but is curious to learn the client-side data from the legitimate client-side messages.
Revised: Namely, the server will follow the protocol honestly under the law enforcement or reputation pressure, but is curious to learn the client-side data from the legitimate client-side messages.

Original: In the FL context, the server wants to get information about the client-side data by studying the local updates received without deviating from the protocol.
Revised: In the FL context, the server wants to get information about the client-side data by studying the local updates received without deviating from the protocol.

Original: The above attack, widely known as the inference attack (Shokri et al., 2017; Yeom et al., 2018; Nasr et al., 2019), can be effectively mitigated using a canonical privacy notation namely differential privacy (DP).
Revised: The above attack, widely known as the inference attack (Shokri et al., 2017; Yeom et al., 2018; Nasr et al., 2019), can be effectively mitigated using a canonical privacy notation namely differential privacy (DP).

Original: Intuitively, DP, in the context of ML, ensures that the trained model is nearly the same regardless of the participation of any arbitrary client.
Revised: Intuitively, DP, in the context of ML, ensures that the trained model is nearly the same regardless of the participation of any arbitrary client.

Original: Deﬁnition 1 ( ( (cid:15), δ ) -DP) .
Revised: Deﬁnition 1 ( ( (cid:15), δ ) -DP) .

Original: A randomized algorithm f : D → R is ( (cid:15), δ ) -differentially private if for every pair of neighboring datasets D and D (cid:48) that differs only by one datapoint, and every possible (measurable) output set E the following inequality holds: P [ f ( D ) ⊆
Revised: A randomized algorithm f : D → R is ( (cid:15), δ ) -differentially private if for every pair of neighboring datasets D and D (cid:48) that differs only by one datapoint, and every possible (measurable) output set E the following inequality holds: P [ f ( D ) ⊆

Original: E ] ≤ e (cid:15) P [ f ( D (cid:48) ) ⊆
Revised: E ] ≤ e (cid:15) P [ f ( D (cid:48) ) ⊆

Original: E ] + δ .
Revised: E ] + δ .

Original: ( (cid:15), δ ) -DP has been used as a privacy notion in most of the existing works of privacy-preserving FL.
Revised: ( (cid:15), δ ) -DP has been used as a privacy notion in most of the existing works of privacy-preserving FL.

Original: However, in this paper, we consider a generalization of DP, R ´ enyi differential privacy (RDP), which is strictly stronger than ( (cid:15), δ ) -DP  and allows tighter analysis for compositing multiple mechanisms.
Revised: However, in this paper, we consider a generalization of DP, R ´ enyi differential privacy (RDP), which is strictly stronger than ( (cid:15), δ ) -DP for δ > 0 and allows tighter analysis for compositing multiple mechanisms.

Original: This second point is particularly appealing, as FL mostly comprises multiple rounds yet the existing works suffer from skyrocketing privacy budgets for multi-round learning.
Revised: This second point is particularly appealing, as FL mostly comprises multiple rounds yet the existing works suffer from skyrocketing privacy budgets for multi-round learning.

Original: Deﬁnition 2 ( ( α, (cid:15) ) -RDP) .
Revised: Deﬁnition 2 ( ( α, (cid:15) ) -RDP) .

Original: For two probability distributions P and Q with the same support, the
Revised: For two probability distributions P and Q with the same support, the

Original: R´enyi divergence of order α > 1 is deﬁned by D α ( P (cid:107) Q ) ∆ =[Equation] P
Revised: R´enyi divergence of order α > 1 is deﬁned by D α ( P (cid:107) Q ) ∆ =

Original: [Equation] Q [Equation] x 
Revised: 1 α − 1 log E x ∼ Q ( P ( x ) Q ( x ) )

Original: [Equation]
Revised: α .

Original: A randomized mechanism f : D → R is ( α, (cid:15) )
Revised: A randomized mechanism f : D → R is ( α, (cid:15) )

Original: -RDP, if for any neighboring datasets D, D (cid:48)
Revised: -RDP, if for any neighboring datasets D, D (cid:48)

Original: ∈ D it holds that
Revised: ∈ D it holds that

Original: D α ( f ( D ) (cid:107) f
Revised: D α ( f ( D ) (cid:107) f

Original: ( D (cid:48) )) ≤ (cid:15) .
Revised: ( D (cid:48) )) ≤ (cid:15) .

Original: The intuition behind RDP is the same as other variants of differential privacy: “Similar inputs should yield similar output distributions,” and the similarity is measured by the R ´ enyi divergence under
Revised: The intuition behind RDP is the same as other variants of differential privacy: “Similar inputs should yield similar output distributions,” and the similarity is measured by the R´enyi divergence under

Original: RDP.
Revised: RDP.

Original: RDP can also be converted to ( (cid:15), δ ) -DP
Revised: RDP can also be converted to ( (cid:15), δ ) -DP

Original: using the following transformation.
Revised: using the following transformation.

Original: Lemma 1 (RDP-DP conversion (Mironov (2017))) .
Revised: Lemma 1 (RDP-DP conversion (Mironov (2017))) .

Original: If M obeys ( α, (cid:15) ) -RDP, then M obeys ( (cid:15) + log(1 /δ ) /
Revised: If M obeys ( α, (cid:15) ) -RDP, then M obeys ( (cid:15) + log(1 /δ ) /

Original: ( α − 1) , δ ) -DP for all 0 < δ < 1 .
Revised: ( α − 1) , δ ) -DP for all 0 < δ < 1 .

Original: RDP enjoys an operationally convenient and quantitatively accurate way of tracking cumulative privacy loss when compositing multiple mechanisms (Lemma 2) or being combined with subsampling (Wang et al., 2018).
Revised: RDP enjoys an operationally convenient and quantitatively accurate way of tracking cumulative privacy loss when compositing multiple mechanisms (Lemma 2) or being combined with subsampling (Wang et al., 2018).

Original: As a result, RDP is particularly suitable for the context of ML.
Revised: As a result, RDP is particularly suitable for the context of ML.

Original: Lemma 2 (Adaptive composition of RDP (Mironov (2017))) .
Revised: Lemma 2 (Adaptive composition of RDP (Mironov (2017))) .

Original: If (randomized) mechanism M 1 obeys ( α, (cid:15) 1 ) -RDP, and M 2 obeys ( α, (cid:15) 2 )
Revised: If (randomized) mechanism M 1 obeys ( α, (cid:15) 1 ) -RDP, and M 2 obeys ( α, (cid:15) 2 )

Original: -RDP, then their composition obeys ( α, (cid:15) 

Revised: -RDP, then their composition obeys ( α, (cid:15)

Original: + (cid:15) 2 ) -RDP.
Revised: + (cid:15) 2 ) -RDP.

Original:  D ISCRETE G AUSSIAN M ECHANISM
Revised:  D ISCRETE G AUSSIAN M ECHANISM

Original: In this section, we present the discrete Gaussian mechanism and establish its privacy guarantee.
Revised: In this section, we present the discrete Gaussian mechanism and establish its privacy guarantee.

Original: We ﬁrst introduce discrete Gaussian distribution.
Revised: We ﬁrst introduce discrete Gaussian distribution.

Original: Deﬁnition 3 (Discrete Gaussian Distribution) .
Revised: Deﬁnition 3 (Discrete Gaussian Distribution) .

Original: Discrete Gaussian is a probability distribution on a discrete additive subgroup L  parameterized by σ .
Revised: Discrete Gaussian is a probability distribution on a discrete additive subgroup L (for instance, a multiple of Z ) parameterized by σ .

Original: For a discrete Gaussian distribution N L ( σ ) and x ∈ L , the probability mass on x is proportional to e − πx 2 / (2 πσ 2 ) .
Revised: For a discrete Gaussian distribution N L ( σ ) and x ∈ L , the probability mass on x is proportional to e − x 2 / (2 σ 2 ) .

Original: Discrete Gaussian mechanism works by adding noise drawn from discrete Gaussian distribution.
Revised: Discrete Gaussian mechanism works by adding noise drawn from discrete Gaussian distribution.

Original: Canonne et al.
Revised: Canonne et al.

Original: (2020) proved concentrated DP for the discrete Gaussian mechanism.
Revised: (2020) proved concentrated DP for the discrete Gaussian mechanism.

Original: However, there lacks tight privacy ampliﬁcation and composition theorem for concentrated DP.
Revised: However, there lacks tight privacy ampliﬁcation and composition theorem for concentrated DP.

Original: To address, we turn to RDP and provide the ﬁrst RDP analysis for the discrete Gaussian mechanism.
Revised: To address, we turn to RDP and provide the ﬁrst RDP analysis for the discrete Gaussian mechanism.

Original: The proof is delayed to Appendix A due to space limitation.
Revised: The proof is delayed to Appendix A due to space limitation.

Original: Theorem 1 (RDP for discrete Gaussian mechanism) .
Revised: Theorem 1 (RDP for discrete Gaussian mechanism) .

Original: If f has sensitivity 1  , then the discrete Gaussian mechanism: f ( · ) + N L ( σ ) satisﬁes ( α, α/ (2 σ 2 ))
Revised: If f has sensitivity 1 and range ( f ) ⊆ L , then the discrete Gaussian mechanism: f ( · ) + N L ( σ ) satisﬁes ( α, α/ (2 σ 2 ))

Original: -RDP.
Revised: -RDP.

Original: Under RDP, discrete Gaussian exhibits tight privacy ampliﬁcation  under sub-sampling (Wang et al., 2018).
Revised: Under RDP, discrete Gaussian exhibits tight privacy ampliﬁcation bound under sub-sampling (Wang et al., 2018).

Original: This suits FL well since a subset of clients is sub-sampled to upload the updates in each round.
Revised: This suits FL well since a subset of clients is sub-sampled to upload the updates in each round.

Original: Theorem 2 (Privacy ampliﬁcation for discrete Gaussian mechanism) .
Revised: Corollary 1 (Privacy ampliﬁcation for discrete Gaussian mechanism (Wang et al., 2018)) .

Original: If a discrete Gaussian mechanism is ( α, α 2 σ 2 ) -RDP, then augmented with subsampling (without replacement), the privacy guarantee is ampliﬁed to (1) ( α, O ( αγ 2 σ 2 )) in the high privacy regime; or (2) ( α, O ( αγ 2 e 1 σ 2 )) in the low privacy regime.
Revised: If a discrete Gaussian mechanism is ( α, α 2 σ 2 ) -RDP, then augmented with subsampling (without replacement), the privacy guarantee is ampliﬁed to (1) ( α, O ( αγ 2 σ 2 )) in the high privacy regime; or (2) ( α, O ( αγ 2 e 1 σ 2 )) in the low privacy regime.

Original: Besides, RDP enables discrete Gaussian mechanism to be composed tightly with analytical moments accountant (Wang et al., 2018), which saves a huge amount of privacy budget in a multi-round FL.
Revised: Besides, RDP enables discrete Gaussian mechanism to be composed tightly with analytical moments accountant (Wang et al., 2018), which saves a huge amount of privacy budget in a multi-round FL.

Revised: Analytical moments accountant is a data structure that tracks the cumulant generating function of the composed mechanisms symbolically.

Revised: Since it has no closed-form solution, we instead introduce the canonical composition of RDP (Mironov, 2017) below for the ease of discussion in Section 6.3.

Original: Theorem 3 (Composition for discrete Gaussian mechanism) .
Revised: Corollary 2 (Composition for discrete Gaussian mechanism (Wang et al., 2018)) .

Original: If a discrete Gaussian mechanism is ( α, α 2 σ 2 ) -RDP, then the sequential composition of T such mechanisms yield ( α, Tα 2 σ 2 )
Revised: If a discrete Gaussian mechanism is ( α, α 2 σ 2 ) -RDP, then the sequential composition of T such mechanisms yield ( α, Tα 2 σ 2 )

Original: -RDP guarantee.
Revised: -RDP guarantee.

Original: If we convert all the RDP guarantees back to ( (cid:15), δ ) -DP, the growth of (cid:15) under the same δ is asymptotically O ( √ T ) .
Revised: If we convert all the RDP guarantees back to ( (cid:15), δ ) -DP, the growth of (cid:15) under the same δ is asymptotically O ( √ T ) .

Original: Note that both privacy ampliﬁcation and composition are given in an asymptotic form for the clarity of presentation.
Revised: Note that both privacy ampliﬁcation and composition are given in an asymptotic form for the clarity of presentation.

Original: For tight bound, we refer the readers to Theorem 27 and Section 3.3 in Wang et al.
Revised: For tight bound, we refer the readers to Theorem 27 and Section 3.3 in Wang et al.

Original:  D 2 P-F ED : A LGORITHM AND P RIVACY A NALYSIS
Revised:  D 2 P-F ED : A LGORITHM AND P RIVACY A NALYSIS

Original: In this section, we formally present D 2 P-F ED and provide rigorous privacy analysis.
Revised: In this section, we formally present D 2 P-F ED and provide rigorous privacy analysis.

Original: 5. A LGORITHM
Revised: 5. A LGORITHM

Original: Algorithm 1 provides the pseudocode for D 2 P-F ED .
Revised: Algorithm 1 provides the pseudocode for D 2 P-F ED .

Original: It follows the general FL pipeline which iteratively performs the following steps: (1) the server broadcasts the global model to a subset of clients; (2) the selected clients train the global model on their local data and upload the resulting model difference; and (3) the server aggregates the model differences uploaded by the clients and updates the global model.
Revised: It follows the general FL pipeline which iteratively performs the following steps: (1) the server broadcasts the global model to a subset of clients; (2) the selected clients train the global model on their local data and upload the resulting model difference; and (3) the server aggregates the model differences uploaded by the clients and updates the global model.

Original: Grounded on the general FL pipeline, D 2 P-F ED introduces some additional steps at the client side as follows to improve communication efﬁciency and privacy.
Revised: Grounded on the general FL pipeline, D 2 P-F ED introduces some additional steps at the client side as follows to improve communication efﬁciency and privacy.

Original: Stochastic Quantization & Random Rotation (line 11-14).
Revised: Stochastic Quantization & Random Rotation (line 11-14).

Original: To lower the communication cost, the clients stochastically quantize the values in the update vectors to some discrete domain.
Revised: To lower the communication cost, the clients stochastically quantize the values in the update vectors to some discrete domain.

Original: Compared with real number encoding which costs 32 or 64 bits per dimension, the quantized value only requires log 2 ( k ) bits per dimension.
Revised: Compared with real number encoding which costs 32 or 64 bits per dimension, the quantized value only requires log 2 ( k ) bits per dimension where k is the level of quantization (see line 12-14 in Algorithm 1 and McMahan et al. (2016) for detailed explanation.).

Original: On the other hand, quantization lowers the ﬁdelity of the update vector and thus leads to some error in estimating the mean of gradients.
Revised: On the other hand, quantization lowers the ﬁdelity of the update vector and thus leads to some error in estimating the mean of gradients.

Original: To lower the estimation error, the clients apply randomized rotation to the updates before quantization as proposed by McMahan et al.
Revised: To lower the estimation error, the clients apply randomized rotation to the updates before quantization as proposed by McMahan et al.

Original: The details are discussed in Section 6.1.
Revised: The details are discussed in Section 6.1.

Original: Discrete Gaussian Mechanism (line 15).
Revised: Discrete Gaussian Mechanism (line 15).

Original: We apply the discrete Gaussian mechanism to ensure
Revised: We apply the discrete Gaussian mechanism to ensure

Original: DP.
Revised: DP.

Original: To determine the noise magnitude to be added, we need to bound the (cid:96) 2 -sensitivity  of the gradient aggregate.
Revised: To determine the noise magnitude to be added, we need to bound the (cid:96) 2 -sensitivity (deﬁned in Section 5.2) of the gradient aggregate.

Original: Withoutthe quantization and random rotation, one could clip the individual gradient update and consequently the (cid:96) 2 -sensitivity is just the clipping threshold.
Revised: Without quantization and random rotation, one could clip the individual gradient update and consequently the (cid:96) 2 -sensitivity is just the clipping threshold.

Original: However, the inclusion of compression steps makes the analysis of (cid:96) 2 -sensitivity more sophisticated.
Revised: However, the inclusion of compression steps makes the analysis of (cid:96) 2 -sensitivity more sophisticated.

Original: We provide the analysis of sensitivity and RDP guarantees for the entire algorithm in Section 5.2.
Revised: We provide the analysis of sensitivity and RDP guarantees for the entire algorithm in Section 5.2.

Original: Each client samples the noise from the discrete Gaussian distribution.
Revised: Each client samples the noise from the discrete Gaussian distribution.

Original: In contrast to prior work where each client adds independent noise, we require the clients to share the same random seed, generate the same noise and add an average share of the noise in each round.
Revised: In contrast to prior work where each client adds independent noise, we require the clients to share the same random seed, generate the same noise and add an average share of the noise in each round.

Original: This is because the sum of multiple independent discrete Gaussians is no longer a discrete Gaussian.
Revised: This is because the sum of multiple independent discrete Gaussians is no longer a discrete Gaussian.

Original: Note that the same random seed is also required for communication reduction in secure aggregation so we can conveniently reuse it here without introducing more further overhead.
Revised: Note that the same random seed is also required for communication reduction in secure aggregation so we can conveniently reuse it here without introducing more further overhead (see Bonawitz et al.

Revised: (2017) for details).

Original: The noise magnitude is set to ensure that the aggregate noise from all clients provides the global DP.
Revised: The noise magnitude is set to ensure that the aggregate noise from all clients provides the global DP.

Original: Algorithm
Revised: Algorithm

Original: Input: Support Lattice: L = 2 g max k − 1 · Z , Noise Scale: σ , Rotation Matrix: R , Random Seed: sφ q ( x ) =
Revised: Input: Support Lattice: L = 2 g max k − 1 · Z , Noise Scale: σ , Rotation Matrix: R , Random Seed: s

Original:  2 g max k − 1 (( k − 1 2 g max x + q − 12 ) mod q − q − 12 ) , q is odd.
Revised: Quantization Level: k , φ q ( x ) = 2 g max k − 1 (( k − 1 2 g max x + q − 12 ) mod q − q − 12 ) , q is odd.

Original: 1 for t ← [ T ] do
Revised: 1 for t ← [ T ] do

Original: Server:
Revised: Server:

Original: Sub-sample a subset of clients S ⊂ [ n ] , | S | = γn and broadcast w t − 1 and g max to S
Revised: Sub-sample a subset of clients S ⊂ [ n ] , | S | = γn and broadcast w t − 1 and g max to S

Original: Client: foreach client i ∈ S do Send u ij to j , u ij ∼ Unif ( L mq ) , L q := { x ∈ L , | x | ≤ q − 12 }
Revised: Client: foreach client i ∈ S do Send u ij to j , u ij ∼ Unif ( L dq ) , L q := { x ∈ L , | x | ≤ q − 12 }

Original: Client: foreach client i ∈ S do
Revised: Client: foreach client i ∈ S do

Original: Train the model w ( i ) t with w t as initialization g ( i ) t = w ( i ) t − w t
Revised: Train the model w ( i ) t with w t as initialization g ( i ) t = w ( i ) t − w t

Original: −/* Compute the difference */ g( i ) t,clipped =
Revised: −/* Compute the difference */ g

Original:  g( i ) t / max(1 , (cid:107) g
Revised: = g t / max( i

Original: ( i ) t (cid:107) 2 D ) /
Revised: (cid:107) 2 D ) /

Original: * Clip the difference */g
Revised: * Clip the difference */

Original: ( i ) t,rotated =

Original: R × g ( i ) t,clipped / * Random Rotation */ let b [ r ] := − g max + 2 r g max k − 1 for every r ∈ [0 , k ) /
Revised: /* Random Rotation */ let b [ r ] := − g max + 2 r g max k − 1 for every r ∈ [0 , k ) /

Original: * Quantize */ for j ∈ d, b [ r ] ≤ ˜ g
Revised: * Quantize */ for j ∈ d, b [ r ] ≤ ˜ g

Original: ( i ) t,quantized [ j ] ≤ b [ r + 1] do ˜ g
Revised: ( i ) t,quantized [ j ] ≤ b [ r + 1] do ˜ g

Original: ( i ) t,quantized [ j ] =
Revised: ( i ) t,quantized [ j ] =

Original:   b [ r ] w.p. g ( i ) t,rotated [ j ] − b
Revised:   b [ r + 1] w.p. g ( i ) t,rotated [ j ] − b

Original: [ r ] b [ r + 1] − b
Revised: [ r ] b [ r + 1] − b

Original: [ r ] b [ r + 1] o.w.
Revised: [ r ] b [ r ] o.w.

Original: [Table]
Revised: ˜ g ( i ) t,dp = ˜ g ( i ) t,quantized + ν i γn , ν i s ∼ N d L ( σ ) /

Original: /* Discrete Gaussian */ 
Revised: * Discrete Gaussian */ ˜ g

Original: [Table]
Revised: ( i ) t = φ q ( φ q ( ˜ g ( i ) t,dp ) + (cid:80) j (cid:54) = i,j ∈ S u ij − (cid:80) j (cid:54) = i,j ∈ S u ji ) /

Original: /* Mask */
Revised: * Mask * /

Original: [Table] ˜ 
Revised: Send ˜ g

Original: [Table] [Table] to the [Table]
Revised: ( i ) t to the server

Original: [Table] ˜ 
Revised: Server: ˜ g

Original: [Equation]
Revised: = γn (cid:80)

Original:  /* Aggregate */ [Table]
Revised: S ˜ g ( i t /* Aggregate */ w t =

Original: [Equation]
Revised: w t − 1 + ˜ g t

Original: Secure aggregation.
Revised: Secure aggregation.

Original: (line 5,16,19)
Revised: (line 5,16,19)

Original: To reduce the noise magnitude for ensuring DP, we hide clients’ individual updates from the central server and only allow it to see the aggregated updates via the technique of secure aggregation.
Revised: To reduce the noise magnitude for ensuring DP, we hide clients’ individual updates from the central server and only allow it to see the aggregated updates via the technique of secure aggregation.

Original: If the individual updates were available to the central server, they should also be protected with the same privacy guarantee as the averaged update; in this case, the required noise scales up with O ( γn ) .
Revised: If the individual updates were available to the central server, they should also be protected with the same privacy guarantee as the averaged update; in this case, the required noise scales up with O ( γn ) .

Original: On the other hand, if the central server can only access the aggregated updates, then the required noise is O (1) .
Revised: On the other hand, if the central server can only access the aggregated updates, then the required noise is O (1) .

Original: Hence, secure aggregation of local updates can lead to signiﬁcant noise reduction.
Revised: Hence, secure aggregation of local updates can lead to signiﬁcant noise reduction.

Original: However, there exists a challenge for integrating secure aggregation with discrete Gaussian mechanism: discrete Gaussian variables have inﬁnite support, thereby incompatible with secure aggregation which operates on ﬁnite ﬁeld.
Revised: However, there exists a challenge for integrating secure aggregation with discrete Gaussian mechanism: discrete Gaussian variables have inﬁnite support, thereby incompatible with secure aggregation which operates on ﬁnite ﬁeld.

Original: In Section 5.2, we propose to address this challenge by mapping the noised vector to a cyclic additive group and then applying secure aggregation and show that the RDP guarantees are preserved under the mapping.
Revised: In Section 5.2, we propose to address this challenge by mapping the noised vector to a cyclic additive group and then applying secure aggregation and show that the RDP guarantees are preserved under the mapping.

Original: 5. P RIVACY A NALYSIS
Revised: 5. P RIVACY A NALYSIS

Original: Before applying discrete Gaussian mechanism to D 2 P-F ED , we need to ﬁgure out how to calibrate the added noise.
Revised: Before applying discrete Gaussian mechanism to D 2 P-F ED , we need to ﬁgure out how to calibrate the added noise.

Original: In differential privacy, the calibration is guided by sensitivity of the function as deﬁned below.
Revised: In differential privacy, the calibration is guided by sensitivity of the function as deﬁned below.

Original: Deﬁnition 4 ( (cid:96) 2 -sensitivity) .
Revised: Deﬁnition 4 ( (cid:96) 2 -sensitivity) .

Original: Given a function f : D → R and two neighboring datasets D and D (cid:48) , the (cid:96) 2 -sensitivity of f : ∆ f is deﬁned as ∆ f = max D,D (cid:48) (cid:107) f
Revised: Given a function f : D → R and two neighboring datasets D and D (cid:48) , the (cid:96) 2 -sensitivity of f : ∆ f is deﬁned as ∆ f = max D,D (cid:48) (cid:107) f

Original: ( D ) − f
Revised: ( D ) − f

Original: ( D (cid:48) )
Revised: ( D (cid:48) )

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: In DP for deep learning, the traditional way to bound (cid:96) 2 -sensitivity is to clip the update vector.
Revised: In DP for deep learning, the traditional way to bound (cid:96) 2 -sensitivity is to clip the update vector.

Original: However, quantization will further inﬂuence the sensitivity after clipping.
Revised: However, quantization will further inﬂuence the sensitivity after clipping.

Original: We provide the sensitivity after quantization as below.
Revised: We provide the sensitivity after quantization as below.

Original: The proof is delayed to Appendix B due to space limitation.
Revised: The proof is delayed to Appendix B due to space limitation.

Original: Theorem 4.
Revised: Theorem 2.

Original: If we clip the l 2 norm of g to D , and quantize it to k = √ d + 1 levels, then the (cid:96)sensitivity of the difference is 4 D .
Revised: If we clip the l 2 norm of g to D , and quantize it to k = √ d + 1 levels, then the (cid:96)sensitivity of the difference is 4 D .

Original: Given Theorem 1 and Theorem 4, we provide the RDP bound for D 2 P-F ED .
Revised: Given Theorem 1 and Theorem 2, we provide the RDP bound for D 2 P-F ED .

Original: Theorem 5 (RDP for D 2 P-F ED ) .
Revised: Corollary 3 (RDP for D 2 P-F ED ) .

Original: Given the clipping bound D , the noise scale σ , D 2 P-F ED follows ( α, 8 αD 2 σ 2 ) -RDP.
Revised: Given the clipping bound D , the noise scale σ , D 2 P-F ED follows ( α, 8 αD 2 σ 2 ) -RDP.

Revised: Remark 1: Comparison with cpSGD

Revised: It seems unclear how to interpret the above bound when compared with Theorem 1 in cpSGD (Agarwal et al., 2018).

Revised: Indeed, the claim that D 2 P-F ED has a better privacy guarantee than cpSGD can be mainly justiﬁed by the following three aspects: (1)

Revised: D 2 P-F ED follows RDP which is a strictly stronger privacy notion than cpSGD which is intrisically limited to ( (cid:15), δ ) -DP; (2) D 2 P-F ED enjoys a tighter composition compared to cpSGD.

Revised: This is of critical signiﬁcance in a FL protocol with potentially thousands of rounds; (3) Our experimental results in Figure 1a also empirically show that D 2 P-F ED enjoys a tighter composition than cpSGD.

Original: Theorem 5 is built on the assumption that the centralized server only has access to the summed updates but not the individual ones.
Revised: The total privacy budget for D 2 P-F ED grows much more slowly than cpSGD as training proceeds. Remark 2: Privacy Effect of Secure Aggregation. Corollary 3 is built on the assumption that the centralized server only has access to the summed updates but not the individual ones.

Original: If the centralized server has access to individual updates, the noise has to scale up γn times to maintain the same privacy guarantee which severely hinders the model accuracy.
Revised: If the centralized server has access to individual updates, the noise has to scale up γn times to maintain the same privacy guarantee which severely hinders the model accuracy.

Original: To consolidate the assumption, we leverage a cryptographic technique, secure aggregation (Bonawitz et al., 2017) which guarantees that the centralized server can only see the aggregated result.
Revised: To consolidate the assumption, we leverage a cryptographic technique, secure aggregation (Bonawitz et al., 2017) which guarantees that the centralized server can only see the aggregated result.

Original: The basic intuition is to mask the inputs with random values canceling out in pairs.
Revised: The basic intuition is to mask the inputs with random values canceling out in pairs.

Original: However, since discrete Gaussian has inﬁnite support, we cannot directly apply random masks to it.
Revised: However, since discrete Gaussian has inﬁnite support, we cannot directly apply random masks to it.

Original: To reconcile secure aggregation with discrete Gaussian, we propose to project the involved values into a quotient group after shifting and then apply the random masks as shown in linein Algorithm 1.
Revised: To reconcile secure aggregation with discrete Gaussian, we propose to project the involved values into a quotient group after shifting and then apply the random masks as shown in line 16 in Algorithm 1.

Original: According to the post-processing theorem of RDP (Mironov (2017)), the result still follows rigorous R´enyi differential privacy as proved in Appendix C. Note that we consider a simpliﬁed version of the full secure aggregation protocol (Bonawitz et al.
Revised: According to the post-processing theorem of RDP (Mironov (2017)), the result still follows rigorous R ´ enyi differential privacy as proved in Appendix C. Note that we consider a simpliﬁed version of the full secure aggregation protocol (Bonawitz et al.

Original: in Algorithmand omit many interesting details such as the generation of the random masks and how to deal with dropout.
Revised: in Algorithm 1 and omit many interesting details such as the generation of the random masks and how to deal with dropout.

Original: We deem this to be enough to clarify the idea behind the reconciliation.
Revised: We deem this to be enough to clarify the idea behind the reconciliation.

Original: The complete version of secure aggregation can be reconciled using exactly the same trick.
Revised: The complete version of secure aggregation can be reconciled using exactly the same trick.

Original: Theorem 6 (Informal) .
Revised: Theorem 3 (Informal) .

Original: Distributed discrete Gaussian mechanism with secure aggregation obeys the same RDP guarantee as vanilla global discrete Gaussian mechanism with the same parameters.
Revised: Distributed discrete Gaussian mechanism with secure aggregation obeys the same RDP guarantee as vanilla global discrete Gaussian mechanism with the same parameters.

Original:  C OMMUNICATION P ROTOCOL & U TILITY A NALYSIS
Revised:  C OMMUNICATION P ROTOCOL & U TILITY A NALYSIS

Original: In this section, we present our communication protocol in detail and discuss the communication cost and the estimation error of D 2 P-F ED with direct comparison to cpSGD.
Revised: In this section, we present our communication protocol in detail and discuss the communication cost and the estimation error of D 2 P-F ED with direct comparison to cpSGD.

Original: The drastic improvement of D 2 P-F ED mainly comes from the tight composition of discrete Gaussian mechanism compared to binomial mechanism in cpSGD.
Revised: The drastic improvement of D 2 P-F ED mainly comes from the tight composition of discrete Gaussian mechanism compared to binomial mechanism in cpSGD.

Original: 6. C OMMUNICATION P ROTOCOL
Revised: 6. C OMMUNICATION P ROTOCOL

Original: As the ﬁrst step, we leverage stochastic k -level quantization proposed by McMahan et al.
Revised: As the ﬁrst step, we leverage stochastic k -level quantization proposed by McMahan et al.

Original: (2016) to lower the communication cost as described in line 13-14 in Algorithm 1.
Revised: (2016) to lower the communication cost as described in line 12-14 in Algorithm 1.

Original: If we denote vanilla stochastic k-level quantization with π k , then we successfully reduce the per-round communication cost C ( π k , g [ n ] ) =
Revised: If we denote vanilla stochastic k-level quantization with π k , then we successfully reduce the per-round communication cost C ( π k , g [ n ] ) =

Original: n · ( d (cid:100) log 2 k (cid:101) + ˜ O (1)) .
Revised: n · ( d (cid:100) log 2 k (cid:101) + ˜ O (1)) .

Original: However, stochastic quantization sacriﬁces some accuracy for communication efﬁciency.
Revised: However, stochastic quantization sacriﬁces some accuracy for communication efﬁciency.

Original: Concretely, E ( π k , g [ n ] ) =
Revised: Concretely, E ( π k , g [ n ] ) =

Original: O ( dn · 1 n (cid:80) ni =1 (cid:107) g
Revised: O ( dn · 1 n (cid:80) ni =1 (cid:107) g

Original: ( i ) (cid:107) 22 ) .
Revised: ( i ) (cid:107) 22 ) .

Original: Since  d is tens of thousands to hundreds of thousands in federated learning, the estimation error of the mean is too large.
Revised: Since the dimension of parameters d is tens of thousands to hundreds of thousands in federated learning, the estimation error of the mean is too large.

Original: Thus to reduce the estimation error, we randomly rotate the difference vector (McMahan et al., 2016) as the second step.
Revised: Thus to reduce the estimation error, we randomly rotate the difference vector (McMahan et al., 2016) as the second step.

Original: The key intuition is that the MSE of stochastic uniform quantization is O ( dn ( g max ) 2 ) .
Revised: The key intuition is that the MSE of stochastic uniform quantization is O ( dn ( g max ) 2 ) .

Original: With random rotation, we can limit g max to (cid:113) log d d w.h.p.
Revised: With random rotation, we can limit g max to (cid:113) log d d w.h.p.

Original: so the MSE will be improved to O ( log dn ) .
Revised: so the MSE will be improved to O ( log dn ) .

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2018) also leverages random rotation to reduce MSE.
Revised: (2018) also leverages random rotation to reduce MSE.

Original: However, in their setting, random rotation intrinsically harms their privacy guarantee because the (cid:96) ∞ -sensitivity might increase with rotation.
Revised: However, in their setting, random rotation intrinsically harms their privacy guarantee because the (cid:96) ∞ -sensitivity might increase with rotation.

Original: A natural advantage of discrete Gaussian is that the privacy guarantee only depends on (cid:96) 2 -sensitivity which is an invariant under rotation.
Revised: A natural advantage of discrete Gaussian is that the privacy guarantee only depends on (cid:96) 2 -sensitivity which is an invariant under rotation.

Original: Thus, random rotation does not harm our privacy guarantee at all.
Revised: Thus, random rotation does not harm our privacy guarantee at all.

Original: We omit the details here and refer the interested readers to McMahan et al.
Revised: We omit the details here and refer the interested readers to McMahan et al.

Original: We denote the protocol using  quantization and random rotation with π ( rot ) k .
Revised: We denote the protocol using k -level quantization and random rotation with π ( rot ) k .

Original: We know that C ( π ( rot ) k , g [ n ] ) remains the same while the MSE error is reduced as E ( π ( rot ) 
Revised: We know that C ( π ( rot ) k , g [ n ] ) remains the same while the MSE error is reduced to E ( π ( rot ) k , g [ n ] )

Original: k
Revised: =

Original: [Equation]
Revised: O ( log d n · 1 n (cid:80) n i =1 (cid:107) g ( i ) (cid:107) 2 2 ) .

Original: 6. C ONVERGENCE R ATE OF D 2 P-F ED
Revised: 6. C ONVERGENCE R ATE OF D 2 P-F ED

Original: In this section, we relate the convergence rate with mean squared error using Corollary 1 and analyze the mean squared error of mean estimation in D 2 P-F ED .
Revised: In this section, we relate the convergence rate with mean squared error using Corollary 4 and analyze the mean squared error of mean estimation in D 2 P-F ED .

Original: Note that we assume each client executes one iteration in each round so g equals to the gradient or belongs to the sub-gradients.
Revised: Note that we assume each client executes one iteration in each round so g equals to the gradient or belongs to the sub-gradients.

Original: Corollary 1 (Ghadimi & Lan (2013)) .
Revised: Corollary 4 (Ghadimi & Lan (2013)) .

Original: Let F ( w ) =
Revised: Let F ( w ) =

Original: L ( w , D ) for some given distribution D .
Revised: L ( w , D ) for some given distribution D .

Original: Let
Revised: Let

Original: F ( w ) be L -smooth and ∀ w , (cid:107)∇ F ( w ) (cid:107) ≤ ρ .
Revised: F ( w ) be L -smooth and ∀ w , (cid:107)∇ F ( w ) (cid:107) ≤ ρ .

Original: Let w 0 satisfy F ( w 0 ) − F ( w ∗ ) ≤ ρ F .
Revised: Let w 0 satisfy F ( w 0 ) − F ( w ∗ ) ≤ ρ F .

Original: Then after T rounds
Revised: Then after T rounds

Original: [Equation]
Revised: [Equation]

Original: , where λ 2 = max 1 ≤ t ≤ T 2 E [ (cid:107) g ( w t ) − ∇ F ( w t ) (cid:107) 22 ] + 2 max 1 ≤ t ≤ T E q

Original: [ (cid:107) g ( w t ) − ˜ g ( w t ) (cid:107) 22 ] , and

Original: B = max 1 ≤ t ≤ T (cid:107) g ( w t ) − ˜ g ( w t ) (cid:107) .

Original: As corollary 1 indicates, for a given gradient bound, the convergence rate approximately grows with the reciprocal of MSE.
Revised: As corollary 4 indicates, for a given gradient bound, the convergence rate approximately grows with the reciprocal of MSE.

Original: Thus, we analyze D 2 P-F ED ’s MSE and obtain the following theorem.
Revised: Thus, we analyze D 2 P-F ED ’s MSE and obtain the following theorem.

Original: The proof is delayed to Appendix D due to space limitation.
Revised: The proof is delayed to Appendix D due to space limitation.

Original: Theorem 7.
Revised: Theorem 4.

Original: If we choose σ ≥ 1 / √ 2 π , the mean squared error isE
Revised: If we choose σ ≥ 1 / √ 2 π , the mean squared error is

Original: [Equation]
Revised: [Equation]

Original: where Φ is the cumulative distribution function (CDF) of the standard normal distribution.
Revised: where Φ is the cumulative distribution function (CDF) of the standard normal distribution.

Original: Remark.
Revised: Remark 1: Choice of g max . As indicated by Theorem 4, the dominant term in MSE is proportional to the square of g max . A natural choice of g max is the clipping bound D . If we want to match up with the MSE guarantee in cpSGD: O ( σ 2 log( d ) n ( k − 1) 2 ) , we need to inherit their choice of g max

Original: If we choose g max as O ( D (cid:113) log( d ) d ) following cpSGD, then the MSE is bounded by 
Revised: = O ( D (cid:113) log( d ) d ) , and this can be achieved by clipping l ∞ norm of the gradient after random rotation. For instance, according to Lemma 8 in Agarwal et al. (2018), we can choose g max =

Original: O ( σ 2 log( d ) n ( k − 1) 2 ) which is on the same order of cpSGD.
Revised: [Equation] d . In that case, the possibility that the maximum of g exceeds g max is at most δ . It follows that the possibility that the (cid:96) ∞ -clipping really changes the update is bounded by δ .

Original: If some values in the updates exceed g max under the above choice, the values should be clipped after random rotation.
Revised: Hence, the RHS of the MSE bound in Theorem 4 evolves to (1 − [Equation] , which is on the same order with the original bound given δ is small. Remark 2: Comparison with cpSGD.

Original: 6. C OMMUNICATION C OST OF D 2 P-F ED 
Revised: As the MSE bound is on the same order with cpSGD (even the constants are close!), a natural question is ”what is the advantage of D 2 P-F ED over cpSGD in terms of MSE?” Indeed, the advantage stems from a smaller standard deviation of the noise. Given a ﬁxed privacy budget, due to the tighter composition of sub-sampled RDP (Wang et al., 2018), each round of D 2 P-F ED can have more privacy budget and thus smaller noise scale. Plugging a smaller σ into Theorem 4 will give a better MSE and thus a better convergence rate as cpSGD follows a similar convergence rate bound.

Revised: Moreover, according to Figure 1 in Agarwal et al.

Revised: (2018), even with the same noise scale, Gaussian noise provides stronger privacy guarantee than Binomial noise.

Revised: As discrete Gaussian noise follows the same RDP bound as Gaussian noise, we believe discrete

Revised: Gaussian can map the same-scale noise to lower privacy cost.

Revised: 6. C OMMUNICATION C OST OF D 2 P-F ED

Original: First, we provide the trivial per-round communication cost which is exactly the same as cpSGD.
Revised: First, we provide the trivial per-round communication cost which is exactly the same as cpSGD.

Original: Theorem 8.
Revised: Theorem 5.

Original: The per-round communication cost of D 2 P-F ED is
Revised: The per-round communication cost of D 2 P-F ED is

Original: [Equation]
Revised: [Equation]

Original: Now let’s compare the number of rounds in cpSGD and D 2 P-F ED qualitatively.
Revised: Now let’s compare the number of rounds in cpSGD and D 2 P-F ED qualitatively.

Original: Note that during the following discussion, we usually omit δ for the ease of clariﬁcation.
Revised: Note that during the following discussion, we usually omit δ for the ease of clariﬁcation and assume that δ is ﬁxed.

Original: For cpSGD, the known tightest bound is by the combination of standard privacy ampliﬁcation (Balle et al., 2018) and advanced composition (Dwork et al., 2010).
Revised: For cpSGD, the known tightest bound is by the combination of standard privacy ampliﬁcation (Balle et al., 2018) and advanced composition (Dwork et al., 2010).

Original: Concretely, if a mechanism costs (cid:15) privacy budget, then after composed with sub-sampling the privacy budget is reduced to O ( γ(cid:15) ) where γ is the sub-sampling rate.
Revised: Concretely, if a mechanism costs (cid:15) privacy budget, then after composed with sub-sampling the privacy budget is reduced to O ( γ(cid:15) ) where γ is the sub-sampling rate.

Original: If the mechanism is composed sequentially T times, the privacy budget grows to O ( (cid:112) T log(1 /δ )
Revised: If the mechanism is composed sequentially T times, the privacy budget grows to O ( (cid:112) T log(1 /δ )

Original: (cid:15) ) .
Revised: (cid:15) ) .

Original: Thus, the total privacy budget of cpSGD is O ( γ (cid:112) T log(1 /δ )
Revised: Thus, the total privacy budget of cpSGD is O ( γ (cid:112) T log(1 /δ )

Original: (cid:15) ) .
Revised: (cid:15) ) .

Original: On the other hand, D 2 P-F ED provides a total privacy budget of O ( γ √ T(cid:15) ) , saving a factor of (cid:112) log(1 /δ ) .
Revised: On the other hand, D 2 P-F ED provides a total privacy budget of O ( γ √ T(cid:15) ) , saving a factor of (cid:112) log(1 /δ ) .

Original: Since δ is typically very small, the saving is quite signiﬁcant.
Revised: Since δ is typically very small, the saving is quite signiﬁcant.

Original: If the privacy budgets for the two protocols are the same, then D 2 P-F ED can use noise with O ( (cid:112) log(1 /δ )) smaller scale than cpSGD in each round.
Revised: If the privacy budgets for the two protocols are the same, then D 2 P-F ED can use noise with O ( (cid:112) log(1 /δ )) smaller scale than cpSGD in each round.

Original: This will lead to a O ( (cid:112) log(1 /δ ))
Revised: This will lead to a O ( (cid:112) log(1 /δ ))

Original: -time faster convergence.
Revised: -time faster convergence.

Original: Then for a given gradient bound, D 2 P-F ED can reach it with O (log(1 /δ ))
Revised: Then for a given gradient bound, D 2 P-F ED can reach it with O (log(1 /δ ))

Original: -time fewer rounds and thus save O (log(1 /δ ))
Revised: -time fewer rounds and thus save O (log(1 /δ ))

Original: -time communication cost.
Revised: -time communication cost.

Original: Remark.

Original: Both D 2 P-F ED and cpSGD intrinsically require secure aggregation to establish their privacy guarantee.
Revised: Both D 2 P-F ED and cpSGD intrinsically require secure aggregation to establish their privacy guarantee.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2018) did not discuss the issue explicitly.
Revised: (2018) did not discuss the issue explicitly.

Original: As pointed out in Bonawitz et al.
Revised: As pointed out in Bonawitz et al.

Original: (2017), once combined with secure aggregation, each ﬁeld has to expand at least γn times ( γn is the number of chosen clients) to prevent overﬂow of the sum.
Revised: (2017), once combined with secure aggregation, each ﬁeld has to expand at least γn times ( γn is the number of chosen clients) to prevent overﬂow of the sum.

Original:  E VALUATION
Revised:  E VALUATION

Original: We would like to answer the following three questions using empirical evaluation: (1) How D 2 PF ED performs in a multi-round federated learning compared to cpSGD under either the same privacy guarantee or the same communication cost?
Revised: We would like to answer the following three questions using empirical evaluation: (1) How D 2 PF ED performs in a multi-round federated learning compared to cpSGD under either the same privacy guarantee or the same communication cost?

Original: (2) How different choices of hyper-parameters affect the performance of D 2 P-F ED ?
Revised: (2) How different choices of hyper-parameters affect the performance of D 2 P-F ED ?

Original: (3) Does D 2 P-F ED work under heterogeneous data distribution?
Revised: (3) Does D 2 P-F ED work under heterogeneous data distribution?

Original: Due to space limitation, we present our main results for (1) in this section and defer the results for (2) and (3) to Appendix E.
Revised: Due to space limitation, we present our main results for (1) in this section and defer the results for (2) and (3) to Appendix E.

Original: 7. E XPERIMENT S ETUP
Revised: 7. E XPERIMENT S ETUP

Original: To answer the above questions, we evaluated D 2 P-F ED and cpSGD on INFIMNIST (Bottou (2007)) and CIFAR10 (Krizhevsky et al.
Revised: To answer the above questions, we evaluated D 2 P-F ED and cpSGD on INFIMNIST (Bottou (2007)) and CIFAR10 (Krizhevsky et al.

Original: We sampled 10M hand-written digits from INFIMNIST and randomly split the data among 100K clients.
Revised: We sampled 10M hand-written digits from INFIMNIST and randomly split the data among 100K clients.

Original: In each round, 100 clients are randomly chosen to upload their difference vectors to train a three-layer MLP.
Revised: In each round, 100 clients are randomly chosen to upload their difference vectors to train a three-layer MLP.

Original: For CIFAR10, we select 10 out ofclients in each round to train a 2-layer convolutional network.
Revised: For CIFAR10, we select 10 out ofclients in each round to train a 2-layer convolutional network.

Original: All RDP bounds are converted to ( (cid:15), δ ) -DP for the ease of comparison and the total δ is set to 1 e − 5 for all experiments.
Revised: All RDP bounds are converted to ( (cid:15), δ ) -DP for the ease of comparison and the total δ is set to 1 e − 5 for all experiments.

Original: [Figure] [Figure] CIFAR [Figure]
Revised: 7. M ODEL A CCURACY VS . P RIVACY B UDGET

Original: To answer the ﬁrst question, we studied model accuracy under the same privacy budget as shown in
Revised: To answer the ﬁrst question, we studied model accuracy under the same privacy budget as shown in

Original: Figure 1a.
Revised: Figure 1a.

Original: Compared with cpSGD, D 2 P-F ED achieves 6.7% higher model accuracy on INFIMNIST  and 9.75% higher model accuracy on CIFAR10 after convergence.
Revised: Compared with cpSGD, D 2 P-F ED achieves 4.7% higher model accuracy on INFIMNIST [Figure] and 13.0% higher model accuracy on CIFAR10 after convergence.

Original: As expected, D 2 P-F ED composes far tighter under sub-sampling as the lines are much sharper than those of cpSGD.
Revised: As expected, D 2 P-F ED composes far tighter under sub-sampling as the lines are much sharper than those of cpSGD.

Original: Consequently, D 2 P-F ED converges at a smaller privacy budget than cpSGD as well.
Revised: Consequently, D 2 P-F ED converges at a smaller privacy budget than cpSGD as well.

Original: Although in Figure 1a cpSGD has better accuracy in the high privacy region, it is not necessarily the case but depends on the scale of the discrete Gaussian noise, as studied in Section E.1.
Revised: Although in Figure 1a cpSGD has better accuracy in the high privacy region, it is not necessarily the case but depends on the scale of the discrete Gaussian noise, as studied in Section E.1.

Original: Note that the results for cpSGD in
Revised: Note that the results for cpSGD in

Original: Figure 1 is different from the results in Figure 2 in the original paper (Agarwal et al., 2018).
Revised: Figure 1 is different from the results in Figure 2 in the original paper (Agarwal et al., 2018).

Original: The reason is that in the original paper, they do not sub-sample clients in each round but assign each client to exactly one round beforehand to avoid composition which cpSGD cannot handle well.
Revised: The reason is that in the original paper, they do not sub-sample clients in each round but assign each client to exactly one round beforehand to avoid composition which cpSGD cannot handle well.

Original: However, the scheme is far from practical in the real world due to the dynamic nature of the clients.
Revised: However, the scheme is far from practical in the real world due to the dynamic nature of the clients.

Original: 7. M ODEL A CCURACY VS . C OMMUNICATION C OST
Revised: 7. M ODEL A CCURACY VS . C OMMUNICATION C OST

Original: To answer the second question, we also studied the model accuracy under the same communication cost.
Revised: To answer the second question we also studied the model accuracy under the same communication cost.

Original: As shown in Figure 1b, D 2 P-F ED consistently achieves better model accuracy under the same communication cost on both INFIMNIST and CIFAR10.
Revised: As shown in Figure 1b, D 2 P-F ED consistently achieves better model accuracy under the same communication cost on both INFIMNIST and CIFAR10.

Original: The main reason is that the tight composition property allows D 2 P-F ED to use smaller per-feature communication cost while still achieving better accuracy.
Revised: The main reason is that the tight composition property allows D 2 P-F ED to use smaller per-feature communication cost while still achieving better accuracy.

Original: As a concrete instance, D 2 P-F ED with 50% compression rate can achieve better accuracy than cpSGD with 25% compression rate.
Revised: As a concrete instance, D 2 P-F ED with 50% compression rate can achieve better accuracy than cpSGD with 25% compression rate.

Original: cpSGD with 50% compression rate either leads to an unacceptable privacy budget or does not converge.
Revised: cpSGD with 50% compression rate either leads to an unacceptable privacy budget or does not converge.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we developed D 2 P-F ED to achieve both differential privacy and communication efﬁciency in the context of federated learning.
Revised: In this work we developed D 2 P-F ED to achieve both differential privacy and communication efﬁciency in the context of federated learning.

Original: By applying the discrete Gaussian mechanism to the private data transmission, D 2 P-F ED provides stronger privacy guarantee, better composability and smaller communication cost than the only prior work, cpSGD, both theoretically and empirically.
Revised: By applying the discrete Gaussian mechanism to the private data transmission, D 2 P-F ED provides stronger privacy guarantee, better composability and smaller communication cost than the only prior work, cpSGD, both theoretically and empirically.

Original: A P ROOF FOR T HEOREM
Revised: A P ROOF FOR T HEOREM

Original: The proof is straightforward as shown below.
Revised: We consider the R ´ enyi divergence between two discrete Gaussian distributions differing in the mean value.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B P ROOF FOR T HEOREM
Revised: (1) Because range ( f ) ⊆ L , we only consider µ ∈ L . Thus the denominator of N L (0 , σ 2 ) and N L ( µ, σ 2 ) cancels out as (cid:80) L exp( − ( x − µ ) 2 / (2 σ 2 )) is periodic.

Revised: (2) (cid:80) L

Revised: [Equation]

Revised: [Equation]

Revised: σ 2 ))

Revised: [Equation]

Revised: L

Revised: [Equation]

Revised: [Equation]

Revised: 2 σ 2 ))

Revised: =

Revised: [Equation]

Revised: e ) ≤ 1 where ϑ is the Jacobi theta function (Wikipedia).

Revised: B P ROOF FOR T HEOREM

Original: Proof.
Revised: Proof.

Original: The l 2 sensitivity of δ is naturally bounded by 2 D .
Revised: The l 2 sensitivity of δ is naturally bounded by 2 D .

Original: The rotation does not change the sensitivity.
Revised: The rotation does not change the sensitivity.

Original: The k -level quantization might expand the space as shown in Figure 2.
Revised: The k -level quantization might expand the space as shown in Figure 2.

Original: An upper bound on the radius of the red circle is D +
Revised: An upper bound on the radius of the red circle is D +

Original: √ d
Revised: √ d

Original: Dk − 1 .
Revised: Dk − 1 .

Original: When we take k = √ d − 1 , it reduces to 2 D .
Revised: When we take k = √ d − 1 , it reduces to 2 D .

Original: Thus, the upper bound on the sensitivity is 4 D .
Revised: Thus, the upper bound on the sensitivity is 4 D .

Original: [Figure]

Original: C P ROOF FOR T HEOREM
Revised: C P ROOF FOR T HEOREM

Original: We ﬁrst prove the following lemma.
Revised: We ﬁrst prove the following lemma.

Revised: [Figure]

Original: Lemma 3.
Revised: Lemma 3.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Then we prove Theorem 6 as follows.
Revised: Then we prove Theorem 3 as follows.

Original: Proof.
Revised: Proof.

Original: Given Lemma 
Revised: Given Lemma 3,

Original: [Equation]
Revised: [Equation]

Original: The expression (cid:80) i ∈ S
Revised: The expression (cid:80) i ∈ S

Original: ˜ g ( i ) t,dp forms a centralized discrete Gaussian mechanism and according to the post-processing theorem, the same RDP guarantee still holds.
Revised: ˜ g ( i ) t,dp forms a centralized discrete Gaussian mechanism and according to the post-processing theorem, the same RDP guarantee still holds.

Original: D P ROOF FOR T HEOREM
Revised: D P ROOF FOR T HEOREM

Original: Proof Sketch.
Revised: Proof Sketch.

Original: Before starting the proof, we introduce two lemmas for the proof.
Revised: Before starting the proof, we introduce two lemmas for the proof.

Original: Lemma 4 (Proposition 19 from Canonne et al.
Revised: Lemma 4 (Proposition 19 from Canonne et al.

Original: For all σ ∈ R with σ > 0 ,
Revised: For all σ ∈ R with σ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, if σ 2 ≤ 1 3 , then V [ N Z (0 , σ 2 )] ≤ 3 · e − 

Revised: Moreover, if σ 2 ≤ 1 3 , then V [ N Z (0 , σ 2 )] ≤ 3 · e −

Original: [Equation]
Revised: [Equation]

Original: Lemma 5 (Proposition 23 from Canonne et al.
Revised: Lemma 5 (Proposition 23 from Canonne et al.

Original: For all m ∈ Z with m ≥ 1 and all σ ∈ R with σ > 0 ,
Revised: For all m ∈ Z with m ≥ 1 and all σ ∈ R with σ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, if σ ≥ 1 / √ 2 π , we have
Revised: Moreover, if σ ≥ 1 / √ 2 π , we have

Original: [Equation]
Revised: [Equation]

Original: Now we start our proof.
Revised: Now we start our proof.

Original: MSE can be rewritten in the following format.
Revised: MSE can be rewritten in the following format.

Original: [Equation]
Revised: [Equation]

Original: For each (cid:63) =
Revised: For each (cid:63) =

Original: E [( ˆ¯ X i ( j ) − X
Revised: E [( ˆ¯ X i ( j ) − X

Original: i
Revised: i

Original: ( j ))
Revised: ( j ))

Original: 2 ] , we need to consider two cases.
Revised: 2 ] , we need to consider two cases.

Original: If no overﬂow happens, due to
Revised: If no overﬂow happens, due to

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: If overﬂow happens, trivially we have (cid:63) o ≤ k 2 .
Revised: If overﬂow happens, trivially we have (cid:63) o ≤ k 2 .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: With Lemma 5, we can bound the probabilities and get the ﬁnal MSE result in the theorem.
Revised: With Lemma 5, we can bound the probabilities and get the ﬁnal MSE result in the theorem.

Original: E O THER E VALUATION E. I NFLUENCE OF N OISE S CALE
Revised: E O THER E VALUATION E. I NFLUENCE OF N OISE S CALE

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The hyper-parameter of the most vital interest in D 2 P-F ED is the scale of the noise.
Revised: The hyper-parameter of the most vital interest in D 2 P-F ED is the scale of the noise.

Original: To understand the effect of the noise scale, we evaluated D 2 P-F ED on INFIMNIST, with 3 different choices of noise scale as shown in Figure 3.
Revised: To understand the effect of the noise scale, we evaluated D 2 P-F ED on INFIMNIST, with 3 different choices of noise scale as shown in Figure 3.

Original: It is no surprise that the higher the noise scale, the smaller the privacy budget and the lower the model accuracy.
Revised: It is no surprise that the higher the noise scale, the smaller the privacy budget and the lower the model accuracy.

Original: This also illustrates the claim we have in Section 7.2 that
Revised: This also illustrates the claim we have in Section 7.2 that

Original: D 2 P-F ED can also have relatively good performance in the high privacy region at the cost of the model accuracy.
Revised: D 2 P-F ED can also have relatively good performance in the high privacy region at the cost of the model accuracy.

Original: E. I NFLUENCE OF H ETEROGENEOUS D ATA D ISTRIBUTION
Revised: E. I NFLUENCE OF H ETEROGENEOUS D ATA D ISTRIBUTION

Original: It is well known that data is sometimes heterogeneously distributed among clients in a federated learning system.
Revised: It is well known that data is sometimes heterogeneously distributed among clients in a federated learning system.

Original: Thus, to better understand D 2 P-F ED ’s behavior under heterogeneous data distribution, we simulated heterogenenous data distribution by distributing the INFIMNIST data according to the classes and evaluated D 2 P-F ED on these clients.
Revised: Thus, to better understand D 2 P-F ED ’s behavior under heterogeneous data distribution, we simulated heterogenenous data distribution by distributing the INFIMNIST data according to the classes and evaluated D 2 P-F ED on these clients.

Original: The results are shown in Figure 4 and we can see that under heterogeneous data distribution the model accuracy drops by more than 10%.
Revised: The results are shown in Figure 4 and we can see that under heterogeneous data distribution the model accuracy drops by more than 10%.

Original: This complies with the previous empirical results and there have been a line of researches focusing on addressing the issue (Yurochkin et al., 2019a;b; Wang et al., 2020).
Revised: This complies with the previous empirical results and there have been a line of researches focusing on addressing the issue (Yurochkin et al., 2019a;b; Wang et al., 2020).

Original: Although orthogonal to this paper, we deem it as an interesting open problem how to integrate these works with D 2 P-F ED . 
Revised: Although orthogonal to this paper, we deem it as an interesting open problem how to integrate these works with D 2 P-F ED . [Figure]

Revised: E. I NFLUENCE OF G ROUP S IZE q

Revised: We also ran D 2 P-F ED with multiple choices of discrete group size q .

Revised: We observe that once the noise scale σ is ﬁxed

Original: [Figure]
Revised: [Figure]

Revised: the performance is relatively robust to q .

Revised: [Figure]