Original: Differentially Private Federated Learning with Normalized Updates
Revised: Differentially Private Federated Learning with Normalized Updates

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: The customary approach for client-level differentially private federated learning (FL) is to add
Revised: The customary approach for client-level differentially private federated learning (FL) is to add

Original: Gaussian noise to the average of the clipped client updates.
Revised: Gaussian noise to the average of the clipped client updates.

Original: Clipping is associated with the following issue: as the client updates fall below the clipping threshold, they get drowned out by the added noise, inhibiting convergence.
Revised: Clipping is associated with the following issue: as the client updates fall below the clipping threshold, they get drowned out by the added noise, inhibiting convergence.

Original: To mitigate this issue, we propose replacing clipping with normalization, where we use only a scaled version of the unit vector along the client updates.
Revised: To mitigate this issue, we propose replacing clipping with normalization, where we use only a scaled version of the unit vector along the client updates.

Original: Normalization ensures that the noise does not drown out the client updates even when the original updates are small.
Revised: Normalization ensures that the noise does not drown out the client updates even when the original updates are small.

Original: We theoretically show that the resulting normalization-based private FL algorithm attains better convergence than its clipping-based counterpart on convex objectives in over-parameterized settings.
Revised: We theoretically show that the resulting normalization-based private FL algorithm attains better convergence than its clipping-based counterpart on convex objectives in over-parameterized settings.

Original: 1. Introduction
Revised: 1. Introduction

Original: The typical approach to client/user -level differentially private federated learning (FL), exemplified by DP-FedAvg [15, 29], involves each client first computing its update as it would have been under vanilla FedAvg [23], then clipping its norm to a pre-determined threshold, and adding noise to the clipped update before sending it to the server.
Revised: The typical approach to client/user -level differentially private federated learning (FL), exemplified by DP-FedAvg [15, 29], involves each client first computing its update as it would have been under vanilla FedAvg [23], then clipping its norm to a pre-determined threshold, and adding noise to the clipped update before sending it to the server.

Original: The level of added noise is always proportional to the clipping threshold, with the proportionality constants depending on the desired level of privacy.
Revised: The level of added noise is always proportional to the clipping threshold, with the proportionality constants depending on the desired level of privacy.

Original: Our key insight is that this clipping -based approach has a fundamental issue: when the client updates have small magnitudes much lesser than the clipping threshold, they are completely drowned out by the added noise whose magnitude is always proportional to the clipping threshold.
Revised: Our key insight is that this clipping -based approach has a fundamental issue: when the client updates have small magnitudes much lesser than the clipping threshold, they are completely drowned out by the added noise whose magnitude is always proportional to the clipping threshold.

Original: Based on this insight, we propose a variant algorithm – one that does not clip the client updates, but instead normalizes updates, i.e., rescales them to always have a fixed norm , before adding noise proportional to this fixed norm for privacy.
Revised: Based on this insight, we propose a variant algorithm – one that does not clip the client updates, but instead normalizes updates, i.e., rescales them to always have a fixed norm , before adding noise proportional to this fixed norm for privacy.

Original: We call our modified method DP-NormFedAvg .
Revised: We call our modified method DP-NormFedAvg .

Original: The rescaling of updates in DP-NormFedAvg ensures that client updates are never overwhelmed by the added noise, even if the original updates are small.
Revised: The rescaling of updates in DP-NormFedAvg ensures that client updates are never overwhelmed by the added noise, even if the original updates are small.

Original: We succinctly state the updates of FedAvg , DP-FedAvg with clipping and DP-NormFedAvg in Table 1.
Revised: We succinctly state the updates of FedAvg , DP-FedAvg with clipping and DP-NormFedAvg in Table 1.

Original: DP-FedAvg with clipping and DP-NormFedAvg are stated in detail in Algorithm 1 Options (i) and (ii), respectively.
Revised: DP-FedAvg with clipping and DP-NormFedAvg are stated in detail in Algorithm 1 Options (i) and (ii), respectively.

Original: We also show an example of the superior convergence of normalization compared to clipping in Figure 1.
Revised: We also show an example of the superior convergence of normalization compared to clipping in Figure 1.

Original: Our main contributions are summarized next:

Original: (a) In Section 3, we present DP-NormFedAvg (Alg

Original: . 1 Option (ii)) where we replace the usual practice of update clipping by update normalization (i

Original: e., using a scaled version of the unit vector along the update) for bounding sensitivity in private FL.

Original: Our motivation

Original: for

Original: advocating normalization is

Original: that normalization

Original: has

Original: a higher

Original: signal (

Original: viz.

Original: , update norm)

Original: to

Original: noise ratio (SNR) than clipping

Original: which should intuitively result in better convergence

Original: for

Original: normalization

Original: ; this aspect

Original: is

Original: discussed

Original: © .

Original: Algorithm:
Revised: Algorithm:

Original: FedAvg
Revised: FedAvg

Original: DP-FedAvg with Clipping
Revised: DP-FedAvg with Clipping

Original: DP
Revised: DP

Original: [Figure]
Revised: [Figure]

Original: NormFedAvg

Original: ClientSends
Revised: Client

Original: [Figure]
Revised: [Figure]

Original: uu min( [Figure] u ∥
Revised: u

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C
Revised: Our main contributions are summarized next: (a)

Original: [Figure]
Revised: In Section 3, we present DP-NormFedAvg (Alg. 1 Option (ii)) where we replace the usual practice of update clipping by update normalization (i

Original: [Figure]
Revised: [Figure]

Original:  in Section 3.1.
Revised: e., using a scaled version of the unit vector along the update) for bounding sensitivity in private FL. Our motivation for advocating normalization is that normalization has a higher signal (viz., update norm) to noise ratio (SNR) than clipping which should intuitively result in better convergence for normalization; this aspect is discussed in Section 3.1.

Original: We provide convergence results for DP-NormFedAvg as well as the standard algorithm of DP-FedAvg with clipping (Alg. 1 Option (i)) in the smooth convex case in Theorem 4.In Section 4.
Revised: We provide convergence results for DP-NormFedAvg as well as the standard algorithm of DP-FedAvg with clipping (Alg. 1 Option (i)) in the smooth convex case in Theorem 4.

Original: [Figure] we compare the convergence results of both algorithms, showing that normalization has better asymptotic convergence than clipping in over-parameterized settings.
Revised: In Section 4.1, we compare the convergence results of both algorithms, showing that normalization has better asymptotic convergence than clipping in over-parameterized settings.

Original: (b) In Appendix F and G, we demonstrate the superiority of normalization over clipping via experiments on a synthetic quadratic problem as well as on three benchmarking datasets, viz.
Revised: (b) In Appendix F and G, we demonstrate the superiority of normalization over clipping via experiments on a synthetic quadratic problem as well as on three benchmarking datasets, viz.

Original: , Fashion
Revised: , Fashion

Original: MNIST, CIFAR-10 and CIFAR-100, respectively.
Revised: MNIST, CIFAR-10 and CIFAR-100, respectively.

Original: For ε = 5 , the improvement offered by normalization over clipping w.r.t.
Revised: For ε = 5 , the improvement offered by normalization over clipping w.r.t.

Original: test accuracy is > 2 .
Revised: test accuracy is > 2 .

Original: 8 %, 2 . 1 % and 1 .
Revised: 8 %, 2 . 1 % and 1 .

Original: 5 % for CIFAR-100, Fashion MNIST and CIFAR-10, respectively; see Table 2 in Appendix G.
Revised: 5 % for CIFAR-100, Fashion MNIST and CIFAR-10, respectively; see Table 2 in Appendix G.

Original: 2. Preliminaries
Revised: 2. Preliminaries

Original: Federated Learning (FL) Setting: There are n clients, each with their own decentralized data, and a central server that has to train a model, parameterized by w ∈ R d , using the clients’ data.
Revised: Federated Learning (FL) Setting: There are n clients, each with their own decentralized data, and a central server that has to train a model, parameterized by w ∈ R d , using the clients’ data.

Original: Suppose the i th client has m training examples drawn from some distribution P i .
Revised: Suppose the i th client has m training examples drawn from some distribution P i .

Original: Then the i th client has an objective function f i ( w ) which is the average loss, w.r.t.
Revised: Then the i th client has an objective function f i ( w ) which is the average loss, w.r.t.

Original: some loss function, over its m samples, and the central server tries to minimize the average 1 loss f ( w ) , over the n clients, i.e., f ( w ) :
Revised: some loss function, over its m samples, and the central server tries to minimize the average 1 loss f ( w ) , over the n clients, i.e., f ( w ) :

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( w ) .
Revised: f i ( w ) .

Original: The setting where the data distributions of all the clients are identical, i.e.
Revised: The setting where the data distributions of all the clients are identical, i.e.

Original: P 1 = . . .
Revised: P 1 = . . .

Original: =
Revised: =

Original: P n , is known as the “homogeneous” setting.
Revised: P n , is known as the “homogeneous” setting.

Original: Other settings are known as “heterogeneous” settings.
Revised: Other settings are known as “heterogeneous” settings.

Original: We are able to naturally quantify the effect of heterogeneity on convergence as follows.
Revised: We are able to naturally quantify the effect of heterogeneity on convergence as follows.

Original: Definition 1 (Heterogeneity) Let w ∗ ∈ arg min
Revised: Definition 1 (Heterogeneity) Let w ∗ ∈ arg min

Original: w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ )
Revised: w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ )

Original: − min
Revised: − min

Original: w ′ ∈
Revised: w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) .
Revised: d f i ( w ′ ) .

Original: The system’s heterogeneity is quantified by some increasing function of { ∆ ∗ i } ni =1 .
Revised: The system’s heterogeneity is quantified by some increasing function of { ∆ ∗ i } ni =1 .

Original: The exact function of ∆ ∗ i ’s quantifying heterogeneity depends on the algorithm as well as data, and this will become clear when we present the convergence results.
Revised: The exact function of ∆ ∗ i ’s quantifying heterogeneity depends on the algorithm as well as data, and this will become clear when we present the convergence results.

Original: Note that if the per-client distributions (i.e, P i ’s) are similar, then we expect the ∆ ∗ i ’s to be small indicating smaller heterogeneity.
Revised: Note that if the per-client distributions (i.e, P i ’s) are similar, then we expect the ∆ ∗ i ’s to be small indicating smaller heterogeneity.

Original: Due to lack of space, we present the definitions of some standard concepts such as differential privacy and the Gaussian mechanism in Appendix A.
Revised: Due to lack of space, we present the definitions of some standard concepts such as differential privacy and the Gaussian mechanism in Appendix A.

Original: Definition 2 (A Key Quantity)
Revised: Definition 2 (A Key Quantity)

Original: All the theoretical results in this paper are expressed in terms of the following key quantity
Revised: All the theoretical results in this paper are expressed in terms of the following key quantity

Original: [Equation]
Revised: [Equation]

Original: where ( ε, δ ) -DP is the desired privacy level, n is the number of clients, d is the parameter dimension and q is the absolute constant in Theorem 3.
Revised: where ( ε, δ ) -DP is the desired privacy level, n is the number of clients, d is the parameter dimension and q is the absolute constant in Theorem 3.

Original: We also assume that n is sufficiently large so that ρ < 1 .
Revised: We also assume that n is sufficiently large so that ρ < 1 .

Original: Note that ρ increases as the level of privacy increases (i.e., ε and δ decrease), and vice versa.
Revised: Note that ρ increases as the level of privacy increases (i.e., ε and δ decrease), and vice versa.

Original: Notation: Throughout the rest of this paper, we denote the ℓ 2 norm by ∥ .
Revised: Notation: Throughout the rest of this paper, we denote the ℓ 2 norm by ∥ .

Original: ∥ (omitting the subscript 2).
Revised: ∥ (omitting the subscript 2).

Original: Vectors and matrices are written in boldface.
Revised: Vectors and matrices are written in boldface.

Original: For any a ∈ N , we denote the set { 1 , . . .
Revised: For any a ∈ N , we denote the set { 1 , . . .

Original: , a } by [ a ] , and the uniform distribution over { 0 , . . . , a } by unif [0 , a ] .
Revised: , a } by [ a ] , and the uniform distribution over { 0 , . . . , a } by unif [0 , a ] .

Original: For a function h and any point θ in its domain Θ , the “ suboptimality gap ” at θ means h ( θ ) − min θ
Revised: For a function h and any point θ in its domain Θ , the “ suboptimality gap ” at θ means h ( θ ) − min θ

Original: ′ ∈
Revised: ′ ∈

Original: Θ h ( θ ′ ) .
Revised: Θ h ( θ ′ ) .

Original: The clipping function clip : R d × R + −→ R d is defined as clip ( z , c ) :=
Revised: The clipping function clip : R d × R + −→ R d is defined as clip ( z , c ) :=

Original: z min(1 , c
Revised: z min(1 , c

Original: ∥ z ∥ ) ; here, c is known as the clipping threshold.
Revised: ∥ z ∥ ) ; here, c is known as the clipping threshold.

Original: The normalization function norm : R d − { 0 d } × R + −→ R d is defined as norm ( z , c ) :=
Revised: The normalization function norm : R d − { 0 d } × R + −→ R d is defined as norm ( z , c ) :=

Original: c z ∥
Revised: c z ∥

Original: z ∥ ; here, c is the scaling factor and it is analogous to the clipping threshold in the clip ( . ) function.
Revised: z ∥ ; here, c is the scaling factor and it is analogous to the clipping threshold in the clip ( . ) function.

Original: Also, note that ∥ clip ( z , c ) ∥ ≤ ∥ norm ( z , c ) ∥ ≤ c .
Revised: Also, note that ∥ clip ( z , c ) ∥ ≤ ∥ norm ( z , c ) ∥ ≤ c .

Revised: Related Work: A concurrent work [33] analyzes DP-SGD with clipping and an operation similar to normalization as we have defined in this work (we elaborate on the difference in Appendix B) in the centralized setting for smooth non-convex objectives.

Revised: In comparison, our theoretical results are for convex objectives.

Original: Related work has been deferred to Appendix B due to lack of space.
Revised: We defer the rest of the related works to Appendix B due to lack of space.

Original: 3. DP-NormFedAvg : Differentially Private FL with Client-Update Normalization
Revised: 3. DP-NormFedAvg : Differentially Private FL with Client-Update Normalization

Original: In Algorithm 1, we jointly state DP-FedAvg with client-update clipping , which is the standard algorithm for client-level private FL, and our proposed algorithm DP-NormFedAvg , which is
Revised: In Algorithm 1, we jointly state DP-FedAvg with client-update clipping , which is the standard algorithm for client-level private FL, and our proposed algorithm DP-NormFedAvg , which is

Original: DP-FedAvg with client-update normalization (instead of clipping) 2 .
Revised: DP-FedAvg with client-update normalization (instead of clipping) 2 .

Original: Note that both these algorithms only differ in line 9.
Revised: Note that both these algorithms only differ in line 9.

Original: Also, we call the parameter C in Algorithm 1 “clipping threshold” for
Revised: Also, we call the parameter C in Algorithm 1 “clipping threshold” for

Original: DP-FedAvg with clipping, and “scaling factor” for DP-NormFedAvg .
Revised: DP-FedAvg with clipping, and “scaling factor” for DP-NormFedAvg .

Original: In contrast to (non-private) FedAvg (presented for completeness in Appendix A), each client in the selected subset of clients in Alg.
Revised: In contrast to (non-private) FedAvg (presented for completeness in Appendix A), each client in the selected subset of clients in Alg.

Original: 1 sends its clipped or normalized update plus zero-mean
Revised: 1 sends its clipped or normalized update plus zero-mean

Original: Gaussian noise (for differential privacy) to the server.
Revised: Gaussian noise (for differential privacy) to the server.

Original: The server then computes the mean of the noisy clipped or normalized client updates that it received (i.e., a k ) and uses it to update the global model similar to FedAvg , except with a potentially different global learning rate ( β k ) than the local learning rate ( η k ).
Revised: The server then computes the mean of the noisy clipped or normalized client updates that it received (i.e., a k ) and uses it to update the global model similar to FedAvg , except with a potentially different global learning rate ( β k ) than the local learning rate ( η k ).

Original: Based on Thm.
Revised: Based on Thm.

Original: 1 in [1], we now specify the value of σ 2 required to make Alg.(with either clipping or normalization) ( ε, δ ) -DP.
Revised: 1 in [1], we now specify the value of σ 2 required to make Alg.(with either clipping or normalization) ( ε, δ ) -DP.

Original: Theorem 3 ([1])
Revised: Theorem 3 ([1])

Original: For any 0 < ε <
Revised: For any 0 < ε <

Original: O (cid:16) r 

Revised: O (cid:16) r

Original: Kn 2 (cid:17) , Algorithm 1 is ( ε, δ ) -DP for σ 2 = qKC 2 (cid:0) log(1 /δ )
Revised: Kn 2 (cid:17) , Algorithm 1 is ( ε, δ ) -DP for σ 2 = qKC 2 (cid:0) log(

Revised: [Equation]

Original:  n 2 ε 2 (cid:1) , where q > 0 is an absolute constant.
Revised: /δ ) n 2 ε 2 (cid:1) , where q > 0 is an absolute constant.

Original: The DP-SGD algorithm of [1] returns the last iterate (i.e., w K ) as the output, and Theorem 1 in [1] guarantees that the last iterate is ( ε, δ ) -DP by setting σ 2 as above.
Revised: The DP-SGD algorithm of [1] returns the last iterate (i.e., w K ) as the output, and Theorem 1 in [1] guarantees that the last iterate is ( ε, δ ) -DP by setting σ 2 as above.

Original: But if the last iterate is ( ε, δ ) -DP, then so is any other iterate (due to additivity of the privacy cost), from which Theorem 3 follows.Algorithm
Revised: But if the last iterate is ( ε, δ ) -DP, then so is any other iterate (due to additivity of the privacy cost), from which Theorem 3 follows.

Original: : Input 2: for k = 0 , . . . , K − 1 do 3: Server sends w  a random set S clients sampling  client 
Revised: Server sends w k to a random set S k of clients, formed by sampling each client ∈ [ n ] with probability r/n .

Original: 4: for client
Revised: 4: for client

Original: i ∈
Revised: i ∈

Original: S k do 5:
Revised: S k do 5:

Original: Set w  i  w 
Revised: Set w ( i ) k, 0 = w k .

Original: 6: for τ = 0 , . . .
Revised: 6: for τ = 0 , . . .

Original: , E − 1 do 7:
Revised: , E − 1 do 7:

Original: Update w ( i ) k,τ +1 ←− w ( i ) k,τ − η k ∇ f i ( w ( i ) k,τ ) .
Revised: Update w ( i ) k,τ +1 ←− w ( i ) k,τ − η k ∇ f i ( w ( i ) k,τ ) .

Original: 8: end for 9:
Revised: 8: end for 9:

Original: Let u ( i ) k
Revised: Let u ( i ) k

Original: = ( w k − w ( i ) k,E ) η k . //
Revised: = ( w k − w ( i ) k,E ) η k . //

Original: u ( i ) k is client
Revised: u ( i ) k is client

Original: i ’s update.
Revised: i ’s update.

Original: Option (i): g ( i ) k
Revised: Option (i): g ( i ) k

Original: = clip (cid:0) u ( i ) k , C (cid:1) =
Revised: = clip (cid:0) u ( i ) k , C (cid:1) =

Original: u ( i ) k min (cid:16) 1 ,
Revised: u ( i ) k min (cid:16) 1 ,

Original: C ∥
Revised: C ∥

Original: u ( i ) k ∥ (cid:17) .
Revised: u ( i ) k ∥ (cid:17) .

Original: // Clipping
Revised: // Clipping

Original: Option (ii): g ( i ) k
Revised: Option (ii): g ( i ) k

Original: = norm (cid:0) u ( i ) k , C (cid:1) =
Revised: = norm (cid:0) u ( i ) k , C (cid:1) =

Original: C u ( i ) k
Revised: C u ( i ) k

Original: ∥ u ( i ) k ∥ .
Revised: ∥ u ( i ) k ∥ .

Original: // Normalization 10:
Revised: // Normalization 10:

Original: Send ( g ( i ) k
Revised: Send ( g ( i ) k

Original: + ζ ( i ) k ) to the server, where ζ ( i ) k ∼ N ( 0 d , rσ 2 I d ) .
Revised: + ζ ( i ) k ) to the server, where ζ ( i ) k ∼ N ( 0 d , rσ 2 I d ) .

Original: 11: end for 12:
Revised: 11: end for 12:

Original: Update w k +

Revised: Update w k +

Original: ←−
Revised: ←−

Original: w k − β k a k , where a k = 1 r (cid:80) i ∈S k ( g ( i ) k
Revised: w k − β k a k , where a k = 1 r (cid:80) i ∈S k ( g ( i ) k

Original: + ζ ( i ) k ) .
Revised: + ζ ( i ) k ) .

Original: 13: end for 14: Return w priv = w ˜ k , where ˜ k ∼ unif [0 , K − 1] .
Revised: 13: end for 14: Return w priv = w ˜ k , where ˜ k ∼ unif [0 , K − 1] .

Original: Now, let us talk about why normalization might be a better choice than clipping.
Revised: Now, let us talk about why normalization might be a better choice than clipping.

Original: 3.1. Intuition of why Normalization may outperform Clipping
Revised: 3.1. Intuition of why Normalization may outperform Clipping

Original: Intuitively, clipping has the following issue with respect to optimization: as the client update norms decrease and fall below the clipping threshold, the norm of the added noise (which has constant expectation proportional to the clipping threshold, regardless of the client update norms) can become arbitrarily larger than the client update norms, which should hamper convergence.
Revised: Intuitively, clipping has the following issue with respect to optimization: as the client update norms decrease and fall below the clipping threshold, the norm of the added noise (which has constant expectation proportional to the clipping threshold, regardless of the client update norms) can become arbitrarily larger than the client update norms, which should hamper convergence.

Original: This issue is not as grave in DP-NormFedAvg because its update- normalization step ensures that the noise norm cannot become arbitrarily larger than the normalized update’s norm, even if the original update’s norm is small.
Revised: This issue is not as grave in DP-NormFedAvg because its update- normalization step ensures that the noise norm cannot become arbitrarily larger than the normalized update’s norm, even if the original update’s norm is small.

Original: In other words, the signal (i.e., update norm) to noise ratio (SNR) of clipping eventually falls below that of normalization , due to which we expect normalization to have better convergence.
Revised: In other words, the signal (i.e., update norm) to noise ratio (SNR) of clipping eventually falls below that of normalization , due to which we expect normalization to have better convergence.

Original: Having said all this, it is worth noting that if the client update norms are lower bounded by C low , then clipping with threshold C ≤ C low is equivalent to normalization with the same scaling factor.
Revised: Having said all this, it is worth noting that if the client update norms are lower bounded by C low , then clipping with threshold C ≤ C low is equivalent to normalization with the same scaling factor.

Original: 4. Convergence Results of DP-FedAvg with Clipping and DP-NormFedAvg
Revised: 4. Convergence Results of DP-FedAvg with Clipping and DP-NormFedAvg

Original: We now present convergence results for DP-FedAvg with clipping and DP-NormFedAvg for the same choices of hyper-parameters in Alg. 1.
Revised: We now present convergence results for DP-FedAvg with clipping and DP-NormFedAvg for the same choices of hyper-parameters in Alg. 1.

Original: The detailed results and their proofs for DP-FedAvg with clipping and DP-NormFedAvg can be found in Appendices C and D.
Revised: The detailed results and their proofs for DP-FedAvg with clipping and DP-NormFedAvg can be found in Appendices C and D.

Original: These results are for the non-vacuous privacy regime (i.e., when ε is finite and δ < 1 ), where ρ = √ qd log(1 /δ ) / nε > 0 .
Revised: These results are for the non-vacuous privacy regime (i.e., when ε is finite and δ < 1 ), where ρ = √ qd log(1 /δ ) / nε > 0 .

Original: Theorem 4 (Convex) Suppose each f i is convex, L -smooth 3 and G -Lipschitz 4 over R d .
Revised: Theorem 4 (Convex) Suppose each f i is convex, L -smooth 3 and G -Lipschitz 4 over R d .

Original: Pick some γ > 0 and α ≥ 1 .
Revised: Pick some γ > 0 and α ≥ 1 .

Original: In Algorithm 1, choose E ≤ α 2 ρ and set β k = η k = ρ 2 αL , C = GE and K = (cid:0) 2 αγC (cid:1) 1 ρ 2 .
Revised: In Algorithm 1, choose E ≤ α 2 ρ and set β k = η k = ρ 2 αL , C = GE and K = (cid:0) 2 αγC (cid:1) 1 ρ 2 .

Original: Sample ˜
Revised: Sample ˜

Original: k ∼ unif [0 , K − 1] .
Revised: k ∼ unif [0 , K − 1] .

Original: Then for any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min w ′ ∈
Revised: Then for any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) , (a) Algorithm 1 Option (i), i.e. DP-FedAvg with clipping, has the following guarantee:(cid:16) 2 − ρE
Revised: d f i ( w ′ ) , (a) Algorithm 1 Option (i), i. e. DP-FedAvg with clipping, has the following guarantee:

Original: [Equation]
Revised: [Equation]

Original: (cid:124)

Original: [Equation]
Revised: [Equation]

Original: b ) Algorithm 1 Option (ii), i.e. DP-NormFedAvg , has the following guarantee:
Revised: (b) Algorithm 1 Option (ii), i.e. DP-NormFedAvg , has the following guarantee:

Original: [Equation]
Revised: [Equation]

Revised: := B 2 (effect of heterogeneity ) .

Original: In the results above, we remind the reader that u ( i ) ˜
Revised: In the results above, we remind the reader that u ( i ) ˜

Original: k is the i th client’s update at a random round number ˜ k
Revised: k is the i th client’s update at a random round number ˜ k

Original: (see line 9 in Algorithm 1).
Revised: (see line 9 in Algorithm 1).

Original: Effect of initialization and heterogeneity: The convergence results above depend on two things: (i) the distance of the initialization w 0 from the optimum w ∗ , appearing in term A in both eq.
Revised: Effect of initialization and heterogeneity: The convergence results above depend on two things: (i) the distance of the initialization w 0 from the optimum w ∗ , appearing in term A in both eq.

Original: (2) and eq.
Revised: (2) and eq.

Original: (3), and (ii) the degree of heterogeneity which depends on the ∆ ∗ i ’s (as per Definition 1), appearing in terms B 1 and B 2 in eq.
Revised: (3), and (ii) the degree of heterogeneity which depends on the ∆ ∗ i ’s (as per Definition 1), appearing in terms B 1 and B 2 in eq.

Original: (2) and eq.
Revised: (2) and eq.

Original: (3), respectively.
Revised: (3), respectively.

Original: A high (respectively, low) degree of heterogeneity implies high (respectively, low) values of ∆ ∗ i ’s, which leads to worse (respectively, better) convergence.
Revised: A high (respectively, low) degree of heterogeneity implies high (respectively, low) values of ∆ ∗ i ’s, which leads to worse (respectively, better) convergence.

Original: Also, as we increase α in Theorem 4, i.e., increase the number of rounds K , the effect of heterogeneity dies down for both clipping and normalization.
Revised: Also, as we increase α in Theorem 4, i.e., increase the number of rounds K , the effect of heterogeneity dies down for both clipping and normalization.

Original: However, the effect of the initialization term (A) cannot be diminished by increasing α .
Revised: However, the effect of the initialization term (A) cannot be diminished by increasing α .

Original: 4.1. Asymptotic Comparison of DP-FedAvg with Clipping and DP-NormFedAvg
Revised: 4.1. Asymptotic Comparison of DP-FedAvg with Clipping and DP-NormFedAvg

Original: Let us theoretically compare clipping and normalization when the number of rounds K → ∞ , i.e. asymptotically .
Revised: Let us theoretically compare clipping and normalization when the number of rounds K → ∞ , i.e. asymptotically .

Original: The asymptotic comparison is reasonable in practice where training happens for a very large number of rounds.
Revised: The asymptotic comparison is reasonable in practice where training happens for a very large number of rounds.

Original: In Theorem 4, recall that we set K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 , where α ≥ 1 is a parameter of our choice; thus we can make K → ∞ by choosing α → ∞ .
Revised: In Theorem 4, recall that we set K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 , where α ≥ 1 is a parameter of our choice; thus we can make K → ∞ by choosing α → ∞ .

Original: Here we keep E
Revised: Here we keep E

Original: = O (1) .
Revised: = O (1) .

Original: Then, the asymptotic convergence bound of DP-FedAvg with clipping can be written as:
Revised: Then, the asymptotic convergence bound of DP-FedAvg with clipping can be written as:

Original: [Equation]
Revised: [Equation]

Original: where ˜ k ∼ unif [0 , K − 1] .
Revised: where ˜ k ∼ unif [0 , K − 1] .

Original: In comparison, the corresponding bound of DP-NormFedAvg is:
Revised: In comparison, the corresponding bound of DP-NormFedAvg is:

Original: [Equation]
Revised: [Equation]

Original: Note that the RHS of both equations is the same; the RHS depends only on the initialization as the effect of heterogeneity is eliminated asymptotically.
Revised: Note that the RHS of both equations is the same; the RHS depends only on the initialization as the effect of heterogeneity is eliminated asymptotically.

Original: The only difference between the LHS of the two equations is due to the (cid:0) GE / ∥
Revised: The only difference between the LHS of the two equations is due to the (cid:0) GE / ∥

Original: u
Revised: u

Original: ( i ) ˜ k
Revised: ( i ) ˜ k

Original: ∥ (cid:1) terms in eq.
Revised: ∥ (cid:1) terms in eq.

Original: Note that (cid:0) GE / ∥
Revised: Note that (cid:0) GE / ∥

Original: u
Revised: u

Original: ( i ) ˜ k
Revised: ( i ) ˜ k

Original: ∥ (cid:1) ≥ 1 because f i is
Revised: ∥ (cid:1) ≥ 1 because f i is

Original: G -Lipschitz 5 .
Revised: G -Lipschitz 5 .

Original: So in situations where f i ( w ˜ k ) ≥ f
Revised: So in situations where f i ( w ˜ k ) ≥ f

Original: i ( w ∗ ) for all i ∈ [ n ] , the LHS of eq.
Revised: i ( w ∗ ) for all i ∈ [ n ] , the LHS of eq.

Original: (5) is at least as large the LHS of eq.
Revised: (5) is at least as large the LHS of eq.

Original: (4); in this case, we expect the convergence of normalization to be at least as good as that of clipping in private optimization (because both equations have the same RHS).
Revised: (4); in this case, we expect the convergence of normalization to be at least as good as that of clipping in private optimization (because both equations have the same RHS).

Original: An example of such a situation is over-parameterization where each minimizer of f is also a minimizer of all the f i ’s but not the other way around [22]; this can be expected in the homogeneous case.
Revised: An example of such a situation is over-parameterization where each minimizer of f is also a minimizer of all the f i ’s but not the other way around [22]; this can be expected in the homogeneous case.

Original: Due to lack of space we defer the empirical results to Appendix F and G.
Revised: Due to lack of space, we defer the empirical results to Appendix F and G.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We proposed DP-NormFedAvg which normalizes client updates instead of clipping them.
Revised: We proposed DP-NormFedAvg which normalizes client updates instead of clipping them.

Original: Theoretically, we argued that DP-NormFedAvg should have better asymptotic convergence than
Revised: Theoretically, we argued that DP-NormFedAvg should have better asymptotic convergence than

Original: DP-FedAvg with clipping, at least in the over-parameterized case.
Revised: DP-FedAvg with clipping, at least in the over-parameterized case.

Original: Intuitively, this happens because normalization has a higher signal (i e., update norm) to noise ratio than clipping.
Revised: Intuitively, this happens because normalization has a higher signal (i.e., update norm) to noise ratio than clipping.

Original: In this work, we did not theoretically compare normalization and clipping in general convex settings where over-parameterization may not hold as well as in the nonconvex case.
Revised: In this work, we did not theoretically compare normalization and clipping in general convex settings where over-parameterization may not hold as well as in the nonconvex case.

Original: These limitations pave the way for interesting future works.
Revised: These limitations pave the way for interesting future works.

Revised: ∥ u ( i ) ˜ k

Revised: ∥ =

Revised: ∥ (cid:80) E −

Revised: F Synthetic Experiments

Revised: G Experiments on Datasets

Original:  Appendix A. More Preliminaries
Revised: H Experimental Details Appendix A. More Preliminaries

Original: Differential Privacy (DP): Suppose we have a set of datasets D c and a query function h : D c −→ X .
Revised: Differential Privacy (DP): Suppose we have a set of datasets D c and a query function h : D c −→ X .

Original: Two datasets D , D ′ ∈
Revised: Two datasets D , D ′ ∈

Original: D c are said to be neighboring if they differ in exactly one sample, and this is denoted by |D − D ′ | = 1 .
Revised: D c are said to be neighboring if they differ in exactly one sample, and this is denoted by |D − D ′ | = 1 .

Original: A randomized mechanism M : X −→ Y is said to be ( ε, δ ) -DP, if for any two neighboring datasets D , D ′ ∈ D c and for any measurable subset of outputs R ∈ Y ,
Revised: A randomized mechanism M : X −→ Y is said to be ( ε, δ ) -DP, if for any two neighboring datasets D , D ′ ∈ D c and for any measurable subset of outputs R ∈ Y ,

Original: [Equation]
Revised: [Equation]

Original: Adding zero-mean Gaussian noise to the output of h ( . ) above is the customary approach to guarantee
Revised: Adding zero-mean Gaussian noise to the output of h ( . ) above is the customary approach to guarantee

Original: DP.
Revised: DP.

Original: This is known as the Gaussian mechanism [13] and it is also employed in private optimization [1].
Revised: This is known as the Gaussian mechanism [13] and it is also employed in private optimization [1].

Original: We formally define the Gaussian mechanism next.
Revised: We formally define the Gaussian mechanism next.

Original: Definition 5 (Gaussian mechanism [13]) Suppose X (i.e., the range of the query function h above) is R p .
Revised: Definition 5 (Gaussian mechanism [13]) Suppose X (i.e., the range of the query function h above) is R p .

Original: Let ∆ 2 :=
Revised: Let ∆ 2 :=

Original: sup D , D ′ ∈
Revised: sup D , D ′ ∈

Original: D c : |D−D ′ | =1 ∥ h ( D ) − h
Revised: D c : |D−D ′ | =1 ∥ h ( D ) − h

Original: ( D ′ )
Revised: ( D ′ )

Original: ∥ 2 .
Revised: ∥ 2 .

Original: If we set M ( h ( D ))
Revised: If we set M ( h ( D ))

Original: =
Revised: =

Original: h ( D )
Revised: h ( D )

Original: + Z , where
Revised: + Z , where

Original: Z ∼ N (cid:16) 

Revised: Z ∼ N (cid:16)

Original: p , 2log(1 .
Revised: p , 2log(1 .

Original: 25 /δ )
Revised: 25 /δ )

Original: ∆ 22 ε
Revised: ∆ 22 ε

Original: I p (cid:17) , then the mechanism M is ( ε, δ ) -DP.
Revised: I p (cid:17) , then the mechanism M is ( ε, δ ) -DP.

Original: We also state the famous FedAvg algorithm of [23] (with local updates using full gradients).
Revised: We also state the famous FedAvg algorithm of [23] (with local updates using full gradients).

Original: Algorithm
Revised: Algorithm

Original: 1: Input: Initial point w 0 , number of rounds of communication K , number of local updates per round E , local learning rates { η k } K − 1 k =0 and number of participating clients in each round r .
Revised: 1: Input: Initial point w 0 , number of rounds of communication K , number of local updates per round E , local learning rates { η k } K − 1 k =0 and number of participating clients in each round r .

Original: 2: for k = 0 , . . .
Revised: 2: for k = 0 , . . .

Original: , K − 1 do 3:
Revised: , K − 1 do 3:

Original: Server sends w k to a random set S k of r clients chosen uniformly at random.
Revised: Server sends w k to a random set S k of r clients chosen uniformly at random.

Original: 4: for client
Revised: 4: for client

Original: i ∈
Revised: i ∈

Original: S k do 5:
Revised: S k do 5:

Original: Set w ( i ) k, 0 = w k .
Revised: Set w ( i ) k, 0 = w k .

Original: 6: for τ = 0 , . . .
Revised: 6: for τ = 0 , . . .

Original: , E − 1 do 7:
Revised: , E − 1 do 7:

Original: Update w ( i ) k,τ +1 ←− w ( i ) k,τ − η k ∇ f i ( w ( i ) k,τ ) .
Revised: Update w ( i ) k,τ +1 ←− w ( i ) k,τ − η k ∇ f i ( w ( i ) k,τ ) .

Original: 8: end for 9:
Revised: 8: end for 9:

Original: Send w k − w ( i ) k,E to the server.
Revised: Send w k − w ( i ) k,E to the server.

Original: 10: end for 11:
Revised: 10: end for 11:

Original: Update w k +1 ←− w k − 1 r (cid:80) i ∈S k ( w k − w ( i ) k,E ) .
Revised: Update w k +1 ←− w k − 1 r (cid:80) i ∈S k ( w k − w ( i ) k,E ) .

Original: // ( The above is equivalent to w k +1 ←− 1 r (cid:80) i ∈S k w ( i ) k,E , so the clients might as well just send the w ( i ) k,E ’s. ) 12: end for
Revised: // ( The above is equivalent to w k +1 ←− 1 r (cid:80) i ∈S k w ( i ) k,E , so the clients might as well just send the w ( i ) k,E ’s. ) 12: end for

Original: Appendix B. Related Workand Significance
Revised: Appendix B. Related Work

Original: Differentially private optimization: Most differentially private optimization algorithms for training
Revised: Differentially private optimization: Most differentially private optimization algorithms for training

Original: ML models (both in the centralized and federated settings) are based off of DP-SGD [1], wherein the optimizer receives a Gaussian noise-perturbed average of the clipped per-sample gradients to guarantee DP.
Revised: ML models (both in the centralized and federated settings) are based off of DP-SGD [1], wherein the optimizer receives a Gaussian noise-perturbed average of the clipped per-sample gradients to guarantee DP.

Original: Similar to and/or related to DP-SGD, there are several papers on private optimization in the centralized setting [3–5, 8, 9, 11, 12, 14, 19, 20, 27, 28, 30, 31, 35] as well as in the federated and distributed (without multiple local updates) setting [2, 15, 16, 21, 25, 26, 29].
Revised: Similar to and/or related to DP-SGD, there are several papers on private optimization in the centralized setting [3–5, 8, 9, 11, 12, 14, 19, 20, 27, 28, 30, 31, 36] as well as in the federated and distributed (without multiple local updates) setting [2, 15, 16, 21, 25, 26, 29].

Original: DP-FedAvg with clipping [15, 29] (stated in Algorithm 1 Option (i)) is the most standard private algorithm in the federated setting for client-level DP.
Revised: DP-FedAvg with clipping [15, 29] (stated in Algorithm 1 Option (i)) is the most standard private algorithm in the federated setting for client-level DP.

Original: [4] show that in the convex Lipschitz centralized case, the suboptimality gap bound of O ( ρ ) (recall ρ = √ qd log(1 /δ ) / nε , as defined in Definition 2) bound is tight.
Revised: [4] show that in the convex Lipschitz centralized case, the suboptimality gap bound of O ( ρ ) (recall ρ = √ qd log(1 /δ ) / nε , as defined in Definition 2) bound is tight.

Original: Normalized gradient descent (GD) and related methods: In the centralized setting, [17] propose (Stochastic)
Revised: Normalized gradient descent (GD) and related methods: In the centralized setting, [17] propose (Stochastic)

Original: Normalized GD.
Revised: Normalized GD.

Original: This is based on a similar idea as DP-NormFedAvg – instead of using the (stochastic) gradient, use the unit vector along the (stochastic) gradient for the update.
Revised: This is based on a similar idea as DP-NormFedAvg – instead of using the (stochastic) gradient, use the unit vector along the (stochastic) gradient for the update.

Original: Extensions of this method incorporating momentum [10, 33, 34] have been shown to significantly improve the training time of very large models such as BERT in the centralized setting.
Revised: Extensions of this method incorporating momentum [10, 34, 35] have been shown to significantly improve the training time of very large models such as BERT in the centralized setting.

Original: In the FL setting, [7] propose Normalized FedAvg , where the server uses a normalized version of the average of client updates (and not the average of normalized client updates, which is what we do) to improve training.
Revised: In the FL setting, [7] propose Normalized FedAvg , where the server uses a normalized version of the average of client updates (and not the average of normalized client updates, which is what we do) to improve training.

Original: However, it must be noted here that these works perform (some kind of) normalization to accelerate non-private training, whereas we are proposing normalization as an alternative sensitivity bounding mechanism to improve private training compared to the usual mechanism of clipping. 
Revised: However, it must be noted here that these works perform (some kind of) normalization to accelerate non-private training, whereas we are proposing normalization as an alternative sensitivity bounding mechanism to improve private training compared to the usual mechanism of clipping. Normalization operation of [33]:

Revised: Given an update u , [33]’s normalized update is (cid:16) u

Revised: ∥ u ∥ + r (cid:17) for some constant r > 0 .

Revised: Compared to our normalized update, viz.

Revised: , (cid:16) C u ∥ u ∥ (cid:17) , note that [33] drop the scaling factor C that we have, and instead they have a constant r in the denominator.

Original: Appendix C. Detailed Result for DP-FedAvg with Clipping and its Proof
Revised: Appendix C. Detailed Result for DP-FedAvg with Clipping and its Proof

Original: Theorem 6 ( DP-FedAvg with Clipping) Suppose each f i is convex and L -smooth over R d .
Revised: Theorem 6 ( DP-FedAvg with Clipping) Suppose each f i is convex and L -smooth over R d .

Original: Let ˆ C :
Revised: Let ˆ C :

Original: = CE , where C is the clipping threshold used in Algorithm 1 Option (i).
Revised: = CE , where C is the clipping threshold used in Algorithm 1 Option (i).

Original: For any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min
Revised: For any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min

Original: w ′ ∈
Revised: w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) ≥ 0 , Algorithm 1 Option (i) with ˆ C ≥ 4 (cid:113) 
Revised: d f i ( w ′ ) ≥ 0 , Algorithm 1 Option (i) with ˆ C ≥ 4 (cid:113) L max j [Equation] [ n ] ∆

Revised: [Equation]

Revised: [Equation]

Original: L max j ∈ [ n ] ∆ ∗ j , β k = η k = η = (cid:0) γ ˆ CLEK (cid:1) 1 ρ and K > (cid:0) 2 γ ˆ CE (cid:1) 1 ρ , where γ > 0 is a constant of our choice, has the following convergence guarantee: 
Revised: (cid:0) γ ˆ CLEK (cid:1) 1 ρ and K > (cid:0) 2 γ ˆ CE (cid:1) 1 ρ , where γ > 0 is a constant of our choice, has the following convergence guarantee: [Equation] [Equation] (cid:17)(cid:33)(cid:35)

Original: [Equation]
Revised: [Equation]

Revised: (cid:35) , with ˜

Original: with ˜ k ∼ unif [0 , K − 1] .
Revised: k ∼ unif [0 , K − 1] .

Original: Specifically, with K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 and E ≤ α 2 ρ , where α ≥ 1 is another constant of our choice,
Revised: Specifically, with K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 and E ≤ α 2 ρ , where α ≥ 1 is another constant of our choice,

Original: Algorithm 1 Option (i) has the following convergence guarantee:
Revised: Algorithm 1 Option (i) has the following convergence guarantee:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4 (a) is obtained by further assuming that each f i is G -Lipschitz and choosing ˆ C
Revised: Theorem 4 (a) is obtained by further assuming that each f i is G -Lipschitz and choosing ˆ C

Original: =
Revised: =

Original: G above, in which case 1 ( ∥ u ( i ) ˜ k
Revised: G above, in which case 1 ( ∥ u ( i ) ˜ k

Original: ∥ ≤ ˆ CE ) = 1 .
Revised: ∥ ≤ ˆ CE ) = 1 .

Original: C.1. Proof of Theorem
Revised: C.1. Proof of Theorem

Original: Proof Let us set η k = β k
Revised: Proof Let us set η k = β k

Original: = η for all k ≥ 0 .
Revised: = η for all k ≥ 0 .

Original: The update rule of the global iterate is:
Revised: The update rule of the global iterate is:

Original: [Equation]
Revised: [Equation]

Original:  where ζ k = 1 r (cid:80) i ∈S k ζ [Equation] i ) k ∼ N ( 0 d , qK log
Revised: (7) where ζ k = 1 r (cid:80) i ∈S k ζ ( i ) k ∼ N ( 0 d , qK log(1 /δ )

Original: [Equation] 1 /δ ) C 2 n [Equation]
Revised: C 2 n 2 ε

Original: I d ) and 
Revised: I d ) and u (

Original: [Equation]
Revised: [Equation]

Original: Taking expectation with respect to the randomness in the current round [Equation] we get for any w ∗ ∈ arg min w ′ ∈
Revised: Taking expectation with respect to the randomness in the current round, we get for any w ∗ ∈ arg min w ′ ∈

Original: R
Revised: R

Original: d f ( w ′ ) :
Revised: d f ( w ′ ) :

Original: [Equation]
Revised: [Equation]

Original: i = − 2 η ⟨ clip ( u ( i ) k , C ) , w k − w ∗ ⟩ + η 2 E S k

Original: (cid:104)(cid:13)(cid:13)(cid:13) 1 r (cid:88)

Original: i ∈S k clip ( u ( i ) k , C ) (cid:13)(cid:13)(cid:13) 2 (cid:105) (11) + η 2 (cid:16)

Original: qKd log(1 /δ )

Original: Cn 2 ε(cid:17) ≤ ∥ w k − w ∗ ∥ 2 + 1 n n (cid:88) i = − 2 η ⟨ clip ( u ( i ) k , C ) , w k − w ∗ ⟩ + η 2 E S k (cid:104) 1 r (cid:88)

Original: i ∈S k (cid:13)(cid:13) clip ( u ( i ) k , C ) (cid:13)(cid:13) 2 (cid:105) (12) + η 2 (cid:16) qKd log(1 /δ )

Original: Cn 2 ε(cid:17)

Original: =

Original: ∥ w k − w ∗ ∥ 2 + 1 n n (cid:88) i =

Original: (cid:110) − 2 η ⟨ clip ( u ( i ) k , C ) , w k − w ∗ ⟩ + η 2 (cid:13)(cid:13) clip ( u ( i ) k , C ) (cid:13)(cid:13)(cid:124) (cid:123)(cid:122) (cid:125)

Original: A i (cid:111) (13) + η 2 (cid:16) qKd log(1 /δ )

Original: Cn 2 ε(cid:17) .

Original: Note that eq.
Revised: Note that eq.

Original: (12) is obtained by using Fact 2.
Revised: (12) is obtained by using Fact 2.

Original: Let us examine A i for each i .
Revised: Let us examine A i for each i .

Original: Case 1: ∥ u ( i ) k ∥ > C .
Revised: Case 1: ∥ u ( i ) k ∥ > C .

Original: So we have clip ( u ( i ) k , C ) =
Revised: So we have clip ( u ( i ) k , C ) =

Original: C ∥
Revised: C ∥

Original: u( i ) k
Revised: u

Original: ∥ u ( i ) k. Thus,
Revised: [Equation] i ) k

Original: A
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: (cid:68) u
Revised: ( i ) k . Thus,

Original: [Equation]
Revised: [Equation]

Original: where the last step follows by using the fact for any two vectors a and b , ⟨ a , b ⟩ = 12 (cid:16) ∥ a ∥ 2 +
Revised: where the last step follows by using the fact for any two vectors a and b , ⟨ a , b ⟩ = 12 (cid:16) ∥ a ∥ 2 +

Original: ∥ b ∥ 2 − ∥ a − b ∥ 2 ) .
Revised: ∥ b ∥ 2 − ∥ a − b ∥ 2 ) .

Original: Next, notice that w k − η u ( i ) k
Revised: Next, notice that w k − η u ( i ) k

Original: = w ( i ) k,E .
Revised: = w ( i ) k,E .

Original: Since f i is convex, we use Lemma 8 to get:
Revised: Since f i is convex, we use Lemma 8 to get:

Original: [Equation]
Revised: [Equation]

Original: [Equation] for η ≤ 12 L with ∆ ∗ i := f i ( w ∗ ) − min w ′ ∈
Revised: 16) for η ≤ 12 L with ∆ ∗ i : = f i ( w ∗ ) − min w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) ≥ 0 .
Revised: d f i ( w ′ ) ≥ 0 .

Original: But:
Revised: But:

Original: [Equation]
Revised: [Equation]

Original: The inequality above follows from Fact 2.
Revised: The inequality above follows from Fact 2.

Original: Using this in eq.
Revised: Using this in eq.

Original: (16), we get:
Revised: (16), we get:

Original: [Equation]
Revised: [Equation]

Original: Plugging this back in eq.
Revised: Plugging this back in eq.

Original: (15), we get:
Revised: (15), we get:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for η ≤ 12 L .
Revised: for η ≤ 12 L .

Original: Let us choose C 2 ≥ 16 LE 2 max j ∈ [ n ] ∆ ∗ j .
Revised: Let us choose C 2 ≥ 16 LE 2 max j ∈ [ n ] ∆ ∗ j .

Original: Then, we have E ∆ ∗ i ≤ C 2 16 LE ≤ C ∥
Revised: Then, we have E ∆ ∗ i ≤ C 2 16 LE ≤ C ∥

Original: u
Revised: u

Original: ( i ) k ∥ 16 LE .
Revised: ( i ) k ∥ 16 LE .

Original: Using this in eq.
Revised: Using this in eq.

Original: (19), we get:
Revised: (19), we get:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for C ≥ 4 E (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .
Revised: for C ≥ 4 E (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .

Original: Case 2: ∥ u ( i ) k
Revised: Case 2: ∥ u ( i ) k

Original: ∥ ≤ C .
Revised: ∥ ≤ C .

Original: So we have clip ( u ( i ) k , C ) =
Revised: So we have clip ( u ( i ) k , C ) =

Original: u ( i ) k .
Revised: u ( i ) k .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: for ηL ≤ 1 ; the inequality ∥ u ( i ) k
Revised: for ηL ≤ 1 ; the inequality ∥ u ( i ) k

Original: ∥ 2 ≤ 2 LE 2 ( f i ( w k

Original: [Equation] − f [Equation] (for ηL ≤ 1 ) is obtained from Lemma 9.
Revised: ∥ 2 ≤ 2 LE 2 ( f i ( w k ) − f ∗ i ) (for ηL ≤ 1 ) is obtained from Lemma 9.

Original: Now:
Revised: Now:

Original: [Equation]
Revised: [Equation]

Original: Note that eq.
Revised: Note that eq.

Original: (27) follows from the convexity of f i , while eq.
Revised: (27) follows from the convexity of f i , while eq.

Original: (28) follows by once again using the convexity of f i , the smoothness of f i as well as the Cauchy-Schwarz inequality.
Revised: (28) follows by once again using the convexity of f i , the smoothness of f i as well as the Cauchy-Schwarz inequality.

Original: Again, from Lemma 9, we have
Revised: Again, from Lemma 9, we have

Original: [Equation]
Revised: [Equation]

Original: 30) for ηL ≤ 1 .
Revised: for ηL ≤ 1 .

Original: Using eq.
Revised: Using eq.

Original: (30) in eq.
Revised: (30) in eq.

Original: (29), we get
Revised: (29), we get

Original: [Equation]

Original: ≥

Original: [Equation]
Revised: [Equation]

Original: Now using eq.
Revised: Now using eq.

Original: (31) in eq.
Revised: (31) in eq.

Original: (23), we get
Revised: (23), we get

Original: [Equation]
Revised: [Equation]

Original: for η ≤ 1 L .
Revised: for η ≤ 1 L .

Original: Combining the results of Case 1 and 2, i.e. eq.
Revised: Combining the results of Case 1 and 2, i.e. eq.

Original: (22) and eq.
Revised: (22) and eq.

Original: (32), we get
Revised: (32), we get

Original: [Equation]
Revised: [Equation]

Original: for C ≥ 4 E (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .
Revised: for C ≥ 4 E (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .

Original: Let us define ˆ C :=
Revised: Let us define ˆ C :=

Original: CE .
Revised: CE .

Original: Then eq.
Revised: Then eq.

Original: (33) can be re-written as:
Revised: (33) can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: where ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .
Revised: where ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j and η ≤ 12 L .

Original: Now using eq.
Revised: Now using eq.

Original: (34) in eq.
Revised: (34) in eq.

Original: [Equation]
Revised: [Equation]

Original: we get:
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: Solving the above recursion after taking expectation throughout and some rearranging, we get:
Revised: Solving the above recursion after taking expectation throughout and some rearranging, we get:

Original: [Equation]
Revised: [Equation]

Original: Let us choose η = γ ˆ CLEK nε √ qd log(1 /δ )
Revised: Let us choose η = [Equation] log(1 /δ )

Original: for some constant γ > 
Revised: for some constant γ > 0 .

Revised: Note that we must have K >

Original: [Equation]
Revised: [Equation]

Original: Note that we must have K > 2 γ ˆ CE nε √ qd log(1 /δ ) for our condition of ηL ≤ 12 to be satisfied [Equation]
Revised: CE nε √ qd log(1 /δ ) for our condition of ηL ≤ 12 to be satisfied.

Original: With that [Equation] we get:
Revised: With that, we get:

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:33)(cid:35) , (37)
Revised: with ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j .

Original: The above equation is equivalent to:
Revised: The above equation is equivalent to:

Original: [Equation]
Revised: [Equation]

Original:  where ˜ k ∼ unif [
Revised: 38) where ˜ k ∼ unif [

Original: [Equation]
Revised: [Equation]

Original: K − 1].
Revised: K − 1]

Revised: [Equation]

Original: Let us set K
Revised: Let us set K

Original: = 2 αγ ˆ CE (cid:16) 
Revised: = 2 αγ ˆ CE (cid:16) nε

Original: [Equation]
Revised: [Equation]

Original: √
Revised: qd log(1 /δ

Original: [Equation]
Revised: [Equation]

Original: (cid:17) 2 in eq.
Revised: (cid:17) 2 in eq.

Original: (38), where α ≥ 1 is a constant of our choice and E ≤ α 2 (cid:16)
Revised: [Equation] where α ≥ 1 is a constant of our choice and E ≤ α 2 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: √
Revised: That gives us: E

Original: [Equation]
Revised: [Equation]

Original: (cid:17) . That gives us:
Revised: The final result follows by substituting ρ = √

Original: [Equation]
Revised: [Equation]

Original: αnε
Revised: log

Original: [Equation]
Revised: [Equation]

Original: The final result follows by substituting ρ =
Revised: /δ

Original: [Equation]
Revised: [Equation]

Original:  Appendix D. Detailed Result for DP-NormFedAvg and its Proof 
Revised: nε . Appendix D. Detailed Result for DP-NormFedAvg and its Proof Theorem

Original: Theorem 7 ( DP-NormFedAvg ) Suppose each f i is convex and L -smooth over R d .
Revised: 7 ( DP-NormFedAvg ) Suppose each f i is convex and L -smooth over R d .

Original: Let ˆ C :
Revised: Let ˆ C :

Original: = C/E , where C is the scaling factor used in Algorithm 1 Option (ii).
Revised: = C/E , where C is the scaling factor used in Algorithm 1 Option (ii).

Original: For any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min
Revised: For any w ∗ ∈ arg min w ′ ∈ R d f ( w ′ ) and ∆ ∗ i := f i ( w ∗ ) − min

Original: w ′ ∈
Revised: w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) ≥ 0 , Algorithm 1 Option (ii) with ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j , β k =
Revised: d f i ( w ′ ) ≥ 0 , Algorithm 1 Option (ii) with ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j , β k =

Original: η k = η = (cid:0) γ ˆ CLEK (cid:1) 1 ρ and K > (cid:0) 2 γ ˆ CE (cid:1) 1 ρ , where γ > 0 is a constant of our choice, has the following convergence guarantee:
Revised: η k = η = (cid:0) γ ˆ CLEK (cid:1) 1 ρ and K > (cid:0) 2 γ ˆ CE (cid:1) 1 ρ , where γ > 0 is a constant of our choice, has the following convergence guarantee:

Original: [Equation]
Revised: [Equation]

Revised: 8 LE (cid:17)(cid:41)(cid:35)

Revised: [Equation]

Original:  with ˜ k ∼ unif [0 , K − 1] .
Revised: , with ˜ k ∼ unif [0 , K − 1] .

Original: Further, this result holds for any w ∗ ∈ arg min
Revised: Further, this result holds for any w ∗ ∈ arg min

Original: w ′ ∈ R d f ( w ′ ) .
Revised: w ′ ∈ R d f ( w ′ ) .

Original: Specifically, with K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 and E ≤ α 2 ρ , where α ≥ 1 is another constant of our choice,
Revised: Specifically, with K = (cid:0) 2 αγ ˆ CE (cid:1) 1 ρ 2 and E ≤ α 2 ρ , where α ≥ 1 is another constant of our choice,

Original: Algorithm 1 Option (ii) has the following convergence guarantee:
Revised: Algorithm 1 Option (ii) has the following convergence guarantee:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4 (b) is obtained by further assuming that each f i is G -Lipschitz and choosing ˆ C
Revised: Theorem 4 (b) is obtained by further assuming that each f i is G -Lipschitz and choosing ˆ C

Original: =
Revised: =

Original: G above, in which case 1 ( ∥ u ( i ) ˜ k
Revised: G above [Equation] in which case 1 ( ∥ u ( i ) ˜ k

Original: ∥ ≤ ˆ CE ) = 1 .
Revised: ∥ ≤ ˆ CE ) = 1 .

Original: D.1. Proof of Theorem
Revised: D.1. Proof of Theorem

Original: Proof Let us again set η k = β k
Revised: Proof Let us again set η k = β k

Original: = η , for all k ≥ 0 .
Revised: = η , for all k ≥ 0 .

Original: Everything remains the same till eq.
Revised: Everything remains the same till eq.

Original: (13) in the proof of Theorem 6, with clip ( . ) replaced by norm ( . ) .
Revised: (13) in the proof of Theorem 6, with clip ( . ) replaced by norm ( . ) .

Original: [Equation]
Revised: [Equation]

Original: Again, let us examine A i for each i .
Revised: Again, let us examine A i for each i .

Original: Also, as used in the proof of Theorem 6, let ˆ C = CE .
Revised: Also, as used in the proof of Theorem 6, let ˆ C = CE .

Original: Case 1: ∥ u ( i ) k ∥ >
Revised: Case 1: ∥ u ( i ) k ∥ >

Original: ˆ CE .
Revised: ˆ CE .

Original: Everything remains the same as Case 1 in the proof of Theorem 6.
Revised: Everything remains the same as Case 1 in the proof of Theorem 6.

Original: Thus,
Revised: Thus,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for ηL ≤
Revised: for ηL ≤

Original: 12 and ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j .
Revised: 12 and ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j .

Original: Case 2: ∥ u ( i ) k
Revised: Case 2: ∥ u ( i ) k

Original: ∥ ≤ ˆ CE .
Revised: ∥ ≤ ˆ CE .

Original: Here:
Revised: Here:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For ease of notation henceforth, let us define:
Revised: For ease of notation henceforth, let us define:

Original: [Equation]
Revised: [Equation]

Original: The bound for B i remains the same as the one in the proof of Theorem 6 (in eq.
Revised: The bound for B i remains the same as the one in the proof of Theorem 6 (in eq.

Original: (31)), i.e.,
Revised: (31)), i.e.,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for ηL ≤ 1 .
Revised: for ηL ≤ 1 .

Original: Using this in eq.
Revised: Using this in eq.

Original: (42), we get:
Revised: (42), we get:

Original: [Equation]
Revised: [Equation]

Original: Combining the results of Case 1 and 2, i.e. eq.
Revised: Combining the results of Case 1 and 2, i.e. eq.

Original: (41) and eq.
Revised: (41) and eq.

Original: (45), we get:
Revised: (45), we get:

Revised: A i ≤ ηE

Original: [Equation]
Revised: [Equation]

Original: for ηL ≤ 1 2 and ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j 
Revised: for ηL ≤ 12 and ˆ C ≥ 4 (cid:113) L max j ∈ [ n ] ∆ ∗ j .

Original: [Equation] Now using the above bound in eq. 
Revised: Now using the above bound in eq. [Equation]

Original: (40), plugging in z ( i ) k
Revised: plugging in z ( i ) k

Original: = ˆ CE ∥ u ( i ) k ∥ , and following the same process and choice of η = γ ˆ CLEKnε √ qd log(1 /δ )
Revised: = ˆ CE ∥ u ( i ) k ∥ [Equation] and following the same process and choice of η = γ ˆ CLEK

Original:  that we used in Theorem 6, we get:
Revised: [Equation] 1 /δ ) that we used in Theorem [Equation] we get:

Original: [Equation]
Revised: [Equation]

Revised: (cid:33)(cid:41)(cid:35) ,

Original:  with ˜ k ∼ unif [0 , K − 1] and K > 2 γ ˆ CE nε √ qd log(1 /δ )
Revised: [Equation] 47) with ˜ k ∼ unif [0 , K − 1] and K > 2 γ ˆ CE nε √ qd log(1 /δ )

Original: (so that ηLE ≤ 12 ).
Revised: (so that ηLE ≤ 12 ).

Original: Now setting K = 2 αγ ˆ CE (cid:16) nε √ qd log(1 /δ )
Revised: Now setting K = [Equation] nε [Equation]

Original: (cid:17) 2 and ρ = √ qd log(1 /δ )
Revised: and ρ [Equation] √ [Equation]

Original:  nε above, and using the fact that 1 (cid:0) ∥ u ( i ) ˜
Revised: δ [Equation] nε above, and using the fact that 1 (cid:0) ∥ u ( i ) ˜

Original: k
Revised: k

Original: ∥ ≤ ˆ CE (cid:1) ≤ 1 gives us the final result.
Revised: ∥ ≤ ˆ CE (cid:1) ≤ 1 gives us the final result.

Original: Appendix E. Lemmas used in the Proofs
Revised: Appendix E. Lemmas used in the Proofs

Original: Lemma 8 Suppose f i is convex and L -smooth over R d .
Revised: Lemma 8 Suppose f i is convex and L -smooth over R d .

Original: Let us set η k ≤ 12 L in round k of Algorithm 1.
Revised: Let us set η k ≤ 12 L in round k of Algorithm 1.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: where ∆ ∗ i :
Revised: where ∆ ∗ i :

Original: = f i ( w ∗ ) − min w ′ ∈
Revised: = f i ( w ∗ ) − min w ′ ∈

Original: R
Revised: R

Original: d f i ( w ′ ) .
Revised: d f i ( w ′ ) .

Original: Proof Let us define f ∗ i :=
Revised: Proof Let us define f ∗ i :=

Original: min w ′ ∈ R d f i ( w ′ ) .
Revised: min w ′ ∈ R d f i ( w ′ ) .

Original: Then, ∆ ∗ i = f i ( w ∗ ) − f ∗ i .
Revised: Then, ∆ ∗ i = f i ( w ∗ ) − f ∗ i .

Original: For any τ ≥ 0 , we have:
Revised: For any τ ≥ [Equation] we have:

Original: [Equation]
Revised: [Equation]

Original: Equation (48) follows by using the fact that each f i is convex.
Revised: Equation (48) follows by using the fact that each f i is convex.

Original: Equation (50) follows using Fact 1.
Revised: Equation (50) follows using Fact 1.

Original: Now if we set η k ≤ 12 L [Equation] then we get:
Revised: Now if we set η k ≤ 12 L , then we get:

Original: [Equation]
Revised: [Equation]

Original: Doing this recursively for τ = 0 through to τ =
Revised: Doing this recursively for τ = 0 through to τ =

Original: E − 1 and adding everything up gives us the desired result.
Revised: E − 1 and adding everything up gives us the desired result.

Original: Lemma 9 Suppose each f i is L -smooth over R d and f ∗ i :=
Revised: Lemma 9 Suppose each f i is L -smooth over R d and f ∗ i :=

Original: min w ′ ∈ R d f i ( w ′ ) .
Revised: min w ′ ∈ R d f i ( w ′ ) .

Original: Let us set η k ≤ 1 L in round k of Algorithm 1.
Revised: Let us set η k ≤ 1 L in round k of Algorithm 1.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from Fact 2.
Revised: where the last step follows from Fact 2.

Original: Next

Original: [Equation] since f i is L -smooth [Equation] we have using Fact 1:
Revised: Next, since f i is L -smooth, we have using Fact 1:

Original: [Equation]
Revised: [Equation]

Original: Applying this in eq.
Revised: Applying this in eq.

Original: [Equation] we get:
Revised: (52), we get:

Original: [Equation]
Revised: [Equation]

Original: But using the L -smoothness of f i 
Revised: But using the L -smoothness of f i , we have for any t ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: we have for any t ≥ 1 :
Revised: i

Original: [Equation]
Revised: [Equation]

Original: for η k L ≤ 1 .
Revised: for η k L ≤ 1 .

Original: Doing this recursively (and recalling that w ( i ) k, 0 = w k ), we get:
Revised: Doing this recursively (and recalling that w ( i ) k, 0 = w k ), we get:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging this in eq.
Revised: Plugging this in eq.

Original: (53), we get:
Revised: (53), we get:

Original: [Equation]
Revised: [Equation]

Original: The upper bound on (cid:13)(cid:13) u ( i ) k
Revised: The upper bound on (cid:13)(cid:13) u ( i ) k

Original: (cid:13)(cid:13) 2 follows by recalling that u ( i )
Revised: (cid:13)(cid:13) 2 follows by recalling that u ( i )

Original: k
Revised: k

Original: = ( w k − w ( i ) k,E ) /η k .
Revised: = ( w k − w ( i ) k,E ) /η k .

Original: Lemma 10 Suppose each f i is L -smooth over R d .
Revised: Lemma 10 Suppose each f i is L -smooth over R d .

Original: Then in Algorithm 1, we have:
Revised: Then in Algorithm 1, we have:

Original: [Equation]
Revised: [Equation]

Original: for any i ∈ [ n ] , k ∈ { 0 , . . .
Revised: for any i ∈ [ n ] , k ∈ { 0 , . . .

Original: , K − 1 } and τ ∈ { 0 , . . .
Revised: , K − 1 } and τ ∈ { 0 , . . .

Original: , E − 1 } .
Revised: , E − 1 } .

Original: Proof Since each f i is L -smooth, we have by using the co-coercivity of the gradient:
Revised: Proof Since each f i is L -smooth [Equation] we have by using the co-coercivity of the gradient:

Original: [Equation]
Revised: [Equation]

Original: Now using the fact that w ( i ) k,τ +1 − w ( i ) k,τ = − η k ∇ f i ( w ( i ) k,τ ) above, we get:
Revised: Now using the fact that w ( i ) k [Equation] τ +1 − w ( i ) k,τ = − η k ∇ f i ( w ( i ) k,τ ) above

Original: [Equation]
Revised: [Equation]

Original: ⟨∇
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above a bit, we get:
Revised: Rearranging the above a bit [Equation] we get:

Original: [Equation]
Revised: [Equation]

Original: But, we also have: ⟨∇
Revised: But, we also have: ⟨∇

Original: [Equation]
Revised: [Equation]

Original: Using this in eq.
Revised: Using this in eq.

Original: (61) and simplifying a bit, we get:
Revised: (61) and simplifying a bit, we get:

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: Fact 1 ([24]) For an L -smooth function h : R d −→ R with h ∗ = min x ∈ R d h ( x ) and L > 0 , ∥∇ h ( x ) ∥ 2 ≤ 2 L ( h ( x ) − h ∗ ) .
Revised: Fact 1 ([24]) For an L -smooth function h : R d −→ R with h ∗ = min x ∈ R d h ( x ) and L > 0 , ∥∇ h ( x ) ∥ 2 ≤ 2 L ( h ( x ) − h ∗ ) .

Original: Fact 2 For any p > 1 vectors { y 1 , . . .
Revised: Fact 2 For any p > 1 vectors { y 1 , . . .

Original: , y[-p } , ∥ (cid:80) pi =
-]
Revised: , y[-p } , ∥ (cid:80) pi =
-]

Original: y i ∥ 2 ≤ p (cid:80) pi = 1 ∥
Revised: [Equation]

Original: y i ∥ 2 .
Revised: [Equation]

Original: Fact 2 follows from Jensen’s inequality.
Revised: Fact 2 follows from Jensen’s inequality.

Original: Appendix F. Synthetic Experiments
Revised: Appendix F. Synthetic Experiments

Original: Here we compare DP-FedAvg with clipping and DP-NormFedAvg on a simple but illustrative quadratic problem.
Revised: Here we compare DP-FedAvg with clipping and DP-NormFedAvg on a simple but illustrative quadratic problem.

Original: We consider f i ( w ) = 12 ( w − w ∗
Revised: We consider f i ( w ) = 12 ( w − w ∗

Original: i ) T Q
Revised: i ) T Q

Original: i ( w − w ∗ i ) , where i ∈ [100] (so, n = 100 ) and w ∈ R 200 (so, d = 200 ).
Revised: i ( w − w ∗ i ) , where i ∈ [100] (so, n = 100 ) and w ∈ R 200 (so, d = 200 ).

Original: Further, w ∗ i is drawn i.i.d.
Revised: Further, w ∗ i is drawn i.i.d.

Original: from N ( 0 200 , I 200 ) and Q i =
Revised: from N ( 0 200 , I 200 ) and Q i =

Original: A i
Revised: A i

Original: A Ti , where A i is a 200 × 20 matrix whose entries are drawn i.i.d from N (0 , 120 2 ) ; hence, Q i is a PSD matrix with bounded maximum eigenvalue, due to which f
Revised: A Ti , where A i is a 200 × 20 matrix whose entries are drawn i.i.d from N (0 , 120 2 ) ; hence, Q i is a PSD matrix with bounded maximum eigenvalue, due to which f

Original: i is convex and smooth.
Revised: i is convex and smooth.

Original: We set ( ε, δ ) =
Revised: We set ( ε, δ ) =

Original: (5 , 10 − 6 ) , K = 500 and E = 20 for this set of experiments.
Revised: (5 , 10 − 6 ) , K = 500 and E = 20 for this set of experiments.

Original: We consider two different initializations with different distances from the global optimum w ∗ – (i) I1 :
Revised: We consider two different initializations with different distances from the global optimum w ∗ – (i) I1 :

Original: w 0 =
Revised: w 0 =

Original: w ∗ + z , and (ii) I2 :
Revised: w ∗ + z , and (ii) I2 :

Original: w 0 =
Revised: w 0 =

Original: w ∗ + z 5 , where each coordinate of z is drawn i.i.d.
Revised: w ∗ + z 5 , where each coordinate of z is drawn i.i.d.

Original: from the continuous uniform distribution with support (0,1).
Revised: from the continuous uniform distribution with support (0,1).

Original: We set η k = β k
Revised: We set η k = β k

Original: = η for all rounds k , and also have full-device participation.
Revised: = η for all rounds k , and also have full-device participation.

Original: In Figure 2, we plot the function suboptimality (i.e., f ( w k ) − f ( w ∗ ) at round number k ) of DP-FedAvg with Clipping and DP-NormFedAvg for different values of η and clipping threshold/scaling factor C , for I1 and I2.
Revised: In Figure 2, we plot the function suboptimality (i.e., f ( w k ) − f ( w ∗ ) at round number k ) of DP-FedAvg with Clipping and DP-NormFedAvg for different values of η and clipping threshold/scaling factor C , for I1 and I2.

Original: In Figure 3, for each round k , we plot the corresponding
Revised: In Figure 3, for each round k , we plot the corresponding

Original: SNR :=
Revised: SNR :=

Original: [Equation]
Revised: [Equation]

Original:  where g [Equation] ) k are the clipped/normalized per-client update and perclient noise, respectively, as defined in Algorithm 1.
Revised: k (cid:13) (cid:13) , where g ( i ) k and ζ ( i ) k are the clipped/normalized per-client update and perclient noise, respectively, as defined in Algorithm 1.

Original: All plots are averaged over 3 independent runs.
Revised: All plots are averaged over 3 independent runs.

Original: For a fair comparison, in each run, the exact same noise vectors (sampled randomly at each round) are used in both algorithms.
Revised: For a fair comparison, in each run, the exact same noise vectors (sampled randomly at each round) are used in both algorithms.

Original: The captions of Figures 2 and 3 discuss the results in detail.
Revised: The captions of Figures 2 and 3 discuss the results in detail.

Original: For further illustration, in Figure 4, we show the 2D projection of the trajectories of DP-FedAvg with clipping and DP-NormFedAvg for 1) C = 100 and η = 0 .
Revised: For further illustration, in Figure 4, we show the 2D projection of the trajectories of DP-FedAvg with clipping and DP-NormFedAvg for 1) C = 100 and η = 0 .

Original: 001 , and 2) C = 50 and η = 0 .
Revised: 001 , and 2) C = 50 and η = 0 .

Original: 003 , both with initialization I1 .
Revised: 003 , both with initialization I1 .

Original: In both cases, note that DP-NormFedAvg reaches closer to the optimum than
Revised: In both cases, note that DP-NormFedAvg reaches closer to the optimum than

Original: DP-FedAvg with clipping.
Revised: DP-FedAvg with clipping.

Original: These plots corroborate our theoretical predictions and intuition.
Revised: These plots corroborate our theoretical predictions and intuition.

Original: [Figure]
Revised: [Figure]

Original: : C =
Revised: : C =

Original: [Figure]
Revised: [Figure]

Original: clipping with threshold C ≤ C low is equivalent to normalization with the same scaling factor
Revised: clipping with threshold C ≤ C low is equivalent to normalization with the same scaling factor

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: where g ( i ) k and ζ ( i ) k are the clipped/normalized per-client update and per-client noise, as defined in Algorithm 1.
Revised: where g ( i ) k and ζ ( i ) k are the clipped/normalized per-client update and per-client noise, as defined in Algorithm 1.

Original: The SNR for only one value of η is shown here as the trend for other values of η is similar (and to avoid congestion).
Revised: The SNR for only one value of η is shown here as the trend for other values of η is similar (and to avoid congestion).

Original: As per our discussion in Section 3.1, the SNR of normalization is never lower than that of clipping, explaining the superiority of the former.
Revised: As per our discussion in Section 3.1, the SNR of normalization is never lower than that of clipping, explaining the superiority of the former.

Original: Also, similar to the trend in Figure 2, the improvement in
Revised: Also, similar to the trend in Figure 2, the improvement in

Original: SNR of normalization is much higher for C
Revised: SNR of normalization is much higher for C

Original: = { 50 , 100 } than C = 40 .
Revised: = { 50 , 100 } than C = 40 .

Original: [Figure]
Revised: [Figure]

Original: Appendix G. Experiments on Datasets
Revised: Appendix G. Experiments on Datasets

Original: We consider the task of private multi-class classification to compare DP-FedAvg with clipping against DP-NormFedAvg ; for brevity, we will often call them just clipping and normalization, respectively.
Revised: We consider the task of private multi-class classification to compare DP-FedAvg with clipping against DP-NormFedAvg ; for brevity, we will often call them just clipping and normalization, respectively.

Original: Our experiments are performed on three benchmarking datasets, Fashion-MNIST [32] (abbreviated as FMNIST henceforth), CIFAR-10 and CIFAR-100, where the first two datasets have 10 classes each and the last one has 100 classes.
Revised: Our experiments are performed on three benchmarking datasets, Fashion-MNIST [32] (abbreviated as FMNIST henceforth), CIFAR-10 and CIFAR-100, where the first two datasets have 10 classes each and the last one has 100 classes.

Original: Specifically, we consider logistic regression on
Revised: Specifically, we consider logistic regression on

Original: FMNIST, CIFAR-10 and CIFAR-100 with ℓ 2 -regularization; the weight decay value in PyTorch for ℓ 2 -regularization is set to 1e-4.
Revised: FMNIST, CIFAR-10 and CIFAR-100 with ℓ 2 -regularization; the weight decay value in PyTorch for ℓ 2 -regularization is set to 1e-4.

Original: For FMNIST, we flatten each image into a 784 -dimensional vector and use that as the feature vector.
Revised: For FMNIST, we flatten each image into a 784 -dimensional vector and use that as the feature vector.

Original: For CIFAR-10 and CIFAR-100, we use 512-dimensional features extracted from the last layer of a ResNet-18 [18] model pretrained on ImageNet.
Revised: For CIFAR-10 and CIFAR-100, we use 512-dimensional features extracted from the last layer of a ResNet-18 [18] model pretrained on ImageNet.

Original: Similar to [23], we simulate a heterogeneous setting by distributing the data among the clients such that each client can have data from at most five classes.
Revised: Similar to [23], we simulate a heterogeneous setting by distributing the data among the clients such that each client can have data from at most five classes.

Original: The exact procedure is described in Appendix H. For the
Revised: The exact procedure is described in Appendix H. For the

Original: CIFAR-10 and CIFAR-100 (respectively, FMNIST) experiment, the number of clients n is set to 5000 (respectively, 3000), with each client having the same number of samples.
Revised: CIFAR-10 and CIFAR-100 (respectively, FMNIST) experiment, the number of clients n is set to 5000 (respectively, 3000), with each client having the same number of samples.

Original: The number of participating clients in each round is set to r = 0 .
Revised: The number of participating clients in each round is set to r = 0 .

Original: 2 n for all datasets, with 20 local client updates per-round.
Revised: 2 n for all datasets, with 20 local client updates per-round.

Original: We consider two privacy levels: ε = { 5 , 2 } with δ = 10 − 5 ; note that ε = 5 (resp., 2 ) corresponds to the low (resp., high) privacy regime.
Revised: We consider two privacy levels: ε = { 5 , 2 } with δ = 10 − 5 ; note that ε = 5 (resp., 2 ) corresponds to the low (resp., high) privacy regime.

Original: For clipping and normalization, the values of C that we tune over are { 500 , 250 , 125 , 62 . 5 , 31 . 25 , 15 . 625 } .
Revised: For clipping and normalization, the values of C that we tune over are { 500 , 250 , 125 , 62 . 5 , 31 . 25 , 15 . 625 } .

Original: Details about the learning rate schedule are in Appendix H.
Revised: Details about the learning rate schedule are in Appendix H.

Original: In Table 2, we show the comparison between clipping and normalization (in terms of test accuracy) for the two aforementioned privacy levels as well as vanilla FedAvg (without any privacy) as the baseline.
Revised: In Table 2, we show the comparison between clipping and normalization (in terms of test accuracy) for the two aforementioned privacy levels as well as vanilla FedAvg (without any privacy) as the baseline.

Original: The results reported here are the best ones for each algorithm by tuning over C and the learning rates, and have been averaged over three different runs.
Revised: The results reported here are the best ones for each algorithm by tuning over C and the learning rates, and have been averaged over three different runs.

Original: In all cases, normalization is clearly superior to clipping.
Revised: In all cases, normalization is clearly superior to clipping.

Original: It is worth noting that the improvement obtained with normalization is more for the low privacy regime (i.e., ε = 5 ).
Revised: It is worth noting that the improvement obtained with normalization is more for the low privacy regime (i.e., ε = 5 ).

Original: [Table]
Revised: [Table]

Original: Appendix H. Experimental Details
Revised: Appendix H. Experimental Details

Original: First, we explain the procedure we have used to generate heterogeneous data for our FL experiments in Appendix G. For each dataset (individually), the training data was first sorted based on labels and then divided into 5 n equal data-shards, where n is the number of clients.
Revised: First, we explain the procedure we have used to generate heterogeneous data for our FL experiments in Appendix G. For each dataset (individually), the training data was first sorted based on labels and then divided into 5 n equal data-shards, where n is the number of clients.

Original: Splitting the data in this way ensures that each shard contains data from only one class for all datasets (and because n was chosen appropriately).
Revised: Splitting the data in this way ensures that each shard contains data from only one class for all datasets (and because n was chosen appropriately).

Original: Now, each client is assigned 5 shards chosen uniformly at random without replacement which ensures that each client can have data belonging to at most 5 distinct classes.
Revised: Now, each client is assigned 5 shards chosen uniformly at random without replacement which ensures that each client can have data belonging to at most 5 distinct classes.

Original: Next, we specify the learning rate schedule for our experiments in Appendix G.
Revised: Next, we specify the learning rate schedule for our experiments in Appendix G.

Original: We use β k = 0 .
Revised: We use β k = 0 .

Original: 5 η k for all k .
Revised: 5 η k for all k .

Original: We employ the learning rate scheme suggested in [6] where we decrease the local learning rate by a factor of 0.99 after every round, i.e. η k = (0 . 99) k η 0 .
Revised: We employ the learning rate scheme suggested in [6] where we decrease the local learning rate by a factor of 0.99 after every round, i.e. η k = (0 . 99) k η 0 .

Original: We search the best initial local learning rates η 0 over { 10 − 3 , 2 × 10 − 3 , 4 × 10 − 3 , 8 × 10 − 3 , 10 − 2 , 2 × 10 − 2 , 4 × 10 − 2 , 8 × 10 − 2 , 10 − 1 , 2 × 10 − 1 , 4 × 10 − 1 , 8 × 10 − 1 } in each case.
Revised: We search the best initial local learning rates η 0 over { 10 − 3 , 2 × 10 − 3 , 4 × 10 − 3 , 8 × 10 − 3 , 10 − 2 , 2 × 10 − 2 , 4 × 10 − 2 , 8 × 10 − 2 , 10 − 1 , 2 × 10 − 1 , 4 × 10 − 1 , 8 × 10 − 1 } in each case.

Original: Server momentum = 0.8 is also applied.
Revised: Server momentum = 0.8 is also applied.

Original: All experiments are run on a single NVIDIA TITAN
Revised: All experiments are run on a single NVIDIA TITAN

Original: Xp GPU.
Revised: Xp GPU.