Original: B IT - AWARE R ANDOMIZED R ESPONSE FOR L OCAL D IFFERENTIAL P RIVACY IN F EDERATED L EARNING
Revised: B IT - AWARE R ANDOMIZED R ESPONSE FOR L OCAL D IFFERENTIAL P RIVACY IN F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we develop BitRand , a bit-aware randomized response algorithm, to preserve local differential privacy (LDP) in federated learning (FL).
Revised: In this paper we develop BitRand , a bit-aware randomized response algorithm, to preserve local differential privacy (LDP) in federated learning (FL).

Original: We encode embedded features extracted from clients’ local data into binary encoding bits, in which different bits have different impacts on the embedded features.
Revised: We encode embedded features extracted from clients’ local data into binary encoding bits, in which different bits have different impacts on the embedded features.

Original: Based upon that, we randomize all the bits to preserve LDP with three key advantages: (1)
Revised: Based upon that, we randomize all the bits to preserve LDP with three key advantages: (1)

Original: Bit-aware: Bits with a more substantial inﬂuence on the model utility have smaller randomization probabilities, and vice-versa, under the same privacy protection; (2) Dimension-elastic: Increasing the dimensions of embedded features, gradients, model outcomes and training rounds marginally affect the randomization probabilities of binary encoding bits under the same privacy protection; and (3) LDP protection is achieved for both embedded features and labels with tight privacy loss and expected error bounds ensuring high model utility.
Revised: Bit-aware: Bits with a more substantial inﬂuence on the model utility have smaller randomization probabilities, and vice-versa, under the same privacy protection; (2) Dimension-elastic: Increasing the dimensions of embedded features, gradients, model outcomes and training rounds marginally affect the randomization probabilities of binary encoding bits under the same privacy protection; and (3) LDP protection is achieved for both embedded features and labels with tight privacy loss and expected error bounds ensuring high model utility.

Original: Extensive theoretical and experimental results show that our BitRand signiﬁcantly outperforms various baseline approaches in text and image classiﬁcation
Revised: Extensive theoretical and experimental results show that our BitRand signiﬁcantly outperforms various baseline approaches in text and image classiﬁcation

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent data privacy and security regulations (GDPR, 2018; Regulation, 2018; Cybersecurity Law, 2016) pose major challenges in collecting and using personally sensitive data in different places for machine learning (ML) applications.
Revised: Recent data privacy and security regulations (GDPR, 2018; Regulation, 2018; Cybersecurity Law, 2016) pose major challenges in collecting and using personally sensitive data in different places for machine learning (ML) applications.

Original: Federated Learning (FL) (McMahan et al., 2017; Kairouz et al., 2019) is a promising way to address these challenges, enabling clients to jointly train ML models by sharing and aggregating gradients computed from clients’ local data through a coordinating server for model updates.
Revised: Federated Learning (FL) (McMahan et al., 2017; Kairouz et al., 2019) is a promising way to address these challenges, enabling clients to jointly train ML models by sharing and aggregating gradients computed from clients’ local data through a coordinating server for model updates.

Original: However, recent attacks (Zhu et al., 2019; Y. et al., 2021; Zhao et al., 2020a) have shown that clients’ training samples, each of which includes an input x and a ground-truth label y x , can be extracted from the shared gradients.
Revised: However, recent attacks (Zhu et al., 2019; Y. et al., 2021; Zhao et al., 2020a) have shown that clients’ training samples, each of which includes an input x and a ground-truth label y x , can be extracted from the shared gradients.

Original: These attacks underscore the implicit privacy risk in FL.
Revised: These attacks underscore the implicit privacy risk in FL.

Original: Our main goal is to provide a strong guarantee that the shared gradients protect the privacy of clients’ local data without undue sacriﬁce in model utility.
Revised: Our main goal is to provide a strong guarantee that the shared gradients protect the privacy of clients’ local data without undue sacriﬁce in model utility.

Original: Local differential privacy (LDP) has emerged as a crucial component in various FL applications (Yang et al., 2019; Kairouz et al., 2019).
Revised: Local differential privacy (LDP) has emerged as a crucial component in various FL applications (Yang et al., 2019; Kairouz et al., 2019).

Original: To achieve our goal, we focus on preserving LDP in cross-device FL, i.e., in which clients jointly train an FL model (Kairouz et al., 2019).
Revised: To achieve our goal, we focus on preserving LDP in cross-device FL, i.e., in which clients jointly train an FL model (Kairouz et al., 2019).

Original: In cross-device FL, existing LDP-preserving approaches can be categorized into three lines: Clients (1) add noise into local gradients derived from their local training samples, e.g., using Gaussian mechanism (Abadi et al., 2016), to protect membership information at the training sample level with
Revised: In cross-device FL, existing LDP-preserving approaches can be categorized into three lines: Clients (1) add noise into local gradients derived from their local training samples, e.g., using Gaussian mechanism (Abadi et al., 2016), to protect membership information at the training sample level with

Original: DP guarantees (Zheng et al., 2021;
Revised: DP guarantees (Zheng et al., 2021;

Original: Dong et al., 2019; Malekzadeh et al., 2021; Geyer et al., 2017;
Revised: Dong et al., 2019; Malekzadeh et al., 2021; Geyer et al., 2017;

Original: Huang et al., 2020), (2) add noise to local gradients using Randomized Response (RR) mechanisms to protect the values of the local gradients with LDP guarantees (Sun et al., 2021; Liu et al., 2020;
Revised: Huang et al., 2020), (2) add noise to local gradients using Randomized Response (RR) mechanisms to protect the values of the local gradients with LDP guarantees (Sun et al., 2021; Liu et al., 2020;

Original: Zhao et al., 2020b; Wang et al., 2019a), and (3) add noise into each training sample, e.g., embedded features and labels, using RR mechanisms to protect the value of each training sample with LDP guarantees (Arachchige et al., 2019;
Revised: Zhao et al., 2020b; Wang et al., 2019a), and (3) add noise into each training sample, e.g., embedded features and labels, using RR mechanisms to protect the value of each training sample with LDP guarantees (Arachchige et al., 2019;

Original: Lyu et al., 2020a), then the clients use LDP-preserved training samples to derive local gradients.
Revised: Lyu et al., 2020a), then the clients use LDP-preserved training samples to derive local gradients.

Original: For all three approaches, in each training round, clients send DP or
Revised: For all three approaches, in each training round, clients send DP or

Original: LDP-preserved local gradients to the coordinating server for model updates, which will be sent back to the clients for the next training round.
Revised: LDP-preserved local gradients to the coordinating server for model updates, which will be sent back to the clients for the next training round.

Original: In this paper, we focus on protecting clients’ training data at the value level with LDP guarantees.
Revised: In this paper, we focus on protecting clients’ training data at the value level with LDP guarantees.

Original: Existing RR mechanisms to preserve LDP in FL suffer from the curse of privacy composition , in which excessive privacy budgets are consumed proportionally to the large dimensions of input or embedded features (Arachchige et al., 2019), gradients (Zhao et al., 2020b; Wang et al., 2019a), and training rounds (Zhao et al., 2020b; Wang et al.,
Revised: Existing RR mechanisms to preserve LDP in FL suffer from the curse of privacy composition , in which excessive privacy budgets are consumed proportionally to the large dimensions of input or embedded features (Arachchige et al., 2019), gradients (Zhao et al., 2020b; Wang et al., 2019a), and training rounds (Zhao et al., 2020b; Wang et al.,

Original: Addressing the curse of privacy composition is non-trivial.
Revised: Addressing the curse of privacy composition is non-trivial.

Original: Existing approaches, such as anonymizers (assumed to be trusted), i.e., shufﬂer (Erlingsson et al., 2019; Sun et al., 2021; L. et al., 2020; Wang et al., 2019b; Cheu et al., 2019; Balle et al., 2019) or anonymity approaches (e.g., faking source 
Revised: Existing approaches, such as anonymizers (assumed to be trusted), i.e., shufﬂer (Erlingsson et al., 2019; Sun et al., 2021; L. et al., 2020; Wang et al., 2019b; Cheu et al., 2019; Balle et al., 2019) or anonymity approaches (e.g., faking source IP,

Original: IP, VPN, Proxy, etc.
Revised: VPN, Proxy, etc.

Original: (Sun et al., 2021; Cormode et al., 2018)), and dimension reduction (Liu et al., 2020; Zhao et al., 2020b; Shin et al., 2018; Xu et al., 2019), mitigate the problem but also have limitations.
Revised: (Sun et al., 2021; Cormode et al., 2018)), and dimension reduction (Liu et al., 2020; Zhao et al., 2020b; Shin et al., 2018; Xu et al., 2019), mitigate the problem but also have limitations.

Original: In the real world, it is possible that the anonymizers can either be compromised or collude with the coordinating server to extract sensitive information from observing LDP-preserved local gradients (Erlingsson et al., 2019).
Revised: In the real world, it is possible that the anonymizers can either be compromised or collude with the coordinating server to extract sensitive information from observing LDP-preserved local gradients (Erlingsson et al., 2019).

Original: In addition, the LDP-preserved local gradients cannot have any discriminating characteristics and must, in particular, all utilize the same local randomizer (since the distribution of random values may be revealing)

Original: (Erlingsson et al., 2019).

Original: That may not always be achievable in practice.

Original: Meanwhile, applying RR mechanisms on reduced sets of embedded features or gradients using dimension reduction techniques can work well with lightweight models, such as logistic regression and SVM (Liu et al., 2020; Zhao et al., 2020b; Wang et al., 2019a).
Revised: Meanwhile, applying RR mechanisms on reduced sets of embedded features or gradients using dimension reduction techniques can work well with lightweight models, such as logistic regression and SVM (Liu et al., 2020; Zhao et al., 2020b; Wang et al., 2019a).

Original: However, it is challenging for these techniques to achieve good model utility under tight LDP guarantees given complex models and tasks, such as DNNs, since the dimensions of reduced embedded features, gradients, and training rounds still need to be sufﬁciently large (Zhao et al., 2020b; Liu et al., 2020).
Revised: However, it is challenging for these techniques to achieve good model utility under tight LDP guarantees given complex models and tasks, such as DNNs, since the dimensions of reduced embedded features, gradients, and training rounds still need to be sufﬁciently large (Zhao et al., 2020b; Liu et al., 2020).

Original: Hence, the curse of privacy composition in preserving LDP by applying RR mechanisms in FL remains a largely open problem.
Revised: Hence, the curse of privacy composition in preserving LDP by applying RR mechanisms in FL remains a largely open problem.

Original: Orthogonal to this, preserving LDP to protect ground-truth labels y x
Revised: Orthogonal to this, preserving LDP to protect ground-truth labels y x

Original: in FL has not been well-studied.
Revised: in FL has not been well-studied.

Original: Two known approaches for centralized training are 1) injecting
Revised: Two known approaches for centralized training are 1) injecting

Original: Laplace noise into the labels (Phan et al., 2020) and 2) applying RR mechanisms on the labels to achieve DP at the label level (Ghazi et al., 2021).
Revised: Laplace noise into the labels (Phan et al., 2020) and 2) applying RR mechanisms on the labels to achieve DP at the label level (Ghazi et al., 2021).

Original: However, centralized training in Ghazi et al.
Revised: However, centralized training in Ghazi et al.

Original: (2021) has not been designed for FL with LDP guarantees since they require centralized and trusted databases.
Revised: (2021) has not been designed for FL with LDP guarantees since they require centralized and trusted databases.

Original: The model utility in Phan et al.
Revised: The model utility in Phan et al.

Original: (2020) is notably affected by the number of model outcomes.
Revised: (2020) is notably affected by the number of model outcomes.

Original: Key Contributions.
Revised: Key Contributions.

Original: To address the curse of privacy composition and optimize the trade-off between privacy and model utility, our paper is structured around the following contributions:
Revised: To mitigate the curse of privacy composition and optimize the trade-off between privacy and model utility, our paper is structured around the following contributions:

Original: 1) We propose BitRand , which is a combination of a novel bit-aware f -RR mechanism and label -RR mechanism, to preserve LDP at both levels of embedded features and labels in FL.
Revised: 1) We propose BitRand , which is a combination of a novel bit-aware f -RR mechanism and label -RR mechanism, to preserve LDP at both levels of embedded features and labels in FL.

Original: In f -RR , we encode embedded features (extracted from x ) into a binary-bit string, which will be adaptively randomized such that bits with a more substantial impact on model utility will have smaller randomization probabilities and vice-versa under the same privacy budget.
Revised: In f -RR , we encode embedded features (extracted from x ) into a binary-bit string, which will be adaptively randomized such that bits with a more substantial impact on model utility will have smaller randomization probabilities and vice-versa under the same privacy budget.

Original: To preserve LDP on y x , we develop a generalized randomization, in which the probability of randomizing label y x from one to another class is a function of the number of model outcomes C .
Revised: To preserve LDP on y x , we develop a generalized randomization, in which the probability of randomizing label y x from one to another class is a function of the number of model outcomes C .

Original: By doing that, we can optimize the trade-off between model utility and privacy loss with signiﬁcantly tighter expected error bounds.
Revised: By doing that, we can optimize the trade-off between model utility and privacy loss with signiﬁcantly tighter expected error bounds.

Original: By incorporating sensitivities of binary encoding bits into a generalized privacy loss bound, we show that increasing the dimensions of embedded features r , encoding bits l , and model outcomes
Revised: By incorporating sensitivities of binary encoding bits into a generalized privacy loss bound, we show that increasing the dimensions of embedded features r , encoding bits l , and model outcomes

Original: C marginally affect the randomization probabilities in BitRand under the same privacy budget.
Revised: C marginally affect the randomization probabilities in BitRand under the same privacy budget.

Original: This dimension-elastic property is crucial to evade the curse of privacy composition by retaining a high value of data transmitted correctly through our randomization given large dimensions of r , l , and C . 3)
Revised: This dimension-elastic property is crucial to evade the curse of privacy composition by retaining a high value of data transmitted correctly through our randomization given large dimensions of r , l , and C . 3)

Original: These bit-aware and dimension-elastic properties allow us to work with complex models and tasks with formal LDP guarantees for training samples( x, y x ) while retaining high model utility.
Revised: These bit-aware and dimension-elastic properties allow us to work with complex models and tasks with formal LDP guarantees for training samples

Original: Extensive theoretical analysis and experimental results conducted on fundamental FL tasks, i
Revised: ( x , y x ) while retaining high

Original: e., text and image classiﬁcation, using benchmark datasets and our collected Security and Exchange Commission ﬁnancial contract dataset show that our BitRand signiﬁcantly outperforms a variety of baseline approaches in terms of model utility under the same privacy budget.
Revised: model utility Extensive theoretical analysis and experimental results conducted on fundamental FL tasks, i.e., text and image classiﬁcation, using benchmark datasets and our collected Security and Exchange Commission ﬁnancial contract dataset show that our BitRand signiﬁcantly outperforms a variety of baseline approaches in terms of model utility under the same privacy budget.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: LDP-preserving mechanisms (Erlingsson et al., 2014; Duchi et al., 2018; Wang et al., 2017; Acharya et al., 2019; Bassily & Smith, 2015) generally build on the ideas of randomized response (Warner, 1965), which was initially introduced to allow survey respondents to provide their correct inputs while maintaining their conﬁdentiality.
Revised: LDP-preserving mechanisms (Erlingsson et al., 2014; Duchi et al., 2018; Wang et al., 2017; Acharya et al., 2019; Bassily & Smith, 2015) generally build on the ideas of randomized response (Warner, 1965), which was initially introduced to allow survey respondents to provide their correct inputs while maintaining their conﬁdentiality.

Original: (cid:15) -LDP is presented as follows:
Revised: (cid:15) -LDP is presented as follows:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: (cid:15) -LDP.
Revised: (cid:15) -LDP.

Original: A randomized algorithm M fulﬁlls (cid:15) -LDP, if for any two inputs x and x (cid:48) , and for all possible outputs O ∈ Range ( M ) , we have: Pr [ M ( x ) =
Revised: A randomized algorithm M fulﬁlls (cid:15) -LDP, if for any two inputs x and x (cid:48) , and for all possible outputs O ∈ Range ( M ) , we have: Pr [ M ( x ) =

Original: O ] ≤ e (cid:15)
Revised: O ] ≤ e (cid:15)

Original: Pr [ M ( x (cid:48) )
Revised: Pr [ M ( x (cid:48) )

Original: =
Revised: =

Original: O ] , where (cid:15) is a privacy budget and Range ( M ) denotes every possible output of the algorithm M .
Revised: O ] , where (cid:15) is a privacy budget and Range ( M ) denotes every possible output of the algorithm M .

Original: The privacy budget (cid:15) controls the amount by which the distributions induced by inputs x and x (cid:48) may differ.
Revised: The privacy budget (cid:15) controls the amount by which the distributions induced by inputs x and x (cid:48) may differ.

Original: A smaller (cid:15) enforces a stronger privacy guarantee.
Revised: A smaller (cid:15) enforces a stronger privacy guarantee.

Original: We revisit RR mechanisms for
Revised: We revisit RR mechanisms for

Original: LDP preservation in Appendix A .
Revised: LDP preservation in Appendix A .

Original: Our approach is a binary encoding-based approach, similar to (Arachchige et al., 2019;
Revised: Our approach is a binary encoding-based approach, similar to (Arachchige et al., 2019;

Original: Lyu et al., 2020a), since it has the potential to overcome the curse of privacy composition.
Revised: Lyu et al., 2020a), since it has the potential to overcome the curse of privacy composition.

Original: In binary encoding, x is converted into an l -bit vector v consisting of 1 sign bit, m bits for the integer part, and l − m − 1 bits for the fraction part (Figure 1), as follows:
Revised: In binary encoding, x is converted into an l -bit vector v consisting of 1 sign bit, m bits for the integer part, and l − m − 1 bits for the fraction part (Figure 1), as follows:

Original: [Equation]
Revised: [Equation]

Original: Each bit in v is randomized by applying a RR mechanism, e.g., (Erlingsson et al., 2014; Bassily & Smith, 2015; Wang et al., 2017), to generate a perturbed l -bit vector v (cid:48) , which preserves LDP.
Revised: Each bit in v is randomized by applying a RR mechanism, e.g., (Erlingsson et al., 2014; Bassily & Smith, 2015; Wang et al., 2017), to generate a perturbed l -bit vector v (cid:48) , which preserves LDP.

Original: However, in our theoretical reassessment ( AppendicesH and I  ), directly applying RR mechanisms on binary encoded vectors as in existing mechanisms, i.e.,
Revised: However, in our theoretical reassessment ( Appendices I and J ), directly applying RR mechanisms on binary encoded vectors as in existing mechanisms, i.e.,

Original: LATENT (Arachchige et al., 2019) and OME (Lyu et al., 2020a), consumes huge privacy budgets since each binary encoding bit cannot be treated as a bit in a hash.
Revised: LATENT (Arachchige et al., 2019) and OME (Lyu et al., 2020a), consumes huge privacy budgets since each binary encoding bit cannot be treated as a bit in a hash.

Original: Each binary encoding bit i has a different sensitivity, i.e., ∆ i = 2 m − i for the integer and fraction parts or ∆ i =
Revised: Each binary encoding bit i has a different sensitivity, i.e., ∆ i = 2 m − i for the integer and fraction parts or ∆ i =

Original: 2 m +1 for the sign bit ( Lemma 1 ), compared with a bit B i in a hash, i.e., ∀ B i : ∆ B i = 1 .
Revised: 2 m +1 for the sign bit ( Lemma 1 ), compared with a bit B i in a hash, i.e., ∀ B i : ∆ B i = 1 .

Original: Our mechanism does not suffer from this problem, thanks to our bit-aware randomization in binary encoding ( Theorem 2 ).
Revised: Our mechanism does not suffer from this problem, thanks to our bit-aware randomization in binary encoding ( Theorem 2 ).

Original:  B IT R AND A LGORITHM
Revised:  B IT R AND A LGORITHM

Original: Let us now present our FL setting, threat model, and BitRand algorithm (Figure 2 and
Revised: Let us now present our FL setting, threat model, and BitRand algorithm (Figure 2 and

Original: Alg. 1, Appendix B), and privacy guarantees.
Revised: Alg. 1, Appendix B), and privacy guarantees.

Original: Then, we will show that our algorithm is dimension-elastic and the ability to optimize the randomization probabilities with expected error bounds in our theoretical analysis.
Revised: Then, we will show that our algorithm is dimension-elastic and the ability to optimize the randomization probabilities with expected error bounds in our theoretical analysis.

Original: Federated Learning Given N clients, each client u ∈ [1 , N ] has a set of training samples D u
Revised: Federated Learning Given N clients, each client u ∈ [1 , N ] has a set of training samples D u

Original: = { ( x, y x ) } n u where x ∈ R d is the input features, and its associated ground-truth label y x ∈ Z C is one-hot
Revised: = { ( x, y x ) } n u where x ∈ R d is the input features, and its associated ground-truth label y x ∈ Z C is one-hot

Original: encoded with C categorical model outcomes { y x, 1 , . .
Revised: encoded with C categorical model outcomes { y x, 1 , . .

Original: , y x,C } , and n u is the number of training samples.
Revised: , y x,C } , and n u is the number of training samples.

Original: In a pre-processing step, each client u extracts r numerical embedded features from x , denoted e x , by using a pre-trained model f pre .
Revised: In a pre-processing step, each client u extracts r numerical embedded features from x , denoted e x , by using a pre-trained model f pre .

Original: In practice, f pre could be trained on large-scale and publicly available datasets without introducing any extra privacy cost to the clients’ local data (He et al., 2016; Devlin et al., 2018).
Revised: In practice, f pre could be trained on large-scale and publicly available datasets without introducing any extra privacy cost to the clients’ local data (He et al., 2016; Devlin et al., 2018).

Original: N clients jointly train the model f θ : R r
Revised: N clients jointly train the model f θ : R r

Original: → R C by minimizing a loss function L (cid:0) f θ ( e x ) , y x (cid:1) on their local training samples in D u that penalizes mismatching between the prediction f θ ( e x ) and the ground-truth label y x  .
Revised: → R C by minimizing a loss function L (cid:0) f θ ( e x ) , y x (cid:1) on their local training samples in D u that penalizes mismatching between the prediction f θ ( e x ) and the ground-truth label y x , given the model’s parameters θ .

Original: In each training round t , each client u receives the most updated model parameters θ t from the coordinating server and then computes local gradients (cid:53) uθ t
Revised: In each training round t , each client u receives the most updated model parameters θ t from the coordinating server and then computes local gradients (cid:53) uθ t

Original: = (cid:80) ( v x ,y x ) ∈ D u (cid:53) θ t L ( f ( e x ) , y x ) /n u , which are sent to the coordinating server for aggregating and model updating: 
Revised: = (cid:80) ( v x ,y x ) ∈ D u (cid:53) θ t L ( f ( e x ) , y x ) /n u , which are sent to the coordinating server for aggregating and model updating: θ t +1 =

Original: [Equation] (cid:53) uθ [Equation] /N .
Revised: θ t − η t (cid:80) u ∈ [1 ,N ] (cid:53) uθ t /N .

Original: Threat Model.
Revised: Threat Model.

Original: The coordinating server strictly follows the training procedure but curious about the training data D u .
Revised: The coordinating server strictly follows the training procedure but curious about the training data D u .

Original: This is a practical threat model in the real world since service providers always aim at providing the best services to the clients (Haeberlen et al., 2011; Truex et al., 2019;
Revised: This is a practical threat model in the real world since service providers always aim at providing the best services to the clients (Haeberlen et al., 2011; Truex et al., 2019;

Original: Lyu et al., 2020b).
Revised: Lyu et al., 2020b).

Original: Given the observed gradients (cid:53) uθ t , the coordinating server can extract the clients’ data { e x , y x } n u by using recently developed attacks (Carlini et al., 2020; Fredrikson et al., 2015).
Revised: Given the observed gradients (cid:53) uθ t , the coordinating server can extract the clients’ data { e x , y x } n u by using recently developed attacks (Carlini et al., 2020; Fredrikson et al., 2015).

Original: In a defense-free environment, { e x , y x } n u can be used to infer the sensitive training data
Revised: In a defense-free environment, { e x , y x } n u can be used to infer the sensitive training data

Original: D u = { ( x, y x ) } n u using the pre-trained model f pre , since f pre ( x ) =
Revised: D u = { ( x, y x ) } n u using the pre-trained model f pre , since f pre ( x ) =

Original: e x (Song & Raghunathan, 2020).
Revised: e x (Song & Raghunathan, 2020).

Original: This poses a severe privacy risk to the sensitive data D u .
Revised: This poses a severe privacy risk to the sensitive data D u .

Original: BitRand Algorithm.
Revised: BitRand Algorithm.

Original: To protect the sensitive training data D u against the threat model, in our algorithm, we preserve LDP on both embedded features e x and labels y x .
Revised: To protect the sensitive training data D u against the threat model, in our algorithm, we preserve LDP on both embedded features e x and labels y x .

Original: Each of the r embedded features in e x is encoded into l binary bits following Eq. 1.
Revised: Each of the r embedded features in e x is encoded into l binary bits following Eq. 1.

Original: Binary encoded features are concatenated together into a vector v x consisting of rl binary bits to represent the embedded features e x (Alg. 1, line 16).
Revised: Binary encoded features are concatenated together into a vector v x consisting of rl binary bits to represent the embedded features e x (Alg. 1, line 16).

Original: Each bit i ∈ [0 , rl − 1] in v x is randomized by our f -RR mechanism (Alg. 1, line 17) with a bit-aware term i % l l optimizing randomization probabilities:
Revised: Each bit i ∈ [0 , rl − 1] in v x is randomized by our f -RR mechanism (Alg. 1, line 17) with a bit-aware term i % l l optimizing randomization probabilities:

Original: [Equation]
Revised: [Equation]

Original: where v x ( i ) ∈ { 0 , 1 } is the value of v x at the bit i , v (cid:48)
Revised: where v x ( i ) ∈ { 0 , 1 } is the value of v x at the bit i , v (cid:48)

Original: x is the perturbed vector created by randomizing all the bits in v x , (cid:15) X is a privacy budget, and α is a parameter bounded in Theorem 2.
Revised: x is the perturbed vector created by randomizing all the bits in v x , (cid:15) X is a privacy budget, and α is a parameter bounded in Theorem 2.

Original: From Eq.
Revised: From Eq.

Original: 2, we also have that P ( v (cid:48)
Revised: 2, we also have that P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − p X if v x ( i ) = 1 , and P ( v (cid:48)
Revised: = 1 − p X if v x ( i ) = 1 , and P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − q X if v x ( i )
Revised: = 1 − q X if v x ( i )

Original: = 0 .
Revised: = 0 .

Revised: We use the bit-aware term i % l l to indicate the location of bit i , which is associated with the sensitivity of the bit at that location, in its l -bit binary encoded vector among rl concatenated binary bits.

Original: One of the key differences between our mechanism and existing works (Sun et al., 2021; Zhao et al., 2020b; Wang et al., 2019a;
Revised: One of the key differences between our mechanism and existing works (Sun et al., 2021; Zhao et al., 2020b; Wang et al., 2019a;

Original: Liu et al., 2020; Arachchige et al., 2019;
Revised: Liu et al., 2020; Arachchige et al., 2019;

Original: Lyu et al., 2020a) is the bit-aware randomization probabilities.
Revised: Lyu et al., 2020a) is the bit-aware randomization probabilities.

Original: By introducing the bit-aware term i % l l , we are able to: 1)
Revised: By introducing the bit-aware term i % l l , we are able to: 1)

Original: Derive signiﬁcantly tighter privacy loss and expected error bounds compared with existing approaches ( Sections 4 and 5 ); and 2) Adaptively control the randomization probabilities across bits, such that bits with a stronger inﬂuence on the model utility, e.g., sign bits and integer bits, have smaller randomization probabilities q X , and vice-versa ( Section 5 ).
Revised: Derive signiﬁcantly tighter privacy loss and expected error bounds compared with existing approaches ( Sections 4 and 5 ); and 2) Adaptively control the randomization probabilities across bits, such that bits with a stronger inﬂuence on the model utility, e.g., sign bits and integer bits, have smaller randomization probabilities q X , and vice-versa ( Section 5 ).

Original: These advantages are crucial to evade the curse of privacy composition enabling us to work with complex tasks and models ( Section 6 ).
Revised: These advantages are crucial to evade the curse of privacy composition enabling us to work with complex tasks and models ( Section 6 ).

Original: In addition, inspired by the LabelDP (Ghazi et al., 2021), we randomize y x using the following label -RR mechanism (Alg. 1, line 18):
Revised: In addition, inspired by the LabelDP (Ghazi et al., 2021), we randomize y x using the following label -RR mechanism (Alg. 1, line 18):

Original: [Equation]
Revised: [Equation]

Original: where ¯ y x is one-hot encoded, and β is a parameter bounded in Theorem 3 under a privacy budget (cid:15) Y .
Revised: where ¯ y x is one-hot encoded, and β is a parameter bounded in Theorem 3 under a privacy budget (cid:15) Y .

Original: Randomizing the label y x provides a complete LDP protection to each local training sample ( e x , y x ) .
Revised: Randomizing the label y x provides a complete LDP protection to each local training sample ( e x , y x ) .

Original: All the perturbed training samples ( e (cid:48) x , y (cid:48) x ) are included in a local dataset D (cid:48) u , which will be used to train the model f θ : R r
Revised: All the perturbed training samples ( e (cid:48) x , y (cid:48) x ) are included in a local dataset D (cid:48) u , which will be used to train the model f θ : R r

Original: → R C , i.e., e (cid:48) x
Revised: → R C , i.e., e (cid:48) x

Original: = E ( v (cid:48) x ) where E ( · ) is a decoding function, (Alg. 1, lines 20-22).
Revised: = E ( v (cid:48) x ) where E ( · ) is a decoding function, (Alg. 1, lines 20-22).

Original: The training will never access the original data D u .
Revised: The training will never access the original data D u .

Original: All other operations remain the same with our aforementioned FL setting.
Revised: All other operations remain the same with our aforementioned FL setting.

Original:  P RIVACY G UARANTEES OF B IT R AND
Revised:  P RIVACY G UARANTEES OF B IT R AND

Original: In this section, we focus on bounding formal privacy loss of BitRand for input and label protection.
Revised: In this section, we focus on bounding formal privacy loss of BitRand for input and label protection.

Original: To achieve our goal, we need to bound α and β in Eqs.
Revised: To achieve our goal, we need to bound α and β in Eqs.

Original: 2 and 3 such that our algorithm preserves LDP for each training sample
Revised: 2 and 3 such that our algorithm preserves LDP for each training sample

Original: ( x, y x ) ∈ D u given ( v (cid:48)
Revised: ( x, y x ) ∈ D u given ( v (cid:48)

Original: x , y (cid:48) x ) ∈ D (cid:48)
Revised: x , y (cid:48) x ) ∈ D (cid:48)

Original: u .
Revised: u .

Original: Note that the decoding e (cid:48) x
Revised: Note that the decoding e (cid:48) x

Original: = E ( v (cid:48) x ) does not incur any extra privacy risk following the post-processing property (Dwork & Roth, 2014).
Revised: = E ( v (cid:48) x ) does not incur any extra privacy risk following the post-processing property (Dwork & Roth, 2014).

Original: Given v x and (cid:101) v x can be different at any bit, for the LDP condition to hold, the ratio of two probabilities P ( f -RR ( v x )= v z )
Revised: Given v x and (cid:101) v x can be different at any bit, for the LDP condition to hold, the ratio of two probabilities P ( f -RR ( v x )= v z )

Original: P ( f -RR ( (cid:101) v x )= v z ) needs to be bounded by exp( (cid:15) X ) with v z ∈ Range ( f -RR ) .
Revised: P ( f -RR ( (cid:101) v x )= v z ) needs to be bounded by exp( (cid:15) X ) with v z ∈ Range ( f -RR ) .

Original: Given a feature a , i.e., one of the r features, let us ﬁrst consider a bit i in the encoding vector v a .
Revised: Given a feature a , i.e., one of the r features, let us ﬁrst consider a bit i in the encoding vector v a .

Original: Intuitively, when we apply our f -RR only on the bit i in v a (i.e., all the other bits remain the same), denoted as f -RR ( v a , i ) , the l 1 -sensitivity of the bit i is 1.
Revised: Intuitively, when we apply our f -RR only on the bit i in v a (i.e., all the other bits remain the same), denoted as f -RR ( v a , i ) , the l 1 -sensitivity of the bit i is 1.

Original: This is because, given two neighboring vectors v a and v a | i that differs only at the bit i , we have that ∀ i ∈ v a : arg max v a (cid:107) f -RR ( v a , i ) − f -RR
Revised: This is because, given two neighboring vectors v a and v a | i that differs only at the bit i , we have that ∀ i ∈ v a : arg max v a (cid:107) f -RR ( v a , i ) − f -RR

Original: ( v a | i , i ) (cid:107) 1 = 1 .
Revised: ( v a | i , i ) (cid:107) 1 = 1 .

Original: However, the coordinating server can infer the decoded features E ( f -RR ( v a )) instead of just the intermediate result f -RR ( v a ) .
Revised: However, the coordinating server can infer the decoded features E ( f -RR ( v a )) instead of just the intermediate result f -RR ( v a ) .

Original: Thus, we need to quantify the l 1 -sensitivity of a single encoding bit i ∈ v a to determine just how accurately we can return the decoded features
Revised: Thus, we need to quantify the l 1 -sensitivity of a single encoding bit i ∈ v a to determine just how accurately we can return the decoded features

Original: E ( f -RR ( v a )) through our randomization f -RR applied on v a .
Revised: E ( f -RR ( v a )) through our randomization f -RR applied on v a .

Original: Following (Dwork & Roth, 2014), the sensitivity ∆ i of a bit i can be quatiﬁed as follows:
Revised: Following (Dwork & Roth, 2014), the sensitivity ∆ i of a bit i can be quatiﬁed as follows:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Bit l 1 -sensitivity.
Revised: Bit l 1 -sensitivity.

Original: Given two neighboring vectors v a and v a | i that differs only at a bit i , the sensitivity ∆ i captures the magnitude by which the bit i can change the decoding function
Revised: Given two neighboring vectors v a and v a | i that differs only at a bit i , the sensitivity ∆ i captures the magnitude by which the bit i can change the decoding function

Original: E ( f -RR ( · )) in the worst case, as follows:
Revised: E ( f -RR ( · )) in the worst case, as follows:

Original: [Equation]
Revised: [Equation]

Original: Based on Eq. 4, sensitivities ∆ i of all the bits in v x are bounded in the following lemma.
Revised: Based on Eq. 4, sensitivities ∆ i of all the bits in v x are bounded in the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: The l 1 -sensitivity of a single binary encoding bit is bounded as follows:
Revised: The l 1 -sensitivity of a single binary encoding bit is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: All the proofs are in Appendix.
Revised: All the proofs are in Appendix.

Original: Unlike existing RR mechanisms, we incorporate the l 1 -sensitivity ∆ i into the privacy loss bound of f -RR by ensuring that the privacy loss in randomizing a bit i is bounded by the privacy loss in the embedded feature space.
Revised: Unlike existing RR mechanisms, we incorporate the l 1 -sensitivity ∆ i into the privacy loss bound of f -RR by ensuring that the privacy loss in randomizing a bit i is bounded by the privacy loss in the embedded feature space.

Original: By doing so, we can derive tighter privacy loss bounds as discussed next.
Revised: By doing so, we can derive tighter privacy loss bounds as discussed next.

Original: Given ∆ i , there always exists a Laplace noise injected into the feature a , i.e., M ( v a , i ) =
Revised: Given ∆ i , there always exists a Laplace noise injected into the feature a , i.e., M ( v a , i ) =

Original: E ( v a ) + Lap (∆ i /(cid:15) i ) , to achieve (cid:15) i -LDP in the embedded feature space (Dwork & Roth, 2014).
Revised: E ( v a ) + Lap (∆ i /(cid:15) i ) , to achieve (cid:15) i -LDP in the embedded feature space (Dwork & Roth, 2014).

Original: In other words, P ( M ( v a ,i )=
Revised: In other words, P ( M ( v a ,i )=

Original: z ) P ( M ( v a
Revised: z ) P ( M ( v a

Original: [Equation]
Revised: [Equation]

Original: ,i )=
Revised: ,i )=

Original: z ) ≤ exp( (cid:15)
Revised: z ) ≤ exp( (cid:15)

Original: [Equation]
Revised: [Equation]

Original: ( v a
Revised: ( v a

Original: [Equation]
Revised: [Equation]

Original: ( v a | i ) | ∆ i ) and z ∈ Range ( M ) .
Revised: ( v a | i ) | ∆ i ) and z ∈ Range ( M ) .

Original: To ensure that the privacy loss in randomizing the bit i is bounded by the privacy loss in the embedded feature space is to ﬁnd α in Eq.
Revised: To ensure that the privacy loss in randomizing the bit i is bounded by the privacy loss in the embedded feature space is to ﬁnd α in Eq.

Original: 2 such that:
Revised: 2 such that:

Original: [Equation]
Revised: [Equation]

Original: However, randomizing a binary encoding bit i given v a and v a | i results in a smaller l 1 -distance |E ( f -RR ( v a , i ))
Revised: However, randomizing a binary encoding bit i given v a and v a | i results in a smaller l 1 -distance |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i )) | compared with |E ( v a ) − E ( v a | i ) | ; since |E ( f -RR ( v a , i ))
Revised: − E ( f -RR ( v a | i , i )) | compared with |E ( v a ) − E ( v a | i ) | ; since |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i ))
Revised: − E ( f -RR ( v a | i , i ))

Original: | ≤ |E ( v a ) − E ( v a | i ) | .
Revised: | ≤ |E ( v a ) − E ( v a | i ) | .

Original: Thus, we can derive a tighter bound by replacing |E ( v a ) − E
Revised: Thus, we can derive a tighter bound by replacing |E ( v a ) − E

Original: ( v a | i ) | with |E ( f -RR ( v a , i ))
Revised: ( v a | i ) | with |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i )) | .
Revised: − E ( f -RR ( v a | i , i )) | .

Original: Given v a ( i ) is the bit i in v a , we have
Revised: Given v a ( i ) is the bit i in v a , we have

Original: [Equation]
Revised: [Equation]

Original: However, ﬁnding a closed-form solution of α for the tight privacy bound in Eq.
Revised: However, ﬁnding a closed-form solution of α for the tight privacy bound in Eq.

Original: 6 is non-trivial, since (cid:15) i is intractable.
Revised: 6 is non-trivial, since (cid:15) i is intractable.

Original: To address this, we consider the privacy loss in the worse case P ( f -RR ( v a ( i ))
Revised: To address this, we consider two cases: (1) P ( f -RR ( v a ( i ))

Original: =
Revised: =

Original: v z ( i ))
Revised: v z ( i ))

Original: P ( f -RR ( v a | i ( i ))
Revised: P ( f -RR ( v a | i ( i ))

Original: = v z ( i )) ≥ 1 .
Revised: = v z ( i )) ≥ 1 below, and (2)

Original: Since ∆ i captures the magnitude by which the bit i can change the decoding function E ( · ) in the worst case, we have
Revised: 0 < P ( f -RR ( v a ( i ))

Original: [Equation] ,i
Revised: = v z ( i ))

Original: [Equation] f [Equation]
Revised: P ( f -RR ( v a | i ( i ))

Original: i [Equation] i ))
Revised: = v z ( i ))

Original: |
Revised: < 1 in Appendix D. Since ∆ i captures the magnitude by which the bit i can change the decoding function E ( · ) in the worst case, we have

Original: [Equation]
Revised: [Equation]

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 7 enables us to quantify a generalized privacy loss bound of a RR mechanism, in which different bits have different sensitivities by randomizing all the bits in v x independently in Theorem 1.
Revised: 7 enables us to quantify a generalized privacy loss bound of a RR mechanism, in which different bits have different sensitivities by randomizing all the bits in v x independently in Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Generalized privacy loss bound.
Revised: Generalized privacy loss bound.

Original: The privacy loss for randomizing a binary encoding vector v x is bounded as follows:
Revised: The privacy loss for randomizing a binary encoding vector v x is bounded as follows:

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Table]
Revised: [Equation]

Original:  Now, we enforce the condition (cid:80) rl − 1 i =0 (cid:15) i =
Revised: [Equation] Now, we enforce the condition (cid:80) rl − 1 i =0 (cid:15) i =

Original: (cid:15) X to ensure that the total privacy budget will be bounded by (cid:15) X .
Revised: (cid:15) X to ensure that the total privacy budget will be bounded by (cid:15) X .

Original: Based upon that, we derive a closed-form solution showing that there always exists an upper bound of α so that v (cid:48) x preserves (cid:15) X -LDP given v x in Theorem 2 (Alg. 1, line 17).
Revised: Based upon that, we derive a closed-form solution showing that there always exists an upper bound of α so that v (cid:48) x preserves (cid:15) X -LDP given v x in Theorem 2 (Alg. 1, line 17).

Original: Theorem 2.
Revised: Theorem 2.

Original: ∀ α : 0 < α ≤ (cid:113)(cid:15)
Revised: ∀ α : 0 < α ≤ (cid:113)

Original: [Equation]
Revised: [Equation]

Original: l ) , the f -RR mechanism satisﬁes (cid:15) X -LDP: 
Revised: , the f -RR mechanism satisﬁes (cid:15) [Equation] -LDP: P

Revised: [Equation]

Original:  P ( f -RR
Revised: ) P [Equation] -RR

Original: [Equation]
Revised: [Equation]

Original: v x )= z ) P ( f -RR ( (cid:101) v x )= z ) < (cid:15) X , where v x and (cid:101) v x can be different at any bit, and z ∈ Range ( f -RR ) .
Revised: x )= z ) < (cid:15) X , where v x and (cid:101) v x can be different at any bit, and z ∈ Range ( f -RR ) .

Original: Regarding to the ground-truth label y x , given a privacy budget (cid:15) Y , we show that there always exists an upper bound of β so that y (cid:48) x preserves (cid:15) Y -LDP given y x .
Revised: Regarding to the ground-truth label y x , given a privacy budget (cid:15) Y , we show that there always exists an upper bound of β so that y (cid:48) x preserves (cid:15) Y -LDP given y x .

Original: Theorem 3.
Revised: Theorem 3.

Original: ∀ β : β ≤ (cid:15)
Revised: ∀ β : β ≤ (cid:15)

Original: Y − ln
Revised: Y − ln

Original: ( C − 1) , the label -RR mechanism (Alg. 1, line 18) satisﬁes (cid:15) Y -LDP: P ( label -RR ( y x )=
Revised: ( C − 1) , the label -RR mechanism (Alg. 1, line 18) satisﬁes (cid:15) Y -LDP: P ( label -RR ( y x )=

Original: z | y x )
Revised: z | y x )

Original: P ( label -RR ( (cid:101) y x )=
Revised: P ( label -RR ( (cid:101) y x )=

Original: z | (cid:101) y x ) ≤ exp( (cid:15) Y ) , given any distinct labels y x
Revised: z | (cid:101) y x ) ≤ exp( (cid:15) Y ) , given any distinct labels y x

Original: and (cid:101) y x , and z ∈ Range ( label -RR ) .
Revised: and (cid:101) y x , and z ∈ Range ( label -RR ) .

Original: From Theorems 2 and 3, our BitRand preserves (cid:15) X -LDP for the vector v x and (cid:15) Y -LDP for label y x .
Revised: From Theorems 2 and 3, our BitRand preserves (cid:15) X -LDP for the vector v x and (cid:15) Y -LDP for label y x .

Original: The gradients (cid:53) u θ t preserve ( (cid:15) X , (cid:15) Y ) -LDP in any training rounds t for any training samples ( x, y x ) ∈ D u and for all clients u ∈ [1 , N ] ; since, ∀ u, t : (cid:53) uθ t are computed from the randomized training samples { ( v (cid:48)
Revised: The gradients (cid:53) uθ t preserve ( (cid:15) X , (cid:15) Y ) -LDP in any training rounds t for any training samples ( x, y x ) ∈ D u and for all clients u ∈ [1 , N ] ; since, ∀ u, t : (cid:53) uθ t are computed from the randomized training samples { ( v (cid:48)

Original: x , y (cid:48) x ) } n u without accessing any further information from { ( x, y x ) } n u .
Revised: x , y (cid:48) x ) } n u without accessing any further information from { ( x, y x ) } n u .

Original:  P RIVACY AND U TILITY T RADE - OFF
Revised:  P RIVACY AND U TILITY T RADE - OFF

Original: As shown in Theorem 1, we can derive a tighter privacy loss bound.
Revised: As shown in Theorem 1, we can derive a tighter privacy loss bound.

Original: Therefore, we focus on understanding how BitRand can address the privacy-utility tradeoff in comparison with existing approaches by theoretically studying: (1) the utility of f -RR regarding the expected error bound ( Theorem 4 ), and (2) the trade-off between privacy budget and randomization probabilities.
Revised: Therefore, we focus on understanding how BitRand can address the privacy-utility tradeoff in comparison with existing approaches by theoretically studying: (1) the utility of f -RR regarding the expected error bound ( Theorem 4 ), and (2) the trade-off between privacy budget and randomization probabilities.

Original: All statistical tests are 2-tail t-tests.
Revised: All statistical tests are 2-tail t-tests.

Original: Expected Error Bounds.
Revised: Expected Error Bounds.

Original: We analyze the data utility through expected error , denoted as ξ a , measuring the expected change of each embedded feature a after applying f -RR: ξ a = E |E ( f -RR ( v a ))
Revised: We analyze the data utility through expected error , denoted as ξ a , measuring the expected change of each embedded feature a after applying f -RR: ξ a = E |E ( f -RR ( v a ))

Original: − E ( v a ) | .
Revised: − E ( v a ) | .

Original: The smaller expected error is, the better data utility the randomization mechanism achieves.
Revised: The smaller expected error is, the better data utility the randomization mechanism achieves.

Original: The expected error ξ a is bounded by (cid:80) i ∈ [0 ,l − 1] q Xi × ∆ i .
Revised: The expected error ξ a is bounded by (cid:80) i ∈ [0 ,l − 1] q Xi × ∆ i .

Original: Theorem 4.
Revised: Theorem 4.

Original: The f -RR expected error bound is quantiﬁed by ξ a = E |E ( f -RR ( v a ))
Revised: The f -RR expected error bound is quantiﬁed by ξ a = E |E ( f -RR ( v a ))

Original: − E ( v a ) | = (cid:80) i ∈
Revised: − E ( v a ) | = (cid:80) i ∈

Original: [0 ,l − 1] q Xi × ∆ i .
Revised: [0 ,l − 1] q Xi × ∆ i .

Original: Theorem 4 can be directly applied to quantify the expected error bounds of f -RR without the bitaware term i % l/l , corrected LATENT (Appendix H), and corrected OME (Appendix I).
Revised: Theorem 4 can be directly applied to quantify the expected error bounds of f -RR without the bitaware term i % l/l , corrected LATENT (Appendix I), and corrected OME (Appendix J).

Original: Regarding existing mechanisms applied on the embedded feature a , including Duchi mechanism (DM) (Duchi & Rogers, 2019), Piecewise mechanism (PM) (Wang et al., 2019a), Hybrid mechanism (HM) (Wang et al., 2019a), Suboptimal mechanism (PM-SUB) (Zhao et al., 2020b), Gaussian and Laplace (Dwork & Roth, 2014), we derive a general form of expected error bounds ξ a for these mechanisms as:
Revised: Regarding existing mechanisms applied on the embedded feature a , including Duchi mechanism (DM) (Duchi et al., 2013), Piecewise mechanism (PM) (Wang et al., 2019a), Hybrid mechanism (HM) (Wang et al., 2019a), Suboptimal mechanism (PM-SUB) (Zhao et al., 2020b), Gaussian and Laplace (Dwork & Roth, 2014), we derive a general form of expected error bounds ξ a for these mechanisms as:

Original: [Equation]
Revised: [Equation]

Original:  where M is an (cid:15)
Revised: | where M is an (cid:15)

Original: [Equation]
Revised: [Equation]

Original: -LDP preserving mechanism,
Revised: -LDP preserving mechanism,

Original: [Equation]
Revised: [Equation]

Original: Figure 3 illustrates the expected error bound of each algorithm as a function of (cid:15) X .
Revised: Figure 3 illustrates the expected error bound of each algorithm as a function of (cid:15) X .

Original: It is obvious that f -RR has signiﬁcantly tighter expected error bounds compared with baseline approaches under a wide range of privacy budgets given reasonable large numbers of embedded features and encoding bits ( p = 0 .
Revised: It is obvious that f -RR has signiﬁcantly tighter expected error bounds compared with baseline approaches under a wide range of privacy budgets given reasonable large numbers of embedded features and encoding bits ( p = 0 .

Original: 02 ); thus indicating that our mechanism achieves better data utility compared with the baselines.
Revised: 02 ); thus indicating that our mechanism achieves better data utility compared with the baselines.

Original: The improvement of our f -RR over existing mechanisms is larger when (cid:15) X increases.
Revised: The improvement of our f -RR over existing mechanisms is larger when (cid:15) X increases.

Original: To profoundly study the data utility, we take a depth look into bit-level analysis by relaxing Theorem 4 into (1) an expected error bound ξ i =
Revised: To profoundly study the data utility, we take a depth look into bit-level analysis by relaxing Theorem 4 into (1) an expected error bound ξ i =

Original: q Xi × ∆ i for each bit i ∈ [0 , l − 1] , and (2) an average top- k expected error bound ξ top − k = 1 /k (cid:80) ki =

Revised: q Xi × ∆ i for each bit i ∈ [0 , l − 1] , and (2) an average top- k expected error bound ξ top − k = 1 /k (cid:80) ki =

Original: ξ i , ∀ k ∈ [0 , l − 1] .
Revised: ξ i , ∀ k ∈ [0 , l − 1] .

Original: Figure 10 (Appendix J) show that, at the bit-level, f -RR achieves smaller values of ξ i for most important bits, especially the sign bit and integer bits, and comparable ξ i for least important bits, i.e., fraction bits.
Revised: Figure 11 (Appendix K) show that, at the bit-level, f -RR achieves smaller values of ξ i for most important bits, especially the sign bit and integer bits, and comparable ξ i for least important bits, i.e., fraction bits.

Original: The gap between f -RR and the baselines are signiﬁcantly larger given the ξ top − k .
Revised: The gap between f -RR and the baselines are signiﬁcantly larger given the ξ top − k .

Original: We obtain smaller values of ξ top − k for all k ∈ [0 , l − 1] , under a tight privacy budget (cid:15) X = 0 .
Revised: We obtain smaller values of ξ top − k for all k ∈ [0 , l − 1] , under a tight privacy budget (cid:15) X = 0 .

Original: Importantly, when increasing privacy budget (cid:15) X (Figures 10b,c), the gap between f -RR and the baselines is larger.
Revised: Importantly, when increasing privacy budget (cid:15) X (Figures 11b,c), the gap between f -RR and the baselines is larger.

Original: In fact, the expected error bounds in f -RR is reduced while the expected error bounds in the baselines are remained the same.
Revised: In fact, the expected error bounds in f -RR is reduced while the expected error bounds in the baselines are remained the same.

Original: Privacy budget and randomization probabilities trade-off.
Revised: Privacy budget and randomization probabilities trade-off.

Original: Compared with existing RR mechanisms (Lyu et al., 2020a; Arachchige et al., 2019; Sun et al., 2021;
Revised: Compared with existing RR mechanisms (Lyu et al., 2020a; Arachchige et al., 2019; Sun et al., 2021;

Original: Zhao et al., 2020b; Duchi &
Revised: Zhao et al., 2020b; Duchi &

Original: Rogers, 2019; Wang et al., 2019a; Liu et al., 2020), in our algorithm, bits with a stronger inﬂuence on the model utility, e.g., sign bits and integer bits, have smaller randomization probabilities q X , and vice-versa.
Revised: Rogers, 2019; Wang et al., 2019a; Liu et al., 2020), in our algorithm, bits with a stronger inﬂuence on the model utility, e.g., sign bits and integer bits, have smaller randomization probabilities q X , and vice-versa.

Original: This is because we consider the inﬂuence of each bit i through the term i % l l in modeling the randomization probabilities q X (and p X ).
Revised: This is because we consider the inﬂuence of each bit i through the term i % l l in modeling the randomization probabilities q X (and p X ).

Original: This unique property of our algorithm enables us to better optimize the trade-off between privacy loss and model utility. 
Revised: This unique property of our algorithm enables us to better optimize the trade-off between privacy loss and model utility. [Figure] [Equation]

Original:  To demonstrate this, we examine the behavior of q X across binary encoding bits under a wide range of (cid:15) X ∈ { 0 . 1 , 1 , 2 } , given reasonable values of r and l , i.e., r = 1 , 000 and l = 10 (Figures 4 and 11, Appendix J).
Revised: , 000 , and (cid:15) X = 1 . To demonstrate this, we examine the behavior of q X across binary encoding bits under a wide range of (cid:15) X ∈ { 0 . 1 , 1 , 2 } , given reasonable values of r and l , i.e., r = 1 , 000 and l = 10 (Figures 4 and 12, Appendix K).

Original: Our mechanism achieves a smaller randomization probability q X than the corrected LATENT in all cases ( p = 3 .
Revised: Our mechanism achieves a smaller randomization probability q X than the corrected LATENT in all cases ( p = 3 .

Original: 7 e − 9 ), especially more signiﬁcant bits, such as sign bits and integer bits.
Revised: 7 e − 9 ), especially more signiﬁcant bits, such as sign bits and integer bits.

Original: The gap is more prominent when (cid:15) X increases.
Revised: The gap is more prominent when (cid:15) X increases.

Original: This observation is less obvious when comparing our mechanism with the corrected OME, given an uneven randomization probability q X across bits in the corrected OME.
Revised: This observation is less obvious when comparing our mechanism with the corrected OME, given an uneven randomization probability q X across bits in the corrected OME.

Original: To better show the comparison, we draw an average top- k measure curve: ∀ k ∈ [0 , l − 1] : q top - k = 1 /k (cid:80) ki =

Revised: To better show the comparison, we draw an average top- k measure curve: ∀ k ∈ [0 , l − 1] : q top - k = 1 /k (cid:80) ki =

Original: q i , where q i is the bit
Revised: q i , where q i is the bit

Original: i ’s q X , to evaluate the average q X across bits.
Revised: i ’s q X , to evaluate the average q X across bits.

Original: The smaller q top - k is, the better the randomization probability q X is.
Revised: The smaller q top - k is, the better the randomization probability q X is.

Original: Given a tight privacy budget (cid:15) X = 0 .
Revised: Given a tight privacy budget (cid:15) X = 0 .

Original: 1 , our mechanism and the corrected OME have a similar q top - k .
Revised: 1 , our mechanism and the corrected OME have a similar q top - k .

Original: However, when (cid:15) X increases, i.e., (cid:15) X ∈ { 1 , 2 } , our mechanism achieves signiﬁcantly smaller values of q top - k than the corrected OME ( p = 5 .
Revised: However, when (cid:15) X increases, i.e., (cid:15) X ∈ { 1 , 2 } , our mechanism achieves signiﬁcantly smaller values of q top - k than the corrected OME ( p = 5 .

Original: 1 e − 3 ).
Revised: 1 e − 3 ).

Original:  D IMENSION -E LASTIC A NALYSIS
Revised:  D IMENSION -E LASTIC A NALYSIS

Original: In existing RR mechanisms, the curse of privacy composition is rooted in the privacy composition across bits l and features r (Duchi et al., 2013; Lyu et al., 2020a;
Revised: In existing RR mechanisms, the curse of privacy composition is rooted in the privacy composition across bits l and features r (Duchi et al., 2013; Lyu et al., 2020a;

Original: Arachchige et al., 2019), gradients (cid:53) uθ t (Zhao et al., 2020b; Wang et al., 2019a), and training rounds T (Zhao et al., 2020b; Wang et al., 2019a).
Revised: Arachchige et al., 2019), gradients (cid:53) u θ t (Zhao et al., 2020b; Wang et al., 2019a), and training rounds T (Zhao et al., 2020b; Wang et al., 2019a).

Original: Thus, increasing the dimensions of l , r , (cid:53) uθ t , and T either signiﬁcantly increases the privacy budget (i.e., resulting in loose privacy protection) (Wang et al., 2019a; 2017) or notably affects the randomization probabilities (i.e., the value transmitted correctly through the randomization becomes smaller resulting in poor utility (Wang et al., 2016a)) (Lyu et 
Revised: Thus, increasing the dimensions of l , r , (cid:53) uθ t , and T either signiﬁcantly increases the privacy budget (i.e., resulting in loose privacy protection) (Wang et al., 2019a; 2017) or notably affects the randomization probabilities (i.e., the value transmitted correctly through the randomization becomes smaller resulting in poor utility (Wang et al., 2016a)) (Lyu et al., 2020a; Arachchige et al., 2019).

Original: In Theorem 1, the privacy composition across bits and features is unavoidable.
Revised: In Theorem 1, the privacy composition across bits and features is unavoidable.

Original: However,in our BitRand, the values of embedded features transmitted correctly through our randomization mechanism is minimally affected when increasing the dimensions of r , l , (cid:53) uθ t , T , and C under the same (cid:15) X and (cid:15) Y .
Revised: However, the values of embedded features transmitted correctly through our randomization mechanism is minimally affected when increasing the dimensions of r , l , (cid:53) uθ t , T , and C under the same (cid:15) X and (cid:15) Y .

Original: That enable us to evade the curse of privacy composition when working with complex models and tasks under rigorous LDP protection.
Revised: That enable us to evade the curse of privacy composition when working with complex models and tasks under rigorous LDP protection.

Original: To shed light into this property, we conduct a theoretical analysis to examine: (1) How the dimensions of r , l , gradients (cid:53) uθ t , and training rounds T impact the privacy budget (cid:15) X and the randomization probabilities q X and p X (= 1 − q X ) ; and (2) How the number of model outcomes C impacts the privacy budget (cid:15) Y and the randomization probabilities q Y and p Y .
Revised: To shed light into this property, we conduct a theoretical analysis to examine: (1) How the dimensions of r , l , gradients (cid:53) uθ t , and training rounds T impact the privacy budget (cid:15) X and the randomization probabilities q X and p X (= 1 − q X ) ; and (2) How the number of model outcomes C impacts the privacy budget (cid:15) Y and the randomization probabilities q Y and p Y .

Original: We select the upper bounds of α = (cid:113) ( (cid:15) X + rl ) / (2 r (cid:80)l −
Revised: We select the upper bounds of α = (cid:113) ( (cid:15) X + rl ) / (2 r (cid:80)

Original: [Figure]
Revised: [Figure]

Original: =0 exp(2 (cid:15) X il )) and β = (cid:15)
Revised: = [Equation] exp(2 (cid:15) X il )) and β = (cid:15)

Original: Y − ln
Revised: Y − ln

Original: ( C − 1)
Revised: ( C − 1)

Original: (Theorems 2, 3) in our analysis.
Revised: (Theorems 2, 3) in our analysis.

Original: Dimensions of gradients (cid:53) uθ t and training rounds T .
Revised: Dimensions of gradients (cid:53) uθ t and training rounds T .

Original: In BitRand, the clients only use the perturbed samples ( v (cid:48)
Revised: In BitRand, the clients only use the perturbed samples ( v (cid:48)

Original: x , y (cid:48) x ) to train their local models without accessing any further information from ( x, y x ) .
Revised: x , y (cid:48) x ) to train their local models without accessing any further information from ( x, y x ) .

Original: Thus, the privacy budgets (cid:15) X and (cid:15) Y are independent of the dimension of gradients (cid:53) uθ t and the training rounds T , i.e., following the post-processing property in DP (Dwork & Roth, 2014).
Revised: Thus, the privacy budgets (cid:15) X and (cid:15) Y are independent of the dimension of gradients (cid:53) uθ t and the training rounds T , i.e., following the post-processing property in DP (Dwork & Roth, 2014).

Original: Also, the size of (cid:53) uθ t and T do not affect the randomization probabilities, since (cid:53) uθ t and T are not used to model q X , p X , q Y , and p Y as in Eqs.
Revised: Also, the size of (cid:53) uθ t and T do not affect the randomization probabilities, since (cid:53) uθ t and T are not used to model q X , p X , q Y , and p Y as in Eqs.

Original: 2 and 3.
Revised: 2 and 3.

Original: Dimensions of embedded features r and encoding bits l .
Revised: Dimensions of embedded features r and encoding bits l .

Original: Varying the dimensions of r and l does not affect the privacy budget (cid:15) X ∈ R + , since there always exists an α for Theorem 2 to hold.
Revised: Varying the dimensions of r and l does not affect the privacy budget (cid:15) X ∈ R + , since there always exists an α for Theoremto hold.

Original: However, it is necessary tocomprehensively understand the inﬂuence of varying r and l on the privacy-utility trade-off.
Revised: However, it is necessary to understand the inﬂuence of varying r and l on the privacy-utility trade-off.

Original: To theoretically analyze theinﬂuence, we turn our focus on understanding the impacts of r and l on the randomization probabilities q X and p X , given ﬁxed values of (cid:15) X .
Revised: We theoretically analyze the impacts of r and l on the randomization probabilities q X and p X , given ﬁxed values of (cid:15) X .

Original: A model is expected to achieve higher model utility given smaller values of q X (higher values of p X ) under the same privacy budget.
Revised: A model is expected to achieve higher model utility given smaller values of q X (higher values of p X ) under the same privacy budget.

Original: We examine q X and p X in the following experiments: (i) Fixing l , then varying r ; and (ii) Fixing r , then varying l .
Revised: We examine q X and p X in the following experiments: (i) Fixing l , then varying r ; and (ii) Fixing r , then varying l .

Original: Figures 5 and 12 (Appendix J) illustrate q X as a function of r , under l ∈ { 5 , 20 , 100 , 1 , 000 } and (cid:15) X ∈ { 0 . 1 , 1 , 2 } .
Revised: Figures 5 and 13 (Appendix K) illustrate q X as a function of r , under l ∈ { 5 , 20 , 100 , 1 , 000 } and (cid:15) X ∈ { 0 . 1 , 1 , 2 } .

Original: Varying r does not affect the randomization probability q X (and p X ) for all the bits in v x .
Revised: Varying r does not affect the randomization probability q X (and p X ) for all the bits in v x .

Original: To explain this, we take a deeper look into the α ’s bound, which is the only factor affecting q X and p X .
Revised: To explain this, we take a deeper look into the α ’s bound, which is the only factor affecting q X and p X .

Original: Given ﬁxed (cid:15) X and l , (cid:80) l − 1 i =0 exp(2 (cid:15) X il ) is a constant, denoted C .
Revised: Given ﬁxed (cid:15) X and l , (cid:80) l − 1 i =0 exp(2 (cid:15) X il ) is a constant, denoted C .

Original: So, we have α = 
Revised: So, we have α = (cid:112) ( (cid:15) X + rl ) / (2 r C ) =

Original: [Equation]
Revised: (cid:112) ( (cid:15)

Original:  , since (cid:15) X r (cid:117) 0 . 0{+
+} in practice.
Revised: X r + l ) / (2 C ) (cid:117) (cid:112) l/ (2 C ) , since (cid:15) X r (cid:117) 0 . {+
+} in practice.

Original: Thus, with ﬁxed (cid:15) X and l , q X and p X are r -elastic, given α approximately is a constant (cid:112) l/ (2 C ) .
Revised: Thus, with ﬁxed (cid:15) X and l , q X and p X are r -elastic, given α approximately is a constant (cid:112) l/ (2 C ) .

Original: Now, we ﬁx r = 1 , 000 , which is a decent number of embedded features, and show q X as a function of l under a wide range of r ∈ [10 , 10 , 000] and (cid:15) X ∈ { 0 . 1 , 1 , 2 } (Figures 6 and 13, Appendix J).
Revised: Now, we ﬁx r = 1 , 000 , which is a decent number of embedded features, and show q X as a function of l under a wide range of r ∈ [10 , 10 , 000] and (cid:15) X ∈ { 0 . 1 , 1 , 2 } (Figures 6 and 14, Appendix K).

Original: Given a tight privacy budget (cid:15) X = 0 . 1 , varying l does not affect the randomization probability q X , i.e., m = (cid:98) l/ 2 (cid:99) in our analysis covering most values of embedded features in practice.
Revised: Given a tight privacy budget (cid:15) X = 0 . 1 , varying l does not affect the randomization probability q X , i.e., m = (cid:98) l/ 2 (cid:99) in our analysis covering most values of embedded features in practice.

Original: However, with higher values of (cid:15) X ∈ { 1 . 0 , 2 . 0 } , using more encoding bits l lowers q X for most important bits, i.e., sign and integer bits; while increasing q X for least important bits, i.e., fraction bits.
Revised: However, with higher values of (cid:15) X ∈ { 1 . 0 , 2 . 0 } , using more encoding bits l lowers q X for most important bits, i.e., sign and integer bits; while increasing q X for least important bits, i.e., fraction bits.

Original: When l is large enough ( l ≥ 20 ), the randomization probability q X is l -elastic since the impact of l becomes marginal to all the bits.
Revised: When l is large enough ( l ≥ 20 ), the randomization probability q X is l -elastic since the impact of l becomes marginal to all the bits.

Original: This is also true for p X .
Revised: This is also true for p X .

Original: Number of model outcomes C .
Revised: Number of model outcomes C .

Original: Similar to (cid:15) X , from Eq.
Revised: Similar to (cid:15) X , from Eq.

Original: 3 and Theorem 3, the privacy budget (cid:15) Y is not directly affected by the number of model outcomes C , since ∀ C, (cid:15) X : ∃ β for Theorem 3 to hold.
Revised: 3 and Theorem 3, the privacy budget (cid:15) Y is not directly affected by the number of model outcomes C , since ∀ C, (cid:15) X : ∃ β for Theorem 3 to hold.

Original: However,the number of model outcomes C may impact the trade-off between privacy and model utility by affecting the randomization probabilities q Y and p Y .
Revised: However, C may impact the trade-off between privacy and model utility by affecting the randomization probabilities q Y and p Y .

Original: Figure 7 shows that, when C is sufﬁciently large, i.e., C ≥ 100 , the randomization probability q Y is C -elastic since the impact of C on q Y is marginal given a wide range of the privacy budget (cid:15) Y .
Revised: Figure 7 shows that, when C is sufﬁciently large, i.e., C ≥ 100 , the randomization probability q Y is C -elastic since the impact of C on q Y is marginal given a wide range of the privacy budget (cid:15) Y .

Original: This is also true for p Y .
Revised: This is also true for p Y .

Original: When (cid:15) Y increases, the randomization probability q Y becomes smaller ( p Y becomes larger).
Revised: When (cid:15) Y increases, the randomization probability q Y becomes smaller ( p Y becomes larger).

Original: This is a reasonable observation.
Revised: This is a reasonable observation.

Original: Thanks to the bit-aware and dimension-elastic properties, our BitRand can achieve high data utility, especially under tight privacy budgets and expected error bounds.
Revised: Thanks to the bit-aware and dimension-elastic properties, our BitRand can achieve high data utility, especially under tight privacy budgets and expected error bounds.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We have conducted an extensive experiment on benchmark datasets under two fundamental FL tasks, text and image classiﬁcation, to shed light on understanding 1) the interplay among privacy budget and model utility in BitRand, 2) the effectiveness of the dimension-elastic and bit-aware properties on model utility, and 3) different settings of applying RR to preserve LDP.
Revised: We have conducted an extensive experiment on benchmark datasets under two fundamental FL tasks, text and image classiﬁcation, to shed light on understanding 1) the interplay among privacy budget and model utility in BitRand, 2) the effectiveness of the dimension-elastic and bit-aware properties on model utility, and 3) different settings of applying RR to preserve LDP.

Original: Baseline Approaches.
Revised: Baseline Approaches.

Original: We consider a variety of LDP-preserving mechanisms as baseline approaches: (1) Binary encoding approaches, including the corrected LATENT (Arachchige et al., 2019) and the corrected OME (Lyu et al., 2020a)(Appendices H and I); (2) LDP-FL
Revised: We consider a variety of LDP-preserving mechanisms as baseline approaches: (1) Binary encoding approaches, including the corrected LATENT (Arachchige et al., 2019) and the corrected OME (Lyu et al., 2020a)

Original:  (Sun et al., 2021); (3) Duchi mechanism ( DM ) (Duchi & Rogers, 2019); (4) Piecewise mechanism ( PM ) (Wang et al., 2019a); (5) Hybrid mechanism ( HM ) (Wang et al., 2019a); (6) Three-Outputs mechanism (Zhao et al., 2020b); (7) Suboptimal mechanism ( PM-SUB ) (Zhao et al., 2020b); and (8) Label-Laplace (Phan et al., 2020).
Revised: (Appendices I and J); (2) LDP-FL (Sun et al., 2021); (3) Duchi mechanism ( DM ) (Duchi et al., 2013); (4) Piecewise mechanism ( PM ) (Wang et al., 2019a); (5) Hybrid mechanism ( HM ) (Wang et al., 2019a); (6) Three-Outputs mechanism (Zhao et al., 2020b); (7) Suboptimal mechanism ( PM-SUB ) (Zhao et al., 2020b); and (8) Label-Laplace (Phan et al., 2020).

Original: Each baseline approach is applied to randomize (when applicable): (i)
Revised: Each baseline approach is applied to randomize (when applicable): (i)

Original: Embedded features e x ; (ii) Gradients (cid:53) uθ t ; and (iii) Gradients (cid:53)
Revised: Embedded features e x ; (ii) Gradients (cid:53) uθ t ; and (iii) Gradients (cid:53)

Original: uθ t with a recent anonymizer (Sun et al., 2021) to reduce the privacy budget consumption.
Revised: uθ t with a recent anonymizer (Sun et al., 2021) to reduce the privacy budget consumption.

Original: In addition, Label-Laplace is used as a baseline to protect the ground-truth labels y x .
Revised: In addition, Label-Laplace is used as a baseline to protect the ground-truth labels y x .

Revised: Note that, in our experiment, we use the upper bound of β resulting in the same randomizing probabilities as in LabelDP.

Original:  These settings are widely accepted to preserve LDP in FL; thus, offering a comprehensive view of preserving LDP in FL.
Revised: Therefore, we do not include LabelDP in comparison. More details about the difference of label -RR and LabelDP are in Appendix G. These settings are widely accepted to preserve LDP in FL; thus, offering a comprehensive view of preserving LDP in FL.

Original: Note that LATENT, OME, and BitRand can only be applied on embedded features e x ; while LDP-FL can only be applied on gradients (cid:53) uθ t with and without the anonymizer (Sun et al., 2021).
Revised: Note that LATENT, OME, and BitRand can only be applied on embedded features e x ; while LDP-FL can only be applied on gradients (cid:53) uθ t with and without the anonymizer (Sun et al., 2021).

Original: We include the Noiseless FL model trained on the original data D u to show upper-bounds and a Random guess model to understanding model utility better.
Revised: We include the Noiseless FL model trained on the original data D u to show upper-bounds and a Random guess model to understanding model utility better.

Original: Datasets, Metrics, and Model Conﬁguration.
Revised: Datasets, Metrics, and Model Conﬁguration.

Original: The complete details of the datasets, metrics, and model conﬁguration are in Appendix K. We carried out our experiments on two textual datasets and two image datasets, including the AG dataset (Gulli et al., 2012), our collected Security and Exchange
Revised: The complete details of the datasets, metrics, and model conﬁguration are in Appendix L. We carried out our experiments on two textual datasets and two image datasets, including the AG dataset (Gulli et al., 2012), our collected Security and Exchange

Original: Commission (SEC) ﬁnancial contract dataset, the large-scale celebFaces attributes (CelebA) dataset (Liu et al., 2015), and the Federated Extended MNIST (FEMNIST) dataset (Caldas et al., 2018).
Revised: Commission (SEC) ﬁnancial contract dataset, the large-scale celebFaces attributes (CelebA) dataset (Liu et al., 2015), and the Federated Extended MNIST (FEMNIST) dataset (Caldas et al., 2018).

Original: We use the test accuracy and the test area under the curve (AUC) as evaluation metrics.
Revised: We use the test accuracy and the test area under the curve (AUC) as evaluation metrics.

Original: Models with higher values of test accuracy and AUC are better.
Revised: Models with higher values of test accuracy and AUC are better.

Original: We use the BERT-Base (Uncased) pre-trained model (ber; Devlin et al., 2018) to extract embedded features in the AG and SEC datasets.
Revised: We use the BERT-Base (Uncased) pre-trained model (ber; Devlin et al., 2018) to extract embedded features in the AG and SEC datasets.

Original: In the
Revised: In the

Original: CelebA and FEMNIST datasets, we use the ResNet-18 pre-trained model (img; He et al., 2016).
Revised: CelebA and FEMNIST datasets, we use the ResNet-18 pre-trained model (img; He et al., 2016).

Original: For text and image classiﬁcation tasks, we use two fully connected layers on top of embedded features, each of which consists of 1 , 500 hidden neurons and uses a ReLU activation function.
Revised: For text and image classiﬁcation tasks, we use two fully connected layers on top of embedded features, each of which consists of 1 , 500 hidden neurons and uses a ReLU activation function.

Original: Evaluation Results.
Revised: Evaluation Results.

Original: Comprehensive results show that BitRand offers stronger privacy protection with higher model utility, compared with all baseline approaches, as discussed next.
Revised: Comprehensive results show that BitRand offers stronger privacy protection with higher model utility, compared with all baseline approaches, as discussed next.

Original: LDP-preserving approaches applied on the embedded features e x .
Revised: LDP-preserving approaches applied on the embedded features e x .

Original: Baseline approaches do not work well when they are applied on embedded features e x
Revised: Baseline approaches do not work well when they are applied on embedded features e x

Original: (Figures 14 and 18, Table 2, Appendix K).
Revised: (Figures 15 and 19, Table 3, Appendix L).

Original: In SEC, AG, and FEMNIST datasets, BitRand achieves the highest model utility compared with the best baseline approach, which is the corrected OME, under a tight privacy budget (cid:15) X = 1 .
Revised: In SEC, AG, and FEMNIST datasets, BitRand achieves the highest model utility compared with the best baseline approach, which is the corrected OME, under a tight privacy budget (cid:15) X = 1 .

Original: In terms of accuracy and AUC values, BitRand ( (cid:15) Y = ∞ ) achieves an improvement of 46 .
Revised: In terms of accuracy and AUC values, BitRand ( (cid:15) Y = ∞ ) achieves an improvement of 46 .

Original: 03% and 38 .
Revised: 03% and 38 .

Original: 51% in the AG dataset ( p = 2 .
Revised: 51% in the AG dataset ( p = 2 .

Original: 7 e − 22 ), 13 .
Revised: 7 e − 22 ), 13 .

Original: 69% and 13 .
Revised: 69% and 13 .

Original: 79% in the SEC dataset ( p = 4 .
Revised: 79% in the SEC dataset ( p = 4 .

Original: 1 e − 12 ), and 21 .
Revised: 1 e − 12 ), and 21 .

Original: 62% and 13 .
Revised: 62% and 13 .

Original: 42% in the FEMNIST dataset ( p = 5 .
Revised: 42% in the FEMNIST dataset ( p = 5 .

Original: 6 e − 11 ) respectively.
Revised: 6 e − 11 ) respectively.

Original: In the CelebA dataset (Table 2, Appendix K), BitRand outperforms the best baseline approach, i.e., PM-SUB, with an average improvement of 1 . 66% across all 40 attributes in terms of AUC measure ( p = 1 .
Revised: In the CelebA dataset (Table 3, Appendix L), BitRand outperforms the best baseline approach, i.e., PM-SUB, with an average improvement of 1 . 66% across all 40 attributes in terms of AUC measure ( p = 1 .

Original: 2 e − 2 ).
Revised: 2 e − 2 ).

Original: Since the
Revised: Since the

Original: CelebA dataset is highly imbalanced, we use the AUC measure instead of the model accuracy.
Revised: CelebA dataset is highly imbalanced, we use the AUC measure instead of the model accuracy.

Original: The gaps between BitRand and the baseline approaches are signiﬁcantly wider when the privacy budget (cid:15) X is larger.
Revised: The gaps between BitRand and the baseline approaches are signiﬁcantly wider when the privacy budget (cid:15) X is larger.

Original: In addition to (cid:15) X = 1 , with a tight privacy budget for the class labels (cid:15) Y ∈ { 1 , 2 . 5 } ,
Revised: In addition to (cid:15) X = 1 , with a tight privacy budget for the class labels (cid:15) Y ∈ { 1 , 2 . 5 } ,

Original: BitRand still outperforms baseline approaches in most of cases, offering stronger privacy protection with better model utility, i.e., LDP at both embedded feature and label levels instead of only LDP on the embedded features as in baseline approaches.
Revised: BitRand still outperforms baseline approaches in most of cases, offering stronger privacy protection with better model utility, i.e., LDP at both embedded feature and label levels instead of only LDP on the embedded features as in baseline approaches.

Original: The key reason is that, in the baseline approaches, the model utility is signiﬁcantly affected by the size of the embedded features.
Revised: The key reason is that, in the baseline approaches, the model utility is signiﬁcantly affected by the size of the embedded features.

Original: Thanks to the dimension-elastic and bit-aware properties, BitRand can achieve high model accuracy and AUC values, especially under tight privacy budgets.
Revised: Thanks to the dimension-elastic and bit-aware properties, BitRand can achieve high model accuracy and AUC values, especially under tight privacy budgets.

Original: In addition,
Revised: In addition,

Original: BitRand achieves the highest improvement in the AG dataset, since it is a balanced dataset compared with the highly imbalanced CelebA dataset, in which BitRand achieves the least improvement.
Revised: BitRand achieves the highest improvement in the AG dataset, since it is a balanced dataset compared with the highly imbalanced CelebA dataset, in which BitRand achieves the least improvement.

Original: Addressing imbalanced data in FL under privacy protection is interesting (Huang et al., 2020), but it is out-of-scope of our study.
Revised: Addressing imbalanced data in FL under DP (Huang et al., 2020) is out-of-scope of our study.

Original: LDP-preserving approaches applied on on gradients (cid:53) uθ t without and with the anonymizer (Sun et al., 2021).
Revised: LDP-preserving approaches applied on on gradients (cid:53) uθ t without and with the anonymizer (Sun et al., 2021).

Original: We observe the same phenomenon when baseline approaches are applied on gradients (cid:53) uθ t without and with the anonymizer (Sun et al., 2021), even though the gaps between
Revised: We observe the same phenomenon when baseline approaches are applied on gradients (cid:53) uθ t without and with the anonymizer (Sun et al., 2021), even though the gaps between

Original: BitRand and baseline approaches are (marginally) smaller (Figures 8, 15-16, 19, and Tables3, 4, 
Revised: BitRand and baseline approaches are (marginally) smaller (Figures 8, 16-17, 20, and Tables 4, 5,

Original: Appendix K).
Revised: Appendix L).

Original: Without using the anonymizer in the baseline approaches, in terms of accuracy and
Revised: Without using the anonymizer in the baseline approaches, in terms of accuracy and

Original: AUC, compared with the best baseline approach PM-SUB, BitRand ( (cid:15) X = 1 , (cid:15) Y = 1 ) achieves an improvement of 44 .
Revised: AUC, compared with the best baseline approach PM-SUB, BitRand ( (cid:15) X = 1 , (cid:15) Y = 1 ) achieves an improvement of 44 .

Original: 95% and 37 .
Revised: 95% and 37 .

Original: 52% in the AG dataset ( p = 3 .
Revised: 52% in the AG dataset ( p = 3 .

Original: 9 e − 20 ), 12 .
Revised: 9 e − 20 ), 12 .

Original: 82% and 12 .
Revised: 82% and 12 .

Original: 92% in the
Revised: 92% in the

Original: SEC dataset ( p = 1 .
Revised: SEC dataset ( p = 1 .

Original: 3 e − 11 ), 24 .
Revised: 3 e − 11 ), 24 .

Original: 17% and 13 .
Revised: 17% and 13 .

Original: 40% in the FEMNIST dataset ( p = 4 .
Revised: 40% in the FEMNIST dataset ( p = 4 .

Original: 1 e − 11 ).
Revised: 1 e − 11 ).

Original: When the anonymizer is applied in the baseline approaches, in terms of accuracy and AUC, compared with the best baseline approache HM, our BitRand achieves an improvement of 39 .
Revised: When the anonymizer is applied in the baseline approaches, in terms of accuracy and AUC, compared with the best baseline approache HM, our BitRand achieves an improvement of 39 .

Original: 38% and 32 .
Revised: 38% and 32 .

Original: 75% in the
Revised: 75% in the

Original: AG dataset ( p = 1 .
Revised: AG dataset ( p = 1 .

Original: 2 e − 16 ), 13 .
Revised: 2 e − 16 ), 13 .

Original: 59% and 13 .
Revised: 59% and 13 .

Original: 69% in the SEC dataset ( p = 2 .
Revised: 69% in the SEC dataset ( p = 2 .

Original: 1 e − 10 ), and 23 .
Revised: 1 e − 10 ), and 23 .

Original: 12% and 37 .
Revised: 12% and 37 .

Original: 02% in the FEMNIST dataset ( p = 2 .
Revised: 02% in the FEMNIST dataset ( p = 2 .

Original: 8 e − 11 ).
Revised: 8 e − 11 ).

Original: Regarding the CelebA dataset, BitRand outperforms the best baseline approaches, which are Three-outputs and HM, with improvements of 1 .
Revised: Regarding the CelebA dataset, BitRand outperforms the best baseline approaches, which are Three-outputs and HM, with improvements of 1 .

Original: 28% and 0 .
Revised: 28% and 0 .

Original: 3% , in the cases of with and without the anonymizer, across all 40 attributes in terms of
Revised: 3% , in the cases of with and without the anonymizer, across all 40 attributes in terms of

Original: AUC ( p = 3 .
Revised: AUC ( p = 3 .

Original: 1 e − 2 ).
Revised: 1 e − 2 ).

Original: The model utility in baselines is affected by the size of gradients and the training rounds (when the anonymizer is not applied) and their ﬁnite numbers of randomization outputs of the gradients (Zhao et al., 2020b).
Revised: The model utility in baselines is affected by the size of gradients and the training rounds (when the anonymizer is not applied) and their ﬁnite numbers of randomization outputs of the gradients (Zhao et al., 2020b).

Original: Thanks to the bit-aware and dimension-agnostic properties, BitRand can achieve high model utility under rigorous LDP protection.
Revised: Thanks to the bit-aware and dimension-agnostic properties, BitRand can achieve high model utility under rigorous LDP protection.

Original: LDP-preserving labels.
Revised: LDP-preserving labels.

Original: Figures 17, 20, and Table 2 (Appendix K) present the model utility of
Revised: Figures 18, 21, and Table 3 (Appendix L) present the model utility of

Original: BitRand as a function of the privacy budgets (cid:15) X and (cid:15) Y , in which label -RR is replaced by the
Revised: BitRand as a function of the privacy budgets (cid:15) X and (cid:15) Y , in which label -RR is replaced by the

Original: Label-Laplace, denoted as f -RR & Label-Laplace.
Revised: Label-Laplace, denoted as f -RR & Label-Laplace.

Original: label -RR outperforms the Label-Laplace in all values of (cid:15) X , (cid:15) Y , and datasets.
Revised: label -RR outperforms the Label-Laplace in all values of (cid:15) X , (cid:15) Y , and datasets.

Original: Under rigorous LDP protection (cid:15) Y = 1 given (cid:15) X ∈ [1 , 10] , there is an average improvement of 9 .
Revised: Under rigorous LDP protection (cid:15) Y = 1 given (cid:15) X ∈ [1 , 10] , there is an average improvement of 9 .

Original: 65% accuracy and 7 . 92% AUC in the AG dataset ( 7 . 3 e − 6 ), 7 . 84% accuracy and 7 . 35% AUC in the SEC dataset ( 1 . 

Revised: 65% accuracy and 7 . 92% AUC in the AG dataset ( 7 . 3 e − 6 ), 7 . 84% accuracy and 7 . 35% AUC in the SEC dataset ( 1 .

Original: e − 5 ), and 9 .
Revised: e − 5 ), and 9 .

Original: 91% accuracy and 10 .
Revised: 91% accuracy and 10 .

Original: 55% AUC in the FEMNIST dataset ( p = 9 .
Revised: 55% AUC in the FEMNIST dataset ( p = 9 .

Original: 8 e − 5 ), and 2 . 04% AUC in the CelebA dataset ( p = 1 .
Revised: 8 e − 5 ), and 2 . 04% AUC in the CelebA dataset ( p = 1 .

Original: 6 e − 2 ).
Revised: 6 e − 2 ).

Original: The gaps are delicately smaller in the AG and SEC datasets, and substantially larger in the FEMNIST and
Revised: The gaps are delicately smaller in the AG and SEC datasets, and substantially larger in the FEMNIST and

Original: CelebA datasets when (cid:15) Y is increased.
Revised: CelebA datasets when (cid:15) Y is increased.

Original: Given (cid:15) Y = 2 . 5 , there is an average improvement of 1 . 71% accuracy and 1 .
Revised: Given (cid:15) Y = 2 . 5 , there is an average improvement of 1 . 71% accuracy and 1 .

Original: 37% AUC in the AG dataset ( 8 . 5 e − 2 ), 4 .
Revised: 37% AUC in the AG dataset ( 8 . 5 e − 2 ), 4 .

Original: 65% accuracy and 4 .
Revised: 65% accuracy and 4 .

Original: 73% AUC in the SEC dataset ( 6 . 9 e − 3 ), and 14 .
Revised: 73% AUC in the SEC dataset ( 6 . 9 e − 3 ), and 14 .

Original: 43% accuracy and 12 .
Revised: 43% accuracy and 12 .

Original: 02% AUC in the FEMNIST dataset ( p = 1 .
Revised: 02% AUC in the FEMNIST dataset ( p = 1 .

Original: 9 e − 4 ), and 4 .
Revised: 9 e − 4 ), and 4 .

Original: 24% AUC in the CelebA dataset ( p = 2 .
Revised: 24% AUC in the CelebA dataset ( p = 2 .

Original: 3 e − 2 ).
Revised: 3 e − 2 ).

Original: The reason is that Label-Laplace injects
Revised: The reason is that Label-Laplace injects

Original: Laplace noise across C classes in a label y x causing more noisy model outcomes compared with label -RR, in which only one of the C model outcomes is selected as the result of the randomization.
Revised: Laplace noise across C classes in a label y x causing more noisy model outcomes compared with label -RR, in which only one of the C model outcomes is selected as the result of the randomization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced a bit-aware algorithm, called BitRand, providing rigorous LDP protection to both embedded features and labels in FL via binary encoding.
Revised: In this paper, we introduced a bit-aware algorithm, called BitRand, providing rigorous LDP protection to both embedded features and labels in FL via binary encoding.

Original: In BitRand, the trade-off between the privacy budget consumption and randomization probabilities is dimension-elastic to the numbers of embedded features, encoding bits, gradients, model outcomes, and training rounds, enabling us to work with complex models and FL tasks effectively.
Revised: In BitRand, the trade-off between the privacy budget consumption and randomization probabilities is dimension-elastic to the numbers of embedded features, encoding bits, gradients, model outcomes, and training rounds, enabling us to work with complex models and FL tasks.

Original: We further optimize the randomization probabilities by having smaller randomization probabilities assigned to more critical bits and vice-versa under the same privacy budgets.
Revised: We further optimize the randomization probabilities by having smaller randomization probabilities assigned to more critical bits and vice-versa under the same privacy budgets.

Original: An expected error bound further guarantees that our mechanism can achieve high model utility under rigorous privacy protection.

Original: Theoretical analysis and extensive experimentsconducted on real-world datasets showed that our BitRand outperforms baseline approaches in text and image classiﬁcation.
Revised: Theoretical analysis and extensive experiments showed that our BitRand outperforms baseline approaches in text and image classiﬁcation.

Original: A PPENDIX A R EVISITING R ANDOMIZED R ESPONSE M ECHANISMS FOR LDP
Revised: A PPENDIX A R EVISITING R ANDOMIZED R ESPONSE M ECHANISMS FOR LDP

Original: To preserve LDP given the client’s input x , we can apply existing RR mechanisms (Wang et al., 2016b;
Revised: To preserve LDP given the client’s input x , we can apply existing RR mechanisms (Wang et al., 2016b;

Original: Fanti et al., 2015; Bassily et al., 2017;
Revised: Fanti et al., 2015; Bassily et al., 2017;

Original: Kim et al., 2018; Ren et al., 2018; Zheng et al., 2017;
Revised: Kim et al., 2018; Ren et al., 2018; Zheng et al., 2017;

Original: Wang & Xu, 2019; Zhao et al., 2019; Gursoy et al., 2019; Alvim et al., 2018; Xiong et al., 2019), such as unary encoding-based approaches (Wang et al., 2017; Erlingsson et al., 2014), hash-based approaches (Wang et al., 2017; Bassily & Smith, 2015; Acharya et al., 2019; Wang et al., 2019c), binary encoding-based approaches (Arachchige et al., 2019;
Revised: Wang & Xu, 2019; Zhao et al., 2019; Gursoy et al., 2019; Alvim et al., 2018; Xiong et al., 2019), such as unary encoding-based approaches (Wang et al., 2017; Erlingsson et al., 2014), hash-based approaches (Wang et al., 2017; Bassily & Smith, 2015; Acharya et al., 2019; Wang et al., 2019c), binary encoding-based approaches (Arachchige et al., 2019;

Original: Lyu et al., 2020a), etc.
Revised: Lyu et al., 2020a), etc.

Original: For instance, hash-based approaches such as those of Google RAPPOR (Erlingsson et al., 2014) and OLH (Wang et al., 2017) hash the client’s input x onto a bloom ﬁlter B of size k using h hash functions.
Revised: For instance, hash-based approaches such as those of Google RAPPOR (Erlingsson et al., 2014) and OLH (Wang et al., 2017) hash the client’s input x onto a bloom ﬁlter B of size k using h hash functions.

Original: Then, for each client’s input x and a bit i ∈ B , RAPPOR creates a perturbed binary value B (cid:48)
Revised: Then, for each client’s input x and a bit i ∈ B , RAPPOR creates a perturbed binary value B (cid:48)

Original: i from B i with the following randomization probability:
Revised: i from B i with the following randomization probability:

Original: [Equation]
Revised: [Equation]

Original: where p is a hyper-parameter.
Revised: where p is a hyper-parameter.

Original: This B (cid:48) is reused as the basis for all future analysis, learning, and reports on this distinct input x .
Revised: This B (cid:48) is reused as the basis for all future analysis, learning, and reports on this distinct input x .

Original: This approach achieves (cid:15) X -LDP, where (cid:15)
Revised: This approach achieves (cid:15) X -LDP, where (cid:15)

Original: X = 2 h ln((1 − p 2 ) / p 2 ) , given that the sensitivity of every bit B i is ∆ B i = 

Revised: X = 2 h ln((1 − p 2 ) / p 2 ) , given that the sensitivity of every bit B i is ∆ B i =

Original: (Erlingsson et al., 2014).
Revised: (Erlingsson et al., 2014).

Original: To deal with numerical inputs, e.g., embedded features, generalized RR mechanisms such as Duchi (Duchi & Rogers, 2019; Wang et al., 2019a), Piece-wise (Wang et al., 2019a), Hybrid (Wang et al., 2019a), Three-outputs (Zhao et al., 2020b), Suboptimal (Zhao et al., 2020b), LDP-FL
Revised: To deal with numerical inputs, e.g., embedded features, generalized RR mechanisms such as Duchi Asymmetric version of RAPPOR (e.g., (Wang et al., 2017)) designs different randomization probabilities for different inputs. The technique is well-applied in the context of frequency estimation and successfully reduce the communication cost from O ( d ) to O (log n ) ( d is data dimension and n is the number of samples).

Original: (Sun et al., 2021), LATENT (Arachchige et al., 2019), and OME (Lyu et al., 2020a) can be applied.
Revised: However, simply applying the mechanism (Wang et al., 2017) does not optimize the model utility and the privacy-utility trade-off when working with machine learning or deep learning models. Another line of work in LDP is Mironov (2012), which addresses the ﬂoating-point arithmetic in implementation of DP applications. The inconsistency between mathematical abstraction of Laplace mechanism with sampling “uniform” ﬂoating-point numbers can be exploited to carry out privacy attacks. Floating-point arithmetic is a leaky abstraction, which is ubiquitous in computer systems and is difﬁcult to argue about formally and hard to get right in applications, including all the RR mechanisms.

Original: B B IT R AND A LGORITHM P SEUDO -C ODE
Revised: However, different from the asymmetric version of RAPPOR and the ﬂoating-point arithmetic, our proposed f -RR mechanism focuses on mitigating the privacy-utility trade-off. To achieve that, besides the asymmetric nature of the randomization probabilities, our designed f -RR consists of two key components: 1) The bit-aware term i % l/l , which indicates the location of the bit i in each embedded feature associated with the sensitivity of the bit at that location; and 2) The adjustable but bounded α , which takes into account the correlation between privacy loss and the sensitivity of embedded features to mitigate the privacy-utility trade-off and the curse of privacy composition. The bit-aware property refers to the bits with a more substantial inﬂuence on the model utility have smaller randomization probabilities, and vice-versa, under the same privacy protection. By incorporating sensitivities of binary encoding bits into a generalized privacy loss bound, we show that increasing the dimensions of embedded features r , encoding bits l , and model outcomes

Original: 1: Input : Privacy budget (cid:15) X and (cid:15) Y , number of training iterations T , learning rate η t ,  binary encodingparameters ( l and m ) 2:
Revised: C marginally affect the randomization probabilities in BitRand under the same privacy budget. This dimension-elastic property is crucial to mitigate the curse of privacy composition by retaining a high value of data transmitted correctly through our randomization given large dimensions of r , l , and C . Besides the f -RR for protecting the data, we also include the label-RR for protecting the label in our proposed BitRand mechanism, that provides a complete protection for every data sample. 1: Input : binary encoding

Original: At server side: 3: Initialize model θ4: Send the pre-trained model f pre toclients 5: for t ∈T do 6:
Revised: At : 3: Initialize 4: Send pre pre to 5: for t ∈ do 6:

Original: Distribute model parameter θ t to each client 7: for each client u do 8: (cid:53) uθ t ← Client-Update ( θ t ) 9: end for 10: θ t +1 =
Revised: Distribute model parameter θ t to each client 7: for each client u do 8: (cid:53) uθ t ← Client-Update ( θ t ) 9: end for 10: θ t +1 =

Original: θ t − η t (cid:80) u ∈ [1 ,N ] (cid:53) u θ t /N 11: end for 12: Output: ( (cid:15) X , (cid:15) Y ) -LDP θ 13: At client side u ∈ [1 , N ] : 14: for each data sample ( x,
Revised: θ t − η t (cid:80) u ∈ [1 ,N ] (cid:53) uθ t /N 11: end for 12: Output: ( (cid:15) X , (cid:15) Y ) -LDP θ 13: At client side u ∈ [1 , N ] : 14: for each data sample ( x,

Original: y x ) ∈
Revised: y x ) ∈

Original: D u do 15:
Revised: D u do 15:

Original: Extracting embedded features: e x ← f pre ( x ) 16: v [Table] ← BinaryEncoding ( e x ) # using Eq.17:
Revised: Extracting embedded features: e x ← f pre ( x ) 16: v x ← BinaryEncoding ( e x ) # using Eq.17:

Original: Randomizing v [Table]
Revised: Randomizing v x : v (cid:48)

Original: [Table] -RR ( v x ) with α = (cid:114) (cid:15) 
Revised: x ← f -RR ( v x ) with α = (cid:114) (cid:15) X

Original: [Table] [Table]
Revised: + rl 2 r (cid:80) l − 1 i =0 exp(2 (cid:15) X il )

Original: Eq. [Table]
Revised: # using Eq.18:

Original: Randomizing the [Table] [Table]
Revised: Randomizing the label y x : y (cid:48)

Original: [Table]
Revised: x ← label -RR

Original:  β = (cid:15) [Table] − [Table]
Revised: ( y x ) with β = (cid:15) Y − ln

Original: [Table]

Original:  − [Table]
Revised: ( C − 1)

Original: # using Eq.19: end for 20: Client-Update ( θ t ): 21: (cid:53) uθ t =
Revised: # using Eq.19: end for 20: Client-Update ( θ t ): 21: (cid:53) uθ t =

Original: (cid:80) ( v (cid:48)
Revised: (cid:80) ( v (cid:48)

Original: x ,y (cid:48) x ) ∈ D (cid:48) u (cid:53) θ t L ( f ( e (cid:48) x ) , y (cid:48) x ) /n
Revised: x ,y (cid:48) x ) ∈ D (cid:48) u (cid:53) θ t L ( f ( e (cid:48) x ) , y (cid:48) x ) /n

Original: u 22: return (cid:53) uθ t
Revised: u 22: return (cid:53) uθ t

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let us study the l 1 -sensitivity of binary encoding bits of a feature’s value a in e x .
Revised: Without loss of generality, let us study the l 1 -sensitivity of binary encoding bits of a feature’s value a in e x .

Original: It is obvious that f -RR ( v a , i ) and f -RR ( v a | i , i ) differs at the bit i in the worst case.
Revised: It is obvious that f -RR ( v a , i ) and f -RR ( v a | i , i ) differs at the bit i in the worst case.

Original: The decoded feature of f -RR ( v a , i ) is a (cid:48)
Revised: The decoded feature of f -RR ( v a , i ) is a (cid:48)

Original: =
Revised: =

Original: E ( f -RR ( v a , i )) .
Revised: E ( f -RR ( v a , i )) .

Original: Let us denote b 0 b 1 . . .
Revised: Let us denote b 0 b 1 . . .

Original: b l − 1 as the binary representation f -RR ( v a , i ) of a (cid:48) , where b 0 is the value of a sign bit (i.e., b 0 = 1 if a (cid:48) >
Revised: b l − 1 as the binary representation f -RR ( v a , i ) of a (cid:48) , where b 0 is the value of a sign bit (i.e., b 0 = 1 if a (cid:48) >

Original: = 0 and b 0 = 

Revised: = 0 and b 0 =

Original: if a (cid:48) < 0 ), and { b i } l − 1 i =1 is the value of integer bits and fraction bits.
Revised: if a (cid:48) < 0 ), and { b i } l − 1 i =1 is the value of integer bits and fraction bits.

Original: We have a decoding function: E ( f -RR ( v a , i ))
Revised: We have a decoding function: E ( f -RR ( v a , i ))

Original: =
Revised: =

Original: (2 b 0 − 1) (cid:80) l − 1 i =

Revised: (2 b 0 − 1) (cid:80) l − 1 i =

Original: b
Revised: b

Original: i × 2 m − i .
Revised: i × 2 m − i .

Original: This decoding function is also applicable to E ( f -RR ( v a | i , i )) .
Revised: This decoding function is also applicable to E ( f -RR ( v a | i , i )) .

Original: Let us denote { b (cid:48)
Revised: Let us denote { b (cid:48)

Original: i } l − 1 i =

Revised: i } l − 1 i =

Original: as the value of the bit i in f -RR ( v a | i , i ) .
Revised: as the value of the bit i in f -RR ( v a | i , i ) .

Original: Following Eq. 4, the l 1 -sensitivity of the bit b
Revised: Following Eq. 4, the l 1 -sensitivity of the bit b

Original: i is computed as follows:
Revised: i is computed as follows:

Revised: • If i is an integer or a fraction bit ( i ∈ [1 , l − 1] ), we have:

Original: [Table]
Revised: [Equation]

Original: If i is an integer or a fraction bit (  i ∈ [
Revised: E ( f -RR ( v a , i )) −E (cid:0) f -RR ( v a | i , i ) (cid:1)

Original: [Table]
Revised: [Equation]

Revised: max

Revised: [Equation]

Revised: [Equation]

Revised: − 1)

Revised: [Equation]

Revised: (cid:80)

Revised: [Equation]

Revised: [Equation]

Original: , l
Revised: =1 ,j [Equation] i [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: 1 ≤ max v a

Revised: [Equation]

Revised: | (2 b 0 − 1) |(cid:107)

Revised: b i − b (cid:48)

Revised: i (cid:107) 1 2 m − i (cid:17) .

Revised: The ∆ i is maximized when b i and b (cid:48)

Revised: i are different, and | 2 b 0 − 1 | = 1 .

Revised: Therefore, ∆ i = 2 m − i .

Revised: • If i is the sign bit ( i = 0 ), we have:

Original: [Table]
Revised: [Equation]

Revised: max

Original: [Table] [Table]
Revised: [Equation] [Equation]

Original: If i is the sign bit ( i 
Revised: [Equation] l [Equation] 1 i =

Original: [Table]
Revised: [Equation]

Original: we
Revised: i

Original: [Table]
Revised: [Equation]

Original: 0 ) (cid:80) l −1 i [-=
-]
Revised: − i [-=
-][Equation]

Original: b i 2 m − i (cid:107) 1 ≤ max v a (cid:16) | 2 b 0 − 2 b (cid:48) 0 |(cid:107) (cid:80) l − 1 i =1 b i 2 m − i (cid:107) 1 (cid:17) .
Revised: [Equation]

Original: The ∆ i is maximized when b 0 and b (cid:48) 0 are different and all { b i } l − 1 i =1 = 1 .
Revised: The ∆ i is maximized when b 0 and b (cid:48) 0 are different and all { b i } l − 1 i =1 = 1 .

Original: Then, ∆ i ≤ max v a | 0 2 (cid:80) l − 1 i =1 2 m − i .
Revised: Then, ∆ i ≤ max v a | 0 2 (cid:80) l − 1 i =1 2 m − i .

Original: Since 2 1 + . ..
Revised: Since 2 1 + . .

Original: + 2 l − 2 = 2 l − 1 − 2 (mat), we have: 2 (cid:80) l − 1 i =1 [-
-] m − i 
Revised: + 2 l − 2 = 2 l − 1 − 2 (mat), we have: 2 (cid:80) l − 1 i =1 [-
-]2 m − i =

Original: = 2 m +1 − ( l − 1) (2 l − 1 − 1) <
Revised: 2 m +1 − ( l − 1) (2 l − 1 − 1) <

Original: 2 m +1 .
Revised: 2 m +1 .

Original: As a result, ∆ i = 2 m +1 .
Revised: As a result, ∆ i = 2 m +1 .

Original: From the aforementioned ∆ i of a feature’s value in e x , it is easily expanded to the entire e x .
Revised: From the aforementioned ∆ i of a feature’s value in e x , it is easily expanded to the entire e x .

Original: Since e x is the concatenation of all l bits of r values in e x , all bits with the same value of i
Revised: Since e x is the concatenation of all l bits of r values in e x , all bits with the same value of i

Original: % l have the same l 1 -sensitivity across rl bits.
Revised: % l have the same l 1 -sensitivity across rl bits.

Original: Therefore, the ∆ i of the sign bit and the integer/fraction bits are 2 m + and 2 m − i % l , respectively.
Revised: Therefore, the ∆ i of the sign bit and the integer/fraction bits are 2 m + and 2 m − i % l , respectively.

Original: Consequently, Lemma 1 hold.
Revised: Consequently, Lemma 1 hold.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Let us consider a bit i belonging to a feature a , i.e., one of the r features.
Revised: Let us consider a bit i belonging to a feature a , i.e., one of the r features.

Original: We denote f -RR ( v a , i ) as vector v a with only the bit i randomized by our f -RR, i.e., all the bits different from i are kept the same in v a ; that is, we only protect the bit i when using the notation f -RR ( v a , i ) .
Revised: We denote f -RR ( v a , i ) as vector v a with only the bit i randomized by our f -RR, i.e., all the bits different from i are kept the same in v a ; that is, we only protect the bit i when using the notation f -RR ( v a , i ) .

Original: Given the l 1 -sensitivity ∆ i , there always exists a Laplace noise injected into the embedded feature a , i.e.,
Revised: Given the l 1 -sensitivity ∆ i , there always exists a Laplace noise injected into the embedded feature a , i.e.,

Original: M ( v a , i ) =
Revised: M ( v a , i ) =

Original: E ( v a ) + Lap (∆ i /(cid:15) i ) , to achieve (cid:15) i -LDP in the embedded feature space (Dwork & Roth, 2014).
Revised: E ( v a ) + Lap (∆ i /(cid:15) i ) , to achieve (cid:15) i -LDP in the embedded feature space (Dwork & Roth, 2014).

Original: In other words, P ( M ( v a ,i )=
Revised: In other words, P ( M ( v a ,i )=

Original: z ) P ( M ( v a | i ,i )=
Revised: z ) P ( M ( v a | i ,i )=

Original: z ) ≤ exp( (cid:15) i |E ( v a ) −E ( v a | i )
Revised: z ) ≤ exp( (cid:15) i |E ( v a ) −E ( v a | i )

Original: | ∆ i ) where v a | i is the vector that differs from v a only at the bit i and z ∈ Range ( M ) .
Revised: | ∆ i ) where v a | i is the vector that differs from v a only at the bit i and z ∈ Range ( M ) .

Original: To ensure that the privacy loss in randomizing the bit i , we need to bound α in Eq.
Revised: To ensure that the privacy loss in randomizing the bit i , we need to bound α in Eq.

Original: 2 such that P ( f -RR ( v a ,i )= v z )
Revised: 2 such that P ( f -RR ( v a ,i )= v z )

Original: P ( f -RR ( v a | i ,i )= v z ) ≤ exp( (cid:15) i |E ( v a ) −E ( v a | i ) | ∆ i ) .
Revised: P ( f -RR ( v a | i ,i )= v z ) ≤ exp( (cid:15) i |E ( v a ) −E ( v a | i ) | ∆ i ) .

Original: However, randomizing a binary encoding bit i given v a and v a | i results in a smaller l 1 -distance |E ( f -RR ( v a , i ))
Revised: However, randomizing a binary encoding bit i given v a and v a | i results in a smaller l 1 -distance |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i )) | compared with |E ( v a ) − E ( v a | i ) | ; since |E ( f -RR ( v a , i ))
Revised: − E ( f -RR ( v a | i , i )) | compared with |E ( v a ) − E ( v a | i ) | ; since |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i ))
Revised: − E ( f -RR ( v a | i , i ))

Original: | ≤ |E ( v a ) − E ( v a | i ) | .
Revised: | ≤ |E ( v a ) − E ( v a | i ) | .

Original: Thus, we can derive a tighter bound by replacing |E ( v a ) − E
Revised: Thus, we can derive a tighter bound by replacing |E ( v a ) − E

Original: ( v a | i ) | with |E ( f -RR ( v a , i ))
Revised: ( v a | i ) | with |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i )) | .
Revised: − E ( f -RR ( v a | i , i )) | .

Original: Given v a ( i ) is the bit i in v a , we have
Revised: Given v a ( i ) is the bit i in v a , we have

Original: P

Original: [Equation]
Revised: [Equation]

Revised: However, ﬁnding a closed-form solution of α for the tight privacy bound in Eq.

Revised: 10 is non-trivial, since (cid:15) i is intractable.

Original:  f -RR[Equation] v a , i ) = v z ) [Equation]
Revised: To address this problem, we consider two cases: (1) P ( f -RR

Original: [Equation]
Revised: [Equation]

Original: l
Revised: i

Original: [Equation]
Revised: [Equation]

Original: [Equation] However, ﬁnding a closed-form solution of α for the tight privacy bound in Eq. 10 is non-trivial, since (cid:15) i is intractable.
Revised: z

Original: To address this problem, we consider the privacy loss in the worse case, in which P ( f -RR ( v a ( i )) =
Revised: [Equation] ≥and (2) 0 < P ( f -RR ( v a ( i )) =

Original: v z ( i ))
Revised: v z ( i ))

Original: P ( f -RR ( v a | i ( i ))
Revised: P ( f -RR ( v a | i ( i ))

Original: = v z ( i )) ≥ 1 .
Revised: = v z ( i )) < 1 .

Original: Also, since ∆ i captures the magnitude by which a bit i can change the decoding function E () in the worst case, we have
Revised: Also, since ∆ i captures the magnitude by which a bit i can change the decoding function E () in the worst case, we have

Original: [Equation]
Revised: [Equation]

Original: As a result, the privacy loss for a bit i can be bounded as follows:
Revised: In the ﬁrst case, the privacy loss for a bit i can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Since the RR mechanism is independently applied on each bit i in v a and on every v a of the r features,
Revised: Since the RR mechanism is independently applied on each bit i in v a and on every v a of the r features [Equation]

Original: Eq.
Revised: Eq.

Original: 11 enables us to quantify a generalized privacy loss bound of a RR mechanism, in which different bits have different sensitivities to the randomized outcome as follows. 
Revised: 11 enables us to quantify a generalized privacy loss bound of a RR mechanism [Equation] in which different bits have different sensitivities to the randomized outcome as follows. [Equation]

Revised: In the second case

Original: [Equation]
Revised: [Equation]

Revised: the privacy loss for a bit i can be bounded as follows:

Revised: [Equation]

Revised: Similarly

Revised: [Equation]

Revised: we obtain the same result with Eq.

Revised: 12 in the second case:

Revised: [Equation]

Original: Consequently, Theorem 1 holds.
Revised: Consequently, Theorem 1 holds [Equation]

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: To ensure that the privacy loss in randomizing the bit i is bounded by the privacy loss in the em- bedded space and to take into account different sensitivities of different bits (Theorem 1) [Equation] we need to ﬁnd α in Eq.
Revised: To ensure that the privacy loss in randomizing the bit i is bounded by the privacy loss in the em- bedded space and to take into account different sensitivities of different bits (Theorem 1), we need to ﬁnd α in Eq.

Original: 2 to solve Eq.
Revised: 2 to solve Eq.

Original: However, solving Eq.
Revised: However, solving Eq.

Original: 12 is not straightforward since the privacy budget (cid:15)
Revised: 12 is not straightforward since the privacy budget (cid:15)

Original: i and the actual l 1 -distance |E ( f -RR ( v x , i ))
Revised: i and the actual l 1 -distance |E ( f -RR ( v x , i ))

Original: − E ( f -RR ( v x | i , i )) | are intractable.
Revised: − E ( f -RR ( v x | i , i )) | are intractable.

Original: To address this problem [Equation] from Eq.
Revised: To address this problem, from Eq.

Revised: 12, we ﬁrst consider the bit i in all the r embedded features, as follows: (cid:81)

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst consider the bit i in all the r embedded features, as follows:
Revised: exp

Original: [Equation]
Revised: [Equation]

Original: Note that all the bits i , e.g., sign bits, in all the features a ∈ [1 , r ] consume the same privacy budget (cid:15)
Revised: Note that all the bits i , e.g., sign bits, in all the features a ∈ [1 , r ] consume the same privacy budget (cid:15)

Original: i with the same sensitivity ∆ i .
Revised: i with the same sensitivity ∆ i .

Original: The term (cid:80) a ∈ [1 ,r ] |E ( f -RR ( v a , i ))
Revised: The term (cid:80) a ∈ [1 ,r ] |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i )) | can be unbiasedly replaced with r × E |E ( f -RR ( v a , i ))
Revised: − E ( f -RR ( v a | i , i )) | can be unbiasedly replaced with r × E |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i ))
Revised: − E ( f -RR ( v a | i , i ))

Original: | where E is the expectation of |E ( f -RR ( v a , i ))
Revised: | where E is the expectation of |E ( f -RR ( v a , i ))

Original: − E ( f -RR ( v a | i , i ))
Revised: − E ( f -RR ( v a | i , i ))

Original: | , since lim r →∞ E |E ( f -RR ( v a , i ))
Revised: | , since lim r →∞ E |E [Equation] f -RR [Equation]

Original: − E ( f -RR ( v a| i , i ))
Revised: v a , i [Equation] f - [Equation]

Original:  | =(cid:80)
Revised: i [Equation] | =

Original: [Equation]
Revised: [Equation]

Original: Hence, we have that
Revised: Hence, we have that

Original: [Equation]
Revised: [Equation]

Original: By considering all the bits i ∈ [ [Equation] l − 1] 
Revised: By considering all the bits i ∈ [0 , l − 1] , from Eq. 15, we have that

Revised: P

Original: [Equation]
Revised: [Equation]

Original: from Eq.
Revised: f -RR ( v x ) = v z )

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: ≤ exp( (cid:88) a ∈ [

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: r ] (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  we need to bound the generalized privacy loss by discovering closed-form solutions of α given the privacy budgets (cid:15) X
Revised: Now, we need to bound the generalized privacy loss by discovering closed-form solutions of α given the privacy budgets (cid:15) X

Original: (Theorem 2).
Revised: (Theorem 2).

Original: In other words, we need to solve Eq.
Revised: In other words, we need to solve Eq.

Original: 14 for discovering the closed-form solution of α .
Revised: 16 for discovering the closed-form solution of α .

Original: To solve it, ﬁrst we need to calculate E |E ( f -RR ( v x , i ))
Revised: To solve it, ﬁrst we need to calculate E |E ( f -RR ( v x , i ))

Original: −
Revised: −

Original: E ( f -RR ( v x | i , i ))
Revised: E ( f -RR ( v x | i , i ))

Original: | .
Revised: | .

Original: To be more precise, let us denote p Xi and q Xi as p X and q X in Eq.
Revised: To be more precise, let us denote p Xi and q Xi as p X and q X in Eq.

Original: 2 for a particular bit i , respectively.
Revised: 2 for a particular bit i , respectively.

Original: Given the worse case of v x and v x | i , there are four possible cases of |E ( f -RR ( v x , i ))
Revised: Given the worse case of v x and v x | i , there are four possible cases of |E ( f -RR ( v x , i ))

Original: − E ( f -RR ( v x | i , i)) | :
Revised: − E ( f -RR ( v x | i , i

Original:  • If f -RR ( v x ( i ))
Revised: )) | : • If f -RR ( v x ( i ))

Original: =
Revised: =

Original: 1 and f -RR ( v x | i ( i ))
Revised: 1 and f -RR ( v x | i ( i ))

Original: = 1 , then |E
Revised: = 1 , then |E

Original: ( f -RR ( v x , i ))
Revised: ( f -RR ( v x , i ))

Original: − E ( f -RR ( v x | i , i ))
Revised: − E ( f -RR ( v x | i , i ))

Original: | = 0 .
Revised: | = 0 .

Original: • If f -RR ( v x ( i ))
Revised: • If f -RR ( v x ( i ))

Original: = 

Revised: =

Original: and f -RR ( v x | i ( i ))
Revised: and f -RR ( v x | i ( i ))

Original: = 0 , then |E ( f -RR ( v x , i ))
Revised: = 0 , then |E ( f -RR ( v x , i ))

Original: − E ( f -RR ( v x | i , i ))
Revised: − E ( f -RR ( v x | i , i ))

Original: | = 0 .
Revised: | = 0 .

Original: • If f -RR ( v x ( i ))
Revised: • If f -RR ( v x ( i ))

Original: =
Revised: =

Original: 1 and f -RR ( v x | i ( i ))
Revised: 1 and f -RR ( v x | i ( i ))

Original: = 0 , then |E ( f -RR ( v x , i ))
Revised: = 0 , then |E ( f -RR ( v x , i ))

Original: − E ( f -RR ( v x | i , i ))
Revised: − E ( f -RR ( v x | i , i ))

Original: | = ∆ i .
Revised: | = ∆ i .

Original: This happens with the probability P ( f -RR ( v x ( i ))
Revised: This happens with the probability P ( f -RR ( v x ( i ))

Original: = 1 , f -RR ( v x | i ( i )) = 0) .
Revised: = 1 , f -RR ( v x | i ( i )) = 0) .

Original: To compute this probability, we use marginal probability and Bayes’ theorem, as follows:
Revised: To compute this probability, we use marginal probability and Bayes’ theorem, as follows:

Original: P ( f -RR ( v x ( i ))
Revised: P ( f -RR ( v x ( i ))

Original: = 1 , f -RR ( v x | i ( i ))
Revised: = 1 , f -RR ( v x | i ( i ))

Original: = 0)
Revised: = 0)

Original: =
Revised: =

Original: P (cid:16) f -RR ( v x ( i ))
Revised: P (cid:16) f -RR ( v x ( i ))

Original: = 1 , f -RR ( v x | i ( i ))
Revised: = 1 , f -RR ( v x | i ( i ))

Original: = 0 , v x ( i ) = 1 , v x | i ( i ) = 0 (cid:17) + P (cid:16) f -RR
Revised: = 0 , v x ( i ) = 1 , v x | i ( i ) = 0 (cid:17) + P (cid:16) f -RR

Original: ( v x ( i ))
Revised: ( v x ( i ))

Original: = 1 , f -RR ( v x | i ( i ))
Revised: = 1 , f -RR ( v x | i ( i ))

Original: = 0 , v x ( i )
Revised: = 0 , v x ( i )

Original: = 0 , v x | i ( i ) = 1 (cid:17) =
Revised: = 0 , v x | i ( i ) = 1 (cid:17) =

Original: P (cid:16) f -RR ( v x ( i ))
Revised: P (cid:16) f -RR ( v x ( i ))

Original: = 

Revised: =

Original: | f -RR ( v x | i ( i ))
Revised: | f -RR ( v x | i ( i ))

Original: = 0 , v x ( i ) = 1 , v x | i ( i ) = 0 (cid:17) × P ( f -RR ( v x | i ( i ))
Revised: = 0 , v x ( i ) = 1 , v x | i ( i ) = 0 (cid:17) × P ( f -RR ( v x | i ( i ))

Original: = 

Revised: =

Original: | v x ( i ) = 1 , v x | i ( i ) = 0) × P ( v x ( i ) = 1 | v x | i ( i ) = 0) × P ( v x | i ( i ) = 0)
Revised: | v x ( i ) = 1 , v x | i ( i ) = 0) × P ( v x ( i ) = 1 | v x | i ( i ) = 0) × P ( v x | i ( i ) = 0)

Original: + P (cid:16)
Revised: + P (cid:16)

Original: f -RR ( v x ( i ))
Revised: f -RR ( v x ( i ))

Original: = 

Revised: =

Original: | f -RR ( v x | i ( i ))
Revised: | f -RR ( v x | i ( i ))

Original: = 0 , v x ( i )
Revised: = 0 , v x ( i )

Original: = 0 , v x | i ( i ) = 1 (cid:17) × P ( f -RR ( v x | i ( i ))
Revised: = 0 , v x | i ( i ) = 1 (cid:17) × P ( f -RR ( v x | i ( i ))

Original: = 

Revised: =

Original: | v x ( i ) = 0 , v x | i ( i ) = 1) × P ( v x ( i ) = 

Revised: | v x ( i ) = 0 , v x | i ( i ) = 1) × P ( v x ( i ) =

Original: | v x | i ( i ) = 1) × P ( v x | i ( i ) = 1)
Revised: | v x | i ( i ) = 1) × P ( v x | i ( i ) = 1)

Original: =
Revised: =

Original: P (cid:16) f -RR ( v x ( i ))
Revised: P (cid:16) f -RR ( v x ( i ))

Original: =
Revised: =

Original: 1 | v x ( i ) = 1 (cid:17) × P ( f -RR ( v x | i ( i ))
Revised: 1 | v x ( i ) = 1 (cid:17) × P ( f -RR ( v x | i ( i ))

Original: = 

Revised: =

Original: | v x | i ( i ) = 0) × P ( v x | i ( i ) = 0)
Revised: | v x | i ( i ) = 0) × P ( v x | i ( i ) = 0)

Original: + P (cid:16)
Revised: + P (cid:16)

Original: f -RR ( v x ( i )
Revised: f -RR ( v x ( i )

Original: [Equation]
Revised: [Equation]

Original: • If f -RR ( v x ( i ))
Revised: • If f -RR ( v x ( i ))

Original: = 

Revised: =

Original: and f -RR ( v x | i ( i ))
Revised: and f -RR ( v x | i ( i ))

Original: = 1 , then |E
Revised: = 1 , then |E

Original: ( f -RR ( v x ( i )))
Revised: ( f -RR ( v x ( i )))

Original: − E ( f -RR ( v x | i ( i )))
Revised: − E ( f -RR ( v x | i ( i )))

Original: | = ∆ i .
Revised: | = ∆ i .

Original: This happens with the probability P ( f -RR ( v x ( i ))
Revised: This happens with the probability P ( f -RR ( v x ( i ))

Original: = 0 , f -RR ( v x
Revised: = 0 , f -RR ( v x

Original: | i ( i ))
Revised: | i ( i ))

Original: = 1) .
Revised: = 1) .

Original: To compute this probability, we use marginal probability and Bayes’ theorem, as follows:
Revised: To compute this probability, we use marginal probability and Bayes’ theorem, as follows:

Original: [Equation]
Revised: [Equation]

Original: Consequently, the expectation E |E ( f -RR ( v x , i )) − E ( f -RR ( v x | i , i )) | is computed as follows:
Revised: Consequently, the expectation E |E ( f -RR ( v x , i )) − E ( f -RR ( v x | i , i )) | is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: From Eqs.
Revised: From Eqs.

Original: 12, 14, and 17, we have that
Revised: 12, 16, and 19, we have that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let us bound the summation in Eq.
Revised: Let us bound the summation in Eq.

Original: 19 using the following inequality:
Revised: 21 using the following inequality:

Original: [Equation]
Revised: [Equation]

Original: The proof of Eq.
Revised: The proof of Eq.

Original: 20 is as follows.
Revised: 22 is as follows.

Original: Let a > 0 , we deﬁne h ( a ) =
Revised: Let a > 0 , we deﬁne h ( a ) =

Original: ln( a ) − a + 1 .
Revised: ln( a ) − a + 1 .

Original: We have: h (cid:48) ( a ) = 1 a − 1 = 0 ⇔ a = 1 , and since h (cid:48)(cid:48) ( a ) =
Revised: We have: h (cid:48) ( a ) = 1 a − 1 = 0 ⇔ a = 1 , and since h (cid:48)(cid:48) ( a ) =

Original: − 1 a 2 < 0 , ∀ a > 0 , we get the maximal point at a = 1 .
Revised: − 1 a 2 < 0 , ∀ a > 0 , we get the maximal point at a = 1 .

Original: We also have: lim a → 0+ h ( a ) =
Revised: We also have: lim a → 0+ h ( a ) =

Original: −∞ = lim a →∞ h ( a ) .
Revised: −∞ = lim a →∞ h ( a ) .

Original: Therefore, a = 1 is the global maximal point and than ∀ a > 0 , h ( a ) ≤ h ( a = 1) = 0 , so ln( a ) − a + 1 ≤ 0 .
Revised: Therefore, a = 1 is the global maximal point and than ∀ a > 0 , h ( a ) ≤ h ( a = 1) = 0 , so ln( a ) − a + 1 ≤ 0 .

Original: Therefore, Eq.does hold.
Revised: Therefore, Eq.does hold.

Original: Note that, to simultaneously satisfy the randomization probabilities p X =
Revised: Note that, to simultaneously satisfy the randomization probabilities p X =

Original: [Equation]
Revised: [Equation]

Original: and q X = α exp( i % l l (cid:15) X ) 1+ α exp
Revised: and q X = α exp( i % l l (cid:15) X ) 1+ α exp

Original: ( i % l l (cid:15) X ) ≥ 0 in Eq. 2, we need to have: (i) 

Revised: ( i % l l (cid:15) X ) ≥ 0 in Eq. 2, we need to have: (i)

Original: + α exp
Revised: + α exp

Original: ( i % l l (cid:15) X ) ≥ 0 and (ii) α exp( i % l l (cid:15) X ) ≥ 0 .
Revised: ( i % l l (cid:15) X ) ≥ 0 and (ii) α exp( i % l l (cid:15) X ) ≥ 0 .

Original: Since exp( i % l l (cid:15) X ) ≥ 0 is always true, from (i) , α ≥ − exp( − i % l l (cid:15) X ) , and from (ii) , α ≥ 0 .
Revised: Since exp( i % l l (cid:15) X ) ≥ 0 is always true, from (i) , α ≥ − exp( − i % l l (cid:15) X ) , and from (ii) , α ≥ 0 .

Original: Therefore, α ≥ 0 is necessary to satisfy the condition p X ≥ 0 and q X ≥ 0 .
Revised: Therefore, α ≥ 0 is necessary to satisfy the condition p X ≥ 0 and q X ≥ 0 .

Original: To apply
Revised: To apply

Original: Eq.
Revised: Eq.

Original: 20 into Eq.
Revised: 22 into Eq.

Original: 19, we need to have a = α 2 exp(2 (cid:15) X
Revised: 21, we need to have a = α 2 exp(2 (cid:15) X

Original: i l ) > 0 ⇒ α (cid:54) = 0 .
Revised: i l ) > 0 ⇒ α (cid:54) = 0 .

Original: As a result, we have that α
Revised: As a result, we have that α

Original: [Equation]
Revised: [Equation]

Original: Applying Eq.
Revised: Applying Eq.

Original: 20 into Eq.
Revised: 22 into Eq.

Original: 19 where a = α 2 exp 
Revised: 21 where a = α 2 exp [Equation] p

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: To bound the logarithm in Eq.
Revised: To bound the logarithm in Eq.

Original: 22, we use: p 2 Xi + q 2 Xi =
Revised: 24, we use: p 2 Xi + q 2 Xi =

Original: p 2 Xi
Revised: p 2 Xi

Original: + (
Revised: + (

Revised: [Equation]

Revised: p Xi

Original: [Equation]
Revised: [Equation]

Original: = (cid:16)
Revised: = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: + (cid:16) α exp
Revised: + (cid:16) α exp

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 1+
Revised: l

Original: [Equation]
Revised: [Equation]

Original: exp
Revised: X

Original: [Equation]
Revised: [Equation]

Original:  i
Revised: 1+ α exp( i

Original: [Equation]
Revised: [Equation]

Original: l l (cid:15)
Revised: l l (cid:15)

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: (cid:16)

Revised: [Equation]

Revised: exp( i

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original:  exp( i % l l (cid:15) X ) (cid:17) 2 ≤ 1 , and p 2 Xi + (1 − p Xi ) 2 ≥ ( p Xi +1 − p Xi )
Revised: (cid:16) 1+ [Equation] exp( i % l l (cid:15) X ) (cid:17) 2 ≤ 1 , and p 2 Xi + (1 − p Xi ) 2 ≥ ( p Xi +1 − p Xi )

Original: 2= 12 (In fact,
Revised: 2= 12 (In fact,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⇔
Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: b ) 2 ≥ 0 , which is true).
Revised: ) 2 ≥ 0 , which is true).

Original: Note that, from Eq.
Revised: Note that, from Eq.

Original: 17, we have that
Revised: 19, we have that

Original: [Equation]
Revised: [Equation]

Original: Applying these inequalities in Eq.
Revised: Applying these inequalities in Eq.

Original: 22, we obtain:
Revised: 24, we obtain:

Original: [Equation]
Revised: [Equation]

Original: By solving Eq.
Revised: By solving Eq.

Original: 23, we have that
Revised: 25, we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, from Eqs.
Revised: Therefore, from Eqs.

Original: 21 and 24, we have that ∀ α : 0 < α ≤ 
Revised: 23 and 26, we have that ∀ α : 0 < α ≤ (cid:113)

Original: [Equation]
Revised: [Equation]

Original: 0 exp
Revised: rl [Equation] r

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:15) X il ) , the f -RR mechanism satisﬁes (cid:15) X -LDP.
Revised: exp [Equation] [Equation] X il ) , the f -RR mechanism satisﬁes (cid:15) X -LDP.

Original: Consequently, Theorem 2 holds.
Revised: Consequently, Theorem 2 holds.

Original: F
Revised: F

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We have: P ( label -RR ( y x )=
Revised: We have: P ( label -RR ( y x )=

Original: z | y x ) P ( label -RR ( (cid:101)
Revised: z | y x ) P ( label -RR ( (cid:101)

Original: y x )=
Revised: y x )=

Original: z | (cid:101) y x ) ≤ max P ( label -RR ( y x )=
Revised: z | (cid:101) y x ) ≤ max P ( label -RR ( y x )=

Original: z | y x ) min P ( label -RR ( (cid:101)
Revised: z | y x ) min P ( label -RR ( (cid:101)

Original: y x )=
Revised: y x )=

Original: z | (cid:101) y
Revised: z | (cid:101) y

Original: x
Revised: x

Original: ) = exp( β ) 1+exp( β )(1+exp( β ))
Revised: ) = exp( β ) 1+exp( β )(1+exp( β ))

Original: ( C − 1) =
Revised: ( C − 1) =

Original: exp( β + ln( C − 1)) ≤ exp( (cid:15) Y ) ⇔ β ≤ (cid:15)
Revised: exp( β + ln( C − 1)) ≤ exp( (cid:15) Y ) ⇔ β ≤ (cid:15)

Original: Y − ln
Revised: Y − ln

Original: ( C − 1) .
Revised: ( C − 1) .

Original: Consequently, Theorem 3 does hold.
Revised: Consequently, Theorem 3 does hold.

Original: G P ROOF OF T HEOREM
Revised: G label -RR AND L ABEL DP COMPARISON

Revised: Although our label-RR is inspired by the randomizing probability (Eq. 1, (Ghazi et al., 2021)) in the LabelDP showcased by (Ghazi et al., 2021), there are two major differences between our label-RR and LabelDP discussed next. (1) We combine f -RR and label-RR to completely protect a data sample.

Revised: As pointed out in (BusaFekete et al., 2021), only protecting the label as in the LabelDP offers a weaker privacy protection than it appears, as the features are sufﬁciently predictive of the label, obscuring the label is not enough, as a classiﬁer can still be trained on such noisy data; hence, a user experiences privacy loss due to both the public release of the features and the private release of the label.

Revised: The RRWithPrior algorithm that is used to guarantee LabelDP requiring publicly available priors and the multi-stage training (LP-MST) illustrating RRWithPrior algorithm cannot be straightforwardly applied to federated learning.

Revised: In LP-MST algorithm, the dataset is partitioned into subsets, then based on the prior probability to randomize the label and add the data with that randomized label to the training data.

Revised: These steps are presently applied on centralized training and it has not been show how to be effectively applied in federated learning.

Revised: Using the upper bound of β (Theorem 3) results in the same randomizing probabilities between labelRR and LabelDP.

Revised: For instance, in Eq. 3, p Y = exp( β ) 1+exp( β ) =

Revised: exp( (cid:15) Y − ln

Revised: ( C − 1))

Revised: 1+exp( (cid:15) Y − ln

Revised: ( C − 1))

Revised: =

Original: {+exp( (cid:15) Y ) C − 1+exp( (cid:15) Y ) and q Y = 
+}
Revised: {+exp( (cid:15) Y ) C − 1+exp( (cid:15) Y ) and q Y = 
+}

Revised: (1+exp( β ))

Revised: ( C − 1)

Original: {+= 1 C − 1+exp( (cid:15) Y ) , which is equivalent to Eq. 
+}
Revised: {+= 1 C − 1+exp( (cid:15) Y ) , which is equivalent to Eq. 
+}

Revised: (Ghazi et al., 2021).

Revised: Therefore, we did not include LabelDP (Ghazi et al., 2021) in comparison.

Revised: H P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We have ξ a = E |E ( f -RR ( v a ))
Revised: We have ξ a = E |E ( f -RR ( v a ))

Original: − E ( v a ) | = (cid:80) i ∈
Revised: − E ( v a ) | = (cid:80) i ∈

Original: [0 ,l − 1] ( p Xi × 0 + q Xi × ∆ i ) =
Revised: [0 ,l − 1] ( p Xi × 0 + q Xi × ∆ i ) =

Original: (cid:80) i ∈ [0 ,l − 1]
Revised: (cid:80) i ∈ [0 ,l − 1]

Original: q Xi × ∆ i .
Revised: q Xi × ∆ i .

Original: Therefore, Theorem 4 hold.
Revised: Therefore, Theorem 4 hold.

Original: H C ORRECTED P RIVACY B UDGET B OUNDS IN LATENT (A RACHCHIGE ET AL., 2019)
Revised: I C ORRECTED P RIVACY B UDGET B OUNDS IN LATENT (A RACHCHIGE ET AL

Original: In this section, we aim at providing corrected privacy budget bounds for LATENT (Arachchige et al., 2019).
Revised: In this section, we aim at providing corrected privacy budget bounds for LATENT (Arachchige et al., 2019).

Original: LATENT ﬁrst encodes embedded features e x into an rl -bit binary vector v x .
Revised: LATENT ﬁrst encodes embedded features e x into an rl -bit binary vector v x .

Original: Then, each bit i ∈ [0 , rl − 1] is randomized by a RR mechanism (i.e., the MOUE algorithm for high sensitivities in
Revised: Then, each bit i ∈ [0 , rl − 1] is randomized by a RR mechanism (i.e., the MOUE algorithm for high sensitivities in

Original: Theorem 3.3 (Arachchige et al., 2019)), denoted f -LT, as follows:
Revised: Theorem 3.3 (Arachchige et al., 2019)), denoted f -LT, as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: From Eq.
Revised: From Eq.

Original: 25, we also have that P ( v (cid:48)
Revised: 27, we also have that P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − p X
Revised: = 1 − p X

Original: =
Revised: =

Original: α 

Revised: α

Original: + α
Revised: + α

Original: if v x ( i ) = 1 , and P ( v (cid:48)
Revised: if v x ( i ) = 1 , and P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: [-= 1 − q X = α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)-]
Revised: [-= 1 − q X = α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)-][Equation]

Original: Xrl ) if v x ( i ) = 0 .

Original: Theorem 5.
Revised: Theorem 5.

Original: LATENT with the randomization probabilities as in Eq.
Revised: LATENT with the randomization probabilities as in Eq.

Original: 25 preserves (cid:15) corrected -LDP, where (cid:15) corrected
Revised: 27 preserves (cid:15) corrected -LDP, where (cid:15) corrected

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Similar to the analysis in Appendix E , we obtain the following inequality:
Revised: Similar to the analysis in Appendix E , we obtain the following inequality:

Original: [Equation]
Revised: [Equation]

Original:  and the expectation E |E ( f -LT ( v x , i ))
Revised: (28) and the expectation E |E ( f -LT ( v x , i ))

Original: − E ( f -LT ( v x | i , i )) | is computed as follows:
Revised: − E ( f -LT ( v x | i , i )) | is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: |

Original: [Equation]
Revised: [Equation]

Original: v
Revised: | i

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Revised: | i ( i )

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: p Xi )

Original: [Equation]
Revised: [Equation]

Revised: v x | i ( i )

Revised: [Equation]

Original: Furthermore, we have:
Revised: 1) + ( [Equation] p Xi ) q Xi P ( v x | i ( i )

Original: [Equation]
Revised: [Equation]

Original: From Eqs. 26-28, we have that [Equation] ≤
Revised: Furthermore

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation] f [Equation] ( i
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] [Equation] i [Equation]
Revised: From Eqs.

Original: i [Equation] [Equation] ×
Revised: 28-30, we have that

Original: [Equation]
Revised: [Equation]

Original: i =

Original: [Equation]

Original: [Equation]

Original: Then, from Eq.
Revised: Then, from Eq.

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Consequently, Theorem 5 holds
Revised: Consequently, Theorem 5 holds.

Original: [Equation]

Original: [Figure]

Original: From Theorem 5, we show the proportion (cid:15) corrected /(cid:15)
Revised: From Theorem 5, we show the proportion (cid:15) corrected /(cid:15)

Original: X as a function of r in Figure 9a and as a function of l in Figure 9b.
Revised: X as a function of r in Figure 9a and as a function of l in Figure 9b.

Original: Following the experiment settings in LATENT (Arachchige et al., 2019), with the commonly used α = 7 , when changing r ∈ { 10 , 100 , 1 , 000 , 10 , 000 } with a ﬁxed l =(Figure 9a), or when changing l ∈ { 5 , 10 , 20 , 100 , 1 , 000 } with a ﬁxed r = 1 , 000 (Figure 9b) and under a tight privacy budget (cid:15) X
Revised: Following the experiment settings in LATENT (Arachchige et al., 2019), with the commonly used α = 7 , when changing r ∈ { 10 , 100 , 1 , 000 , 10 , 000 } with a ﬁxed l =(Figure 9a), or when changing l ∈ { 5 , 10 , 20 , 100 , 1 , 000 } with a ﬁxed r = 1 , 000 (Figure 9b) and under a tight privacy budget (cid:15) X

Original: = 0 .
Revised: = 0 .

Original: 1 , the proportion (cid:15) corrected /(cid:15)
Revised: 1 , the proportion (cid:15) corrected /(cid:15)

Original: X moderately changes among [4 . 57 , 4 . 75] .
Revised: X moderately changes among [4 . 57 , 4 . 75] .

Original: In other words, the (cid:15) corrected is remarkably larger than (cid:15) X , for most r and l values in practice.
Revised: In other words, the (cid:15) corrected is remarkably larger than (cid:15) X , for most r and l values in practice.

Original: Unlike LATENT, our mechanism does not suffer from this problem, i.e., in BitRand, (cid:15) corrected /(cid:15)
Revised: Unlike LATENT, our mechanism does not suffer from this problem, i.e., in BitRand, (cid:15) corrected /(cid:15)

Original: X
Revised: X

Original: = 1 , thanks to our bit-aware randomization probabilities for LDP in binary encoding ( Theorem 2 ).
Revised: = 1 , thanks to our bit-aware randomization probabilities for LDP in binary encoding ( Theorem 2 ).

Original: I C ORRECTED P RIVACY B UDGET B OUNDS IN OME (L YU ET AL ., 2020 A )
Revised: J C ORRECTED P RIVACY B UDGET B OUNDS IN OME (L YU ET AL ., 2020 A )

Original: In this section, we aim at providing corrected privacy budget bounds for OME.
Revised: In this section, we aim at providing corrected privacy budget bounds for OME.

Original: OME ﬁrst encodes embedded features e x into an rl -bit binary vector v x .
Revised: OME ﬁrst encodes embedded features e x into an rl -bit binary vector v x .

Original: Then, each bit i ∈ [0 , rl − 1] is randomized by the following f -OME mechanism:
Revised: Then, each bit i ∈ [0 , rl − 1] is randomized by the following f -OME mechanism:

Original: [Equation]
Revised: [Equation]

Original: X = α 1 + α, if i ∈ 2 j, v x ( i ) = p 2 X
Revised: ) , if

Original: [-= 
-]
Revised: [-= 
-][Equation]

Original: [-+ α 3 , if i ∈ 2 j + 1 , v x ( i ) = q X = 
 + α exp( (cid:15) X rl ) , if v x ( i ) =(31)-]
Revised: [-+ α 3 , if i ∈ 2 j + 1 , v x ( i ) = q X = 
 + α exp( (cid:15) X rl ) , if v x ( i ) =(31)-][Equation]

Original: From Eq.
Revised: From Eq.

Original: 31, we also have that P ( v (cid:48)
Revised: 33, we also have that P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − p 1 X = 1 

Revised: = 1 − p 1 X = 1

Original: + α
Revised: + α

Original: if v x ( i ) =
Revised: if v x ( i ) =

Original: and i ∈ 2 j , P ( v (cid:48)
Revised: and i ∈ 2 j , P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − p 2 X
Revised: = 1 − p 2 X

Original: =
Revised: =

Original: α 3 1+ α 3 if v x ( i ) = 

Revised: α 3 1+ α 3 if v x ( i ) =

Original: and i ∈ 2 j + 1 , and P ( v (cid:48)
Revised: and i ∈ 2 j + 1 , and P ( v (cid:48)

Original: x ( i ) = 0)
Revised: x ( i ) = 0)

Original: = 1 − q X
Revised: = 1 − q X

Original: =
Revised: =

Original: α exp( (cid:15)X rl ) 1+{+
 ++} α exp( (cid:15)Xrl ) if v x ( i ) = 0 .
Revised: α exp( (cid:15)Xrl ) {+
 ++} α exp( (cid:15)Xrl ) if v x ( i ) = 0 .

Original: Theorem 6.
Revised: Theorem 6.

Original: OME with the randomization probabilities as in Eq.
Revised: OME with the randomization probabilities as in Eq.

Original: 31 preserves (cid:15) corrected -LDP, where (cid:15) corrected =
Revised: 33 preserves (cid:15) corrected -LDP, where (cid:15) corrected =

Original: [Equation] rlQ [Equation] rlQ 
Revised: ( rlQ 1 − rlQ 2 ) ln( α ) + (cid:15)

Original: [Equation]
Revised: X 2 Q 1 + (cid:15)

Original:  2 in which Q 1 ={+
+}
Revised: X 2 Q 2 in which Q {+
+}

Original:  α 

Revised: = α

Original: + α α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)Xrl )
Revised: + α α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)Xrl )

Original: + {+1 
+}
Revised: + {+1 
+}

Original: [Equation]{++ α exp( (cid:15)Xrl ) 1 
 + α and Q 2 = 1 
 + α 3 α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)Xrl )+}
Revised: {++ α exp( (cid:15)Xrl ) 1 
 + α and Q 2 = 1 
 + α 3 α exp( (cid:15)Xrl ) 
 + α exp( (cid:15)Xrl )+}

Original: + α{+1 
+}
Revised: + {+1 
+}

Original: [Equation] 1 1+ α and Q 2 = [Equation]{+exp( (cid:15)Xrl ) α 3 
 + α 3 .+}
Revised: + α {+exp( (cid:15)Xrl ) α 3 
 + α 3 .+}

Original: Proof.
Revised: Proof.

Original: Similar to the analysis in Appendix E and Appendix H , we obtain: 
Revised: Similar to the analysis in Appendix E and Appendix I , we obtain: [Figure] [Equation] [Figure]

Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: and the expectation E |E ( f -OME ( v x , i ))
Revised: and the expectation E |E ( f -OME ( v x , i ))

Original: − E ( f -OME ( v x | i , i )) | is computed as follows:
Revised: − E ( f -OME ( v x | i , i )) | is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where Q1 = p
Revised: ) where Q

Original: [Equation]
Revised: [Equation]

Original: andQ 2 = p 2 Xi [Equation] =
Revised: and

Original: [Equation]
Revised: [Equation]

Original: From Eqs.
Revised: From Eqs.

Original: 32 and
Revised: 34 and

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then, from Eq. 34, we have: 
Revised: Then, from Eq. 36, we have: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Consequently, Theorem 6 does hold.
Revised: Consequently, Theorem 6 does hold.

Original: From Theorem 6, we show the proportion (cid:15) corrected /(cid:15)
Revised: From Theorem 6, we show the proportion (cid:15) corrected /(cid:15)

Original: X as a function of r in Figure 9a and as a function of l in Figure 9b.
Revised: X as a function of r in Figure 9a and as a function of l in Figure 9b.

Original: Following the experiment settings in OME (Lyu et al., 2020a), with the commonly used α = 100 , when changing r ∈ { 10 , 100 , 1 , 000 , 10 , 000 } with a ﬁxed l =(Figure 9a), or when changing l ∈ { 5 , 10 , 20 , 100 , 1 , 000 } with a ﬁxed r = 1 , 000 (Figure 9b) and under a tight privacy budget (cid:15) X
Revised: Following the experiment settings in OME (Lyu et al., 2020a), with the commonly used α = 100 , when changing r ∈ { 10 , 100 , 1 , 000 , 10 , 000 } with a ﬁxed l =(Figure 9a), or when changing l ∈ { 5 , 10 , 20 , 100 , 1 , 000 } with a ﬁxed r = 1 , 000 (Figure 9b) and under a tight privacy budget (cid:15) X

Original: = 0 .
Revised: = 0 .

Original: 1 , the proportion (cid:15) corrected /(cid:15)
Revised: 1 , the proportion (cid:15) corrected /(cid:15)

Original: X signiﬁcantly changes among [4 . 6 e + 6 , 4 .
Revised: X signiﬁcantly changes among [4 . 6 e + 6 , 4 .

Original: 6 e + 9] .
Revised: 6 e + 9] .

Original: In other words, the (cid:15) corrected is extremely larger than (cid:15) X , for most r and l values in practice.
Revised: In other words, the (cid:15) corrected is extremely larger than (cid:15) X , for most r and l values in practice.

Original: Since α = 100 causes the extreme privacy exaggeration, in our experiment, to compare with OME, we use α = 1 .
Revised: Since α = 100 causes the extreme privacy exaggeration, in our experiment, to compare with OME, we use α = 1 .

Original: This value is used in OME (Lyu et al., 2020a) and generates (cid:15) corrected /(cid:15)
Revised: This value is used in OME (Lyu et al., 2020a) and generates (cid:15) corrected /(cid:15)

Original: X ≈ 2 , which offers a reasonable range to apply OME in practice.
Revised: X ≈ 2 , which offers a reasonable range to apply OME in practice.

Original: Unlike OME, our mechanism does not suffer from this problem, i.e., in BitRand, (cid:15) corrected /(cid:15)
Revised: Unlike OME, our mechanism does not suffer from this problem, i.e., in BitRand, (cid:15) corrected /(cid:15)

Original: X
Revised: X

Original: = 1 , thanks to our bit-aware randomization probabilities for LDP in binary encoding ( Theorem 2 ).[Figure]
Revised: = 1 , thanks to our bit-aware randomization probabilities for LDP in binary encoding ( Theorem 2 ).

Original:  S UPPLEMENTARY THEORETICAL RESULTS
Revised: K S UPPLEMENTARY THEORETICAL RESULTS

Revised: Setting for Gaussian and Laplace mechanisms.

Revised: The Gaussian and Laplace mechanisms naturally apply an addition operation, which add noise into the data or embedded features.

Revised: Therefore, in our analysis of expected error bound comparison for an embedded feature (Figure 3), we add noise into the embedded feature following the Gaussian and Laplace mechanisms.

Revised: The sensitivity captures the magnitude by which an embedded feature can change in the worst case.

Revised: In our experiment and analysis, we use l = 10 bits in which 1 sign bit, 5 bits for the integer, and 4 bits for the fraction part.

Original: [Figure]{+Therefore, the maximum the embedded feature can be change, i.e., the sensitivity, is 2 (cid:80) 
 i = − 4 2 i . Note that, we multiply (cid:80) 4 i = − 4 2 i by 2 since when we ﬂip the sign bit, it signiﬁcantly changes the value of the embedded feature from − a to a in which a = (cid:80) 4 i = − 4 2 i . RMSE error comparison in mean estimation.+}
Revised: {+Therefore, the maximum the embedded feature can be change, i.e., the sensitivity, is 2 (cid:80) 
 i = − 4 2 i . Note that, we multiply (cid:80) 4 i = − 4 2 i by 2 since when we ﬂip the sign bit, it signiﬁcantly changes the value of the embedded feature from − a to a in which a = (cid:80) 4 i = − 4 2 i . RMSE error comparison in mean estimation.+}

Revised: To investigate how our proposed approach f -RR works with statistical query, we study our f -RR and other baselines with a mean estimation.

Original: [Figure]
Revised: We created a synthetic data that consists of N = 1 , 000 data samples { x i } Ni =1 , each of them has d =dimensions. The mean estimation is calculated over each dimension as f j ( D ) = 1 N (cid:80) Ni =1 x ij for j ∈ [1 , d ] . Root mean square error (RMSE) is used to evaluate the error between the original vector and the randomized/estimated vector.

Revised: The binary-encoding-based approaches (i.e., f -RR, corrected

Revised: LATENT, and corrected OME) achieve a signiﬁcantly small error compared with others.

Revised: As can be seen in Figure 10, f -RR obtains the smallest error, which further shows the effectiveness of our proposed mechanism.

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: K S UPPLEMENTARY E XPERIMENTAL R ESULTS
Revised: L S UPPLEMENTARY E XPERIMENTAL R ESULTS

Original: Datasets and Data Processing.
Revised: Datasets and Data Processing.

Original: We carried out our experiments on two textual datasets and two image datasets, including the AG dataset (Gulli et al., 2012), our collected Security and Exchange
Revised: We carried out our experiments on two textual datasets and two image datasets, including the AG dataset (Gulli et al., 2012), our collected Security and Exchange

Original: Commission (SEC) ﬁnancial contract dataset, the large-scale celebFaces attributes (CelebA) dataset (Liu et al., 2015), and the Federated Extended MNIST (FEMNIST) dataset (Caldas et al., 2018).
Revised: Commission (SEC) ﬁnancial contract dataset, the large-scale celebFaces attributes (CelebA) dataset (Liu et al., 2015), and the Federated Extended MNIST (FEMNIST) dataset (Caldas et al., 2018).

Original: The
Revised: The

Original: AG dataset is a collection of news articles gathered from more than 2 , 000 news sources by (Com).
Revised: AG dataset is a collection of news articles gathered from more than 2 , 000 news sources by (Com).

Original: It is categorized into four classes: world, sport, business, and science/technology classes.
Revised: It is categorized into four classes: world, sport, business, and science/technology classes.

Original: Our SEC dataset consists of over 1 , 000 contract clauses collected from contracts submitted in SEC ﬁlings 1 .
Revised: Our SEC dataset consists of over 1 , 000 contract clauses collected from contracts submitted in SEC ﬁlings 1 .

Original: The CelebA dataset consists of more than 200 , 000 celebrity images, each with 40 attributes, e.g., attractive face, big lips, big noses, black hair, etc., which are used as binary classes.
Revised: The CelebA dataset consists of more than 200 , 000 celebrity images, each with 40 attributes, e.g., attractive face, big lips, big noses, black hair, etc., which are used as binary classes.

Original: The FEMNIST dataset is built by partitioning the images in Extended MNIST (Cohen et al., 2017) based on the writer of the handwritten digits and characters.
Revised: The FEMNIST dataset is built by partitioning the images in Extended MNIST (Cohen et al., 2017) based on the writer of the handwritten digits and characters.

Original: For data preprocessing, we changed all words in the
Revised: For data preprocessing, we changed all words in the

Original: AG and SEC datasets to lower-case and removed punctuation marks.
Revised: AG and SEC datasets to lower-case and removed punctuation marks.

Original: The breakdown of the datasets is in Table 1.
Revised: The breakdown of the datasets is in Table 1.

Original: [Table]
Revised: [Table]

Original: Model Conﬁguration.
Revised: Model Conﬁguration.

Original: We use the test accuracy and the test area under the curve (AUC) as evaluation metrics.
Revised: We use the test accuracy and the test area under the curve (AUC) as evaluation metrics.

Original: Models with higher values of test accuracy and AUC are better.
Revised: Models with higher values of test accuracy and AUC are better.

Original: We use the BERT-Base (Uncased) pre-trained model (ber; Devlin et al., 2018) to extract embedded features in the AG and
Revised: We use the BERT-Base (Uncased) pre-trained model (ber; Devlin et al., 2018) to extract embedded features in the AG and

Original: SEC datasets.
Revised: SEC datasets.

Original: In the CelebA and FEMNIST datasets, we use the ResNet-18 pre-trained model (img;
Revised: In the CelebA and FEMNIST datasets, we use the ResNet-18 pre-trained model (img;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Dimension of the extracted embedded features in the AG and SEC datasets is r = 768 , and in the CelebA and FEMNIST datasets is r = 512 .
Revised: Dimension of the extracted embedded features in the AG and SEC datasets is r = 768 , and in the CelebA and FEMNIST datasets is r = 512 .

Original: For text and image classiﬁcation tasks, we use two fully connected layers on top of embedded features, each of which consists of 1 , 500 hidden neurons and uses a ReLU activation function.
Revised: For text and image classiﬁcation tasks, we use two fully connected layers on top of embedded features, each of which consists of 1 , 500 hidden neurons and uses a ReLU activation function.

Original: The output dimension is corresponding to the number of classes, i.e., 4 , 2 , 40 , and 62 in the AG, SEC, CelebA, and FEMNIST datasets.
Revised: The output dimension is corresponding to the number of classes, i.e., 4 , 2 , 40 , and 62 in the AG, SEC, CelebA, and FEMNIST datasets.

Original: SGD optimizer with the learning rate is 0 .
Revised: SGD optimizer with the learning rate is 0 .

Original: 01 in the AG and SEC datasets, 0 .
Revised: 01 in the AG and SEC datasets, 0 .

Original: 1 in the FEMNIST and CelebA datasets.
Revised: 1 in the FEMNIST and CelebA datasets.

Original: [Table]
Revised: Experimental setting for anonymization (Sun et al., 2021). In LDP-FL (Sun et al., 2021), they design a LDP mechanism to perturb the weights at the local client, then each local client applies a split and shufﬂe mechanism on the weights of local model and sends each weight through an anonymous mechanism to the cloud. The purpose of the shufﬂing mechanism is to break the linkage among the model weight updates from the same clients and to mix them among updates from other clients, making it harder for the cloud to combine more than one piece of updates to infer more information about any client. Therefore, the key idea of the shufﬂe mechanism in LDP-FL is to mitigate the privacy degradation by high data dimension and many training/query iterations.

Revised: In other words, the client anonymity is preserved, and the privacy budget will not accumulate.

Original: -Laplace
Revised: When comparing with LDP-FL, we maintain their mechanism’ spirits of no privacy accumulation. In the submission, we consider there is no privacy accumulation over the training iterations. It is equivalent to the shufﬂing step that breaks the linkage among the model weight updates with associated clients. We also used the same randomized response mechanism in the paper, which is

Revised: Eq.

Revised: ((Sun et al., 2021)), to perturb the weight.

Revised: In the revision, we added an experiment that do not consider the privacy accumulation over data dimension and training/query iterations.

Revised: This completely follows the gist of LDP-FL.

Revised: In addition, the weights we used for LDP-FL. without actual shufﬂing or splitting can be considered a lossless process, therefore the results we reported here can be counted as an upper-bound result for LDP-FL.

Original: [Table]
Revised: As shown in Table 2, we obtained the slightly higher accuracy of LDP-FL compared with f -RR on the AG dataset. The key component of LDP-FL that helps to reduce the privacy accumulation issue is the shufﬂing mechanism. However, as pointed out in (Erlingsson et al., 2019), in the real world, it is possible that the anonymizers (i.e., shufﬂer) can either be compromised or collude with the coordinating server to extract sensitive information. Even though there is a marginally lower trade-off between privacy loss and model utility compared with LDP-FL, the advantage of f -RR is that it perturbs the data only once, then used the perturbed data for training process without facing an extra privacy risk potentially caused by the compromised or colluded anonymizer.

Original: [Table]
Revised: [Table]

Original:  [Table] 
Revised: [Figure] [Table] [Table] -Laplace

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: BitRand
Revised: BitRand

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] 53.50 (cid:15)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: (cid:15)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table] [Table]
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table] [Table]
Revised: 63.31 (cid:15)

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: 53.84 (cid:15)

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 58.16 (cid:15)
Revised: 58.16 (cid:15)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 50.22 (cid:15) X =
Revised: 50.22 (cid:15) X =

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =51.
Revised: =51.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BitRand (cid:15)
Revised: BitRand (cid:15)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: = ∞ 59.
Revised: = ∞ 59.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =54.
Revised: =54.

Original: 50.91 (cid:15)
Revised: 50.91 (cid:15)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =59.
Revised: =59.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: X = 5 , (cid:15) Y =51.
Revised: X = 5 , (cid:15) Y =51.

Original: 50.21 (cid:15) X = 10 , (cid:15) Y =51.
Revised: 50.21 (cid:15) X = 10 , (cid:15) Y =51.

Original: 50.12 (cid:15) X = 10 , (cid:15) Y =
Revised: 50.12 (cid:15) X = 10 , (cid:15) Y =

Original: 50.16 (cid:15) X
Revised: 50.16 (cid:15) X

Original: = 10 , (cid:15) Y =50.
Revised: = 10 , (cid:15) Y =50.

Original: BitRand (cid:15) X = 1 ,(cid:15) Y = ∞ 59.
Revised: BitRand (cid:15) X = 1 ,(cid:15) Y = ∞ 59.

Original: 51.86 (cid:15) X
Revised: 51.86 (cid:15) X

Original: = 1 , (cid:15) Y =54.
Revised: = 1 , (cid:15) Y =54.

Original: 50.91 (cid:15) X
Revised: 50.91 (cid:15) X

Original: = 1 , (cid:15) Y = 2 .59.
Revised: = 1 , (cid:15) Y = 2 .59.

Original: 51.52 (cid:15) X
Revised: 51.52 (cid:15) X

Original: = 1 , (cid:15) Y =59.
Revised: = 1 , (cid:15) Y =59.

Original: 51.77 (cid:15) X
Revised: 51.77 (cid:15) X

Original: = 5 ,(cid:15) Y = ∞ 67.
Revised: = 5 ,(cid:15) Y = ∞ 67.

Original: 54.99 (cid:15) X
Revised: 54.99 (cid:15) X

Original: = 5 , (cid:15) Y =58.
Revised: = 5 , (cid:15) Y =58.

Original: 51.82 (cid:15) X
Revised: 51.82 (cid:15) X

Original: = 5 , (cid:15) Y =
Revised: = 5 , (cid:15) Y =

Original: [Table] [Table]
Revised: [Table] [Table]