Subject Level Differential Privacy with Hierarchical Gradient Averaging
Abstract
Subject Level Differential Privacy (DP) is a granularity of privacy recently studied in the Federated Learning (FL) setting, where a subject is deﬁned as an individual whose private data is embodied by multiple data records that may be distributed across a multitude of federation users.
This granularity is distinct from item level and user level privacy appearing in the literature.
Prior work on subject level privacy in FL focuses on algorithms that are derivatives of group DP or enforce user level Local DP (LDP) .
In this paper, we present a new algorithm – Hierarchical Gradient Averaging ( HiGradAvgDP ) – that achieves subject level DP by constraining the effect of individual subjects on the federated model.
We prove the privacy guarantee for HiGradAvgDP and empirically demonstrate its effectiveness in preserving model utility on the FEMNIST and Shakespeare datasets.
We also report, for the ﬁrst time, a unique problem of privacy loss composition, which we call horizontal composition , that is relevant only to subject level DP in FL.
We show how horizontal composition can adversely affect model utility by either increasing the noise necessary to achieve the DP guarantee, or by constraining the amount of training done on the model.
Introduction
Cross-silo Federated Learning (FL) comprises a collection of institutions collaborating to train a common Machine Learning (ML) model [13].
This allows these institutions (also called silos ) to jointly train the common model without sharing their private training data.
Interestingly, a silo’s private training data can be made up of private data of individuals (e g. health records of patients of a hospital that is participating in a FL federation).
In such instances, preservation of privacy of these individuals (also termed data subjects ) is of paramount importance.
However, the private data of a subject may be embodied by multiple data records that could be distributed across a multitude of federation silos.
As an example consider a patient visiting different hospitals for treatment of different ailments.
Each hospital contains multiple data records forming the patient’s health history.
These hospitals may decide to participate in a federation that uses their respective patients’ health history records in their training datasets.
Thus potentially distinct health history records of the same data subject can appear in the datasets of multiple hospitals.
In the end, it is the privacy of these subjects that we want to preserve in a FL federation.
The notion of subject level privacy perfectly captures this condition [19].
Subject level Differential Privacy (DP) is a recently introduced granularity of privacy enforcement proposed in the FL setting [19].
This privacy guarantee is distinct from previously studied granularities of item level privacy [1, 9] and user level privacy [20], both of which may make sense in cross-device FL settings.
Prior proposed algorithms [19] to enforce subject level DP either leverage the notion of group DP [10] or user-level Local DP (LDP) [8, 14, 30].
However, these algorithms incur signiﬁcant model utility degradation, as we demonstrate in Section 5, since they amplify the noise to the contribution of individual subjects in the parameter updates.
In this paper, we present a novel algorithm, called Hierarchical Gradient Averaging ( HiGradAvgDP ) , that enforces subject level DP in the FL setting.
The key insight behind HiGradAvgDP is that instead of amplifying the noise signal to match a subject’s contribution to parameter updates in a training minibatch, we can constrain a subject’s contribution to match a single data record’s contribution to achieve subject level DP with less noise.
We formally show that HiGradAvgDP leads to lower model utility loss for convex loss functions.
We furthermore verify these insights empirically using the FEMNIST and Shakespeare datasets [6].
We also report an interesting observation about privacy loss composition for subject level DP in the FL setting.
Speciﬁcally, we show that subject privacy loss composes across federation users in a FL training round.
We call this horizontal composition of privacy loss and show that it can either lead to increase in the noise needed to enforce subject level DP, or can constrain the amount of training done on the model.
Further exploration of solutions to the horizontal composition problem is a topic for future work.
The rest of the paper is organized as follows: We revisit the formal deﬁnition of subject level DP in Section 2.
Section 3 brieﬂy overviews existing algorithms for subject level DP in FL and then details our HiGradAvgDP algorithm, with formal arguments for its privacy guarantee.
We present our empirical evaluation in Section 5 and conclude in Section 6.
Subject Level Differential Privacy
We ﬁrst recap the deﬁnition of Differential Privacy [9].
Informally, DP bounds the maximum impact a single data item can have on the output of a randomized algorithm A .
Formally,
Deﬁnition 2.1.
A randomized algorithm A : D → R is said to be ( ε , δ )-differentially private if for any two adjacent datasets D , D (cid:48) ∈ D , and set R ⊆ R ,
[Equation]
where D , D (cid:48) are adjacent to each other if they differ from each other by a single data item.
δ is the probability of failure to enforce the ε privacy loss bound.
The above deﬁnition of DP can be easily recast in terms of data subjects to deﬁne subject level DP.
Let S be the set of subjects whose data is hosted by the federation’s users U .
Our deﬁnition of subject level DP is based on the observation that, even though the data of individual subjects s ∈ S may be physically scattered across multiple users in U , the aggregate data across U can be logically divided into its subjects in S (i.e. D U =
(cid:83) s ∈S D s ).
Deﬁnition 2.2.
Given a FL training algorithm F : D U → M , we say that F is subject level ( ε, δ )-differentially private if for any two adjacent subject sets S , S (cid:48)
⊆ S , and R ⊆ M ,
[Equation]
where S and S (cid:48) are adjacent subject sets if they differ from each other by a single subject.
Subject Level Differential Privacy with Hierarchical Gradient Averaging
We assume a federation that contains a federation server that is responsible for (i) initialization and distribution of the model architecture to the federation users, (ii) coordination of training rounds, (iii) aggregation and application of model updates coming from different users in each training round, and (iv) redistribution of the updated model back to the users.
Each federation user (i) receives updated models from the federation server, (ii) retrains the received models using its private training data, and (iii) returns updated model parameters to the federation server.
We assume that the federation users and the server behave as honest-but-curious participants in the federation: They do not interfere with or manipulate the distributed training process in any way, but may be interested in analyzing received model updates.
Federation users do not trust each other or the federation server, and must locally enforce privacy guarantees for their private data.
HiGradAvgDP is based on a federated version of the DP-SGD algorithm by Abadi et al.
DPSGD was originially not designed for FL, but can be easily extended to enforce item level DP in
FL: The federation server samples a random set of users for each training round and sends them a request to perform local training.
Each user trains the model locally using DP-SGD .
[-= 
-]
Formally, the parameter update at step t in DP-SGD can be summarized in the following equation:
[Equation]
where, B is the sampled minibatch of data items, (cid:79) L Cb is the loss function’s gradient, for data item b , clipped by the threshold C , σ is the noise scale calculated using the moments accountant method,
N is the Gaussian distribution used to calculate noise, and η is the learning rate.
Note that, in each sampled mini-batch B , the gradient for each data item is computed and clipped separately to limit the inﬂuence ( sensitivity ) of each data item on the loss function’s gradient.
The users ship back updated model parameters to the federation server, which averages the updates received from all the sampled users.
The server redistributes the updated model and triggers another training round if needed.
The original paper [1] also proposed the moments accountant method for tighter composition of privacy loss bounds compared to prior work on strong composition [11].
3.1 Alternate Algorithms for Subject Level Differential Privacy
Prior work by Marathe and Kanani [19] presents two different algorithms for subject level DP enforcement in FL.
Both algorithms are variants of the DP-SGD algorithm in FL as described above.
The ﬁrst algorithm is called LocalGroupDP , which uses the notion of group differential privacy [10] to enforce subject level DP.
The intuition here is that a subject’s contribution to gradients in a minibatch can be trivially mapped to a group of data items sampled in the minibatch, thus making group
DP a natural solution for subject level DP.
The second algorithm is UserLDP that enforces a stronger user level Local Differential Privacy (LDP) [8, 14, 30] guarantee that ampliﬁes the notion of a group to the entire sampled minibatch.
More detailed description of both algorithms and their privacy guarantee formalism appears in Appendix A and Appendix B.
3.2 Hierarchical Gradient Averaging ( HiGradAvgDP )
The key challenge to enforce subject level DP is that the following constraint seems fundamental:
To guarantee subject level DP, any training algorithm must obfuscate the entire contribution made by any subject in the model’s parameter updates.
UserLDP complies with this constraint by scaling up the noise to the entire user signal, whereas LocalGroupDP complies with the constraint by scaling up the noise to a subject’s signal in every minibatch.
While LocalGroupDP may seem like an attractive alternative to UserLDP , the former’s utility penalty due to group DP can be signiﬁcant – the privacy loss ε incurred due to group DP increases linearly with the group size, and the failure probability δ increases exponentially [10, 19].
For instance, even a group of size 2 effectively halves the available privacy budget ε for training.
Our new algorithm, called HiGradAvgDP (Algorithm 1), takes a diametrically opposite view to comply with the same constraint: Instead of scaling the noise to a subject’s group size (as is done in LocalGroupDP ), HiGradAvgDP scales down each subject
’s mini-batch gradient contribution to the clipping threshold C .
This is done in three steps: (i) collect data items belonging to a common subject in the sampled mini-batch, (ii) compute and clip gradients using the threshold C for each individual data item of the subject, and (iii) average those clipped gradients for the subject, denoted by g ( S Sa ) .
Clipping and then averaging gradients ensures that the entire subject’s gradient norm is bounded by C .
Subsequently, HiGradAvgDP sums all the per-subject averaged gradients along with the Gaussian noise, which are then averaged over the mini-batch size B .
HiGradAvgDP gets its name from this average-of-averages step.
The Gaussian noise scale σ is calculated independently at each user
u i using standard parameters – the privacy budget ε , the failure probability δ , total number of mini-batches T.R , and the sampling fraction per mini-batch B | D i | .
The calculation uses the moments accountant method to compute σ .
1 HiGradAvgDP( u ): for t 1 to T do S = random sample
B data items
D i for a ∈ subjects ( S ) do for s i ∈
S Sa do
Compute gradients: g ( s i ) =
(cid:79) L ( θ, s i )
Clip gradients: ¯ g ( s i ) =
Clip ( g ( s i ) , C )
Average subject a ’s gradients: g ( S Sa )
{+= 
+}
|S Sa | ( (cid:80) i ¯ g ( s i ))
˜ g S = 1 B ( (cid:80) a ∈ subjects ( S ) g
( S Sa )
+
N (0 , σ 2 C 2 I ))
θ =
θ − η ˜ g S return M
Loop: do
U s =
s
U s do θ i
θ = 1 s (cid:80)
i
Send M in U
To formally prove that HiGradAvgDP enforces subject level DP, we ﬁrst provide a formal deﬁnition of subject sensitivity in a sampled mini-batch.
Deﬁnition 3.1 (Subject Sensitivity) .
Given a model M , and a sampled mini-batch S of training data, we deﬁne subject sensitivity S S for S as the maximum difference caused by any single subject a ∈ subjects ( S ) in M ’s parameter gradients computed over S .
Lemma 3.1.
For every sampled mini-batch S in a sampled user u i ’s training round in HiGradAvgDP , the subject sensitivity S S for S is bounded by C ; i.e. S S ≤ | C | .
Theorem 3.2.
HiGradAvgDP locally enforces subject level ( ε , δ )
-differential privacy.
Proofs for Lemma 3.1 and Theorem 3.2 appear in Appendix D. Moreover, due to space constraints we leave the formal analysis on utility loss bounds of HiGradAvgDP , as well as UserLDP and
LocalGroupDP , to Appendix C.
3.3 Composition Over Multiple Training Rounds
Composition of privacy loss across multiple training rounds can be done by straightforward application of DP composition results, such as the moments accountant method that we use in our work.
Thus the privacy loss ε r incurred in any single training round r ampliﬁes by a factor of √ R when federated training runs for R rounds.
We note that privacy losses are incurred by federation users independently of other federation users.
Foreknowledge of the number of training rounds R lets us calculate the Gaussian noise distribution’s standard deviation σ for a privacy loss budget of ( ε , δ ) for the aggregate training over R rounds.
Composing Subject Level DP Across Federation Users
At the beginning of a training round r , each sampled user receives a copy of the global model, with parameters Θ r − 1 , which it then retrains using its private data.
Since all sampled users start retraining from the same model M Θ r − 1 , and independently retrain the model using their respective private data, parallel composition of privacy loss across these sampled users may seem to apply naturally [21].
In that case, the aggregate privacy loss incurred across multiple federation users, via an aggregation such as federated averaging, remains identical to the privacy loss ε r incurred individually at each user.
However, parallel composition was proposed for item level privacy, where an item belongs to at most one participant.
With subject level privacy, a subject’s data items can span across multiple users, which limits application of parallel privacy loss composition to only those federations where each subject’s data is restricted to at most one federation user.
In the more general case, we show that subject level privacy loss composes sequentially via the federated averaging aggregation algorithm used in our FL training algorithms.
Formally, consider a FL training algorithm F =
( F l , F g ) , where F l is the user local component, and
F g the global aggregation component of F .
Given a federation user u i , let F l : ( M , D u i )
→ P u i , where M is a model, D u i is the private dataset of user u i , and P u i is the updated parameters produced by F l .
Let F g = 1 n (cid:80)
i P u
i , a parameter update averaging algorithm over a set of n federation users u i .
Theorem 4.1.
Given a FL training algorithm F =
( F l , F g ) , in the most general case where a subject’s data resides in the private datasets of multiple federation users
u i , the aggregation algorithm
F g sequentially composes subject level privacy losses incurred by F l at each federation user.
We term this sequential composition of privacy loss across federation users as horizontal composition .
Horizontal composition has a signiﬁcant effect on the number of federated training rounds permitted under a given privacy loss budget.
Theorem 4.2.
Consider a FL training algorithm F =
( F l , F g ) that samples s users per training round, and trains the model M for R rounds.
Let F l at each participating user, over the aggregate of
R training rounds, locally enforce subject level ( ε , δ )-DP.
Then F globally enforces the same subject level ( ε , δ )-DP guarantee by training for R √ s rounds.
The main intuition behind Theorem 4.2 is that the s -way horizontal composition via F g results in an increase in training mini-batches by a factor of s .
As a result, the privacy loss calculated by the moments accountant method ampliﬁes by a factor of √ s , thereby forcing a reduction in number of training rounds by a factor of √ s to counteract the privacy loss ampliﬁcation.
This reduction in training rounds can have a signiﬁcant impact on the resulting model’s performance, as we demonstrate in Section 5.
Proofs for Theorem 4.1 and Theorem 4.2 appear in Appendix D.
An alternate approach to account for horizontal composition of privacy loss is to simply scale the number of training minibatches (called lots by Abadi et al.
[1]) by the number of federation users sampled in each training round.
The scaled minibatch (lot) count can be used by each user to privately calculate the noise scale σ at the beginning of the entire federated training process.
An increase in the number of total minibatches does lead to a signiﬁcant increase in the noise introduced in each minibatch’s gradients, resulting in model performance degradation.
Empirical Evaluation
We implemented UserLDP , LocalGroupDP , and HiGradAvgDP , and a version of the DP-SGD algorithm by Abadi et al.
[1] that enforces item level DP in the FL setting ( LocalItemDP ).
We also compare these algorithms with a FL training algorithm, FedAvg [16], that does not enforce any privacy guarantees.
All our algorithms are implemented in our distributed FL framework built on distributed PyTorch.
We focus our evaluation on Cross-Silo FL [13], containing 16 federation users (silos), which we believe is the most appropriate setting for the subject level privacy problem.
We use the FEMNIST and Shakespeare datasets [6] for our evaluation.
Due to space constraints, we report simple test accuracy results here (Figure 1) and the more exhaustive evaluation appears in Appendix E.
In FEMNIST, the hand-written numbers and letters can be divided based on authors, which ordinarily serve as federation users in FL experiments by most researchers.
In Shakespeare, each character in the Shakespeare plays serves as a federation user.
In our experiments however, the FEMNIST authors and Shakespeare play characters are treated as data subjects.
To emulate the cross-silo FL setting, we report evaluation on a 16-user federation.
We use the CNN model on FEMNIST appearing in the LEAF benchmark suite [6] as our target model to train.
More speciﬁcally, the model consists of two convolution layers interleaved with
FEMNIST
[Figure]
ReLU activations and maxpooling, followed by two fully connected layers before a ﬁnal log softmax layer.
For the Shakespeare dataset we use a stacked LSTM model with two linear layers at the end.
In our implementations of UserLDP , LocalGroupDP , and HiGradAvgDP , we used the privacy loss horizontal composition accounting technique that reduces the number of training rounds by √ s , where s is the number of sampled users per training round.
We experimented with the alternative approach that scales up the number of minibatches by s to calculate a larger noise scale σ , but this approach consistently yielded worse model utility than our ﬁrst approach.
Hence here we report only the performance of our ﬁrst approach.
Figure 1 shows performance of the models trained using the algorithms.
FedAvg performs the best since it does not incur any DP enforcement penalties.
Item level privacy enforcement in LocalItemDP results in performance degradation of 8% for FEMNIST and 22% for Shakespeare.
The utility cost of user level LDP in UserLDP is quite clear from the ﬁgure.
This cost is also reﬂected in the relatively high observed loss for the respective model.
LocalGroupDP performs signiﬁcantly better than UserLDP , but worse than LocalItemDP , by 15% on FEMNIST, and 18% on Shakespeare.
The reason for LocalGroupDP ’s worse performance is clear from Figure 1(c) and (g): the group size for a mini-batch tends to be dominated by 3 on both FEMNIST and Shakespeare, which cuts the privacy budget for these mini-batches by a factor of 3 , leading to greater Gaussian noise, which in turn leads to model performance degradation.
HiGradAvgDP performs competitively with LocalItemDP for the 25 and 50 rounds it is trained for on FEMNIST and Shakespeare respectively.
Figure 1 (d) and (h) show that instances of sampling multiple data items corresponding to the same subject in a single mini-batch are relatively low – the number of distinct subjects sampled per mini-batch of 512 for FEMNIST averages to 475 , and per mini-batch of 100 for Shakespeare averages to 86 .
As a result, HiGradAvgDP incurs insigniﬁcant performance degradation for both datasets.
However, the training round restriction does result in degradation of the ﬁnal model produced by HiGradAvgDP compared to LocalItemDP :
For
FEMNIST, HiGradAvgDP gives 75 .
24% prediction accuracy after 25 rounds compared to 77 .
96% accuracy after 100 rounds with LocalItemDP .
For Shakespeare, HiGradAvgDP gives 41 .
58% model accuracy after 50 rounds compared to 45 .
91% accuracy with LocalItemDP after 200 rounds.
Conclusion
In this paper we presented a novel algorithm called Hierarchical Gradient Averaging ( HiGradAvgDP ) that enforces the recently studied subject level DP guarantee in the FL setting [19].
We showed that the approach taken by HiGradAvgDP – scale a subject’s signal down to a single data item’s signal – instead of prior work’s approach to scale the noise to a subject’s signal, leads to much better model utility.
We also studied the novel problem of horizontal composition of privacy loss for subjects in the FL setting, which can further degrade model utility.
We leave mitigation of this utility degradation to future research.
A User Level Local Differential Privacy ( UserLDP )
In general, user level privacy [20] does not guarantee subject level privacy.
However, we observe that a stronger privacy guarantee, called Local Differential Privacy (LDP) [8, 14, 30], enforced at user granularity, is sufﬁcient to guarantee subject level privacy.
There are strong parallels between the traditional LDP setting, where a data analyst can get access to the data only after it has been perturbed, and privacy in the FL setting, where the federation server gets access to parameter updates from users after they have been locally perturbed by the users.
In fact, in the FL setting [27], LDP is a much stronger privacy guarantee than item level or user level DP in that it obfuscates the entire signal from a user to the extent that an adversary, even the federation server, cannot tell the difference between the signals coming from any two different users.
Deﬁnition A.1.
We say that FL algorithm F l : D U → M is user level ( ε , δ )-locally differentially private , where D U is the dataset domain of users in set U , and M is the model parameter domain, if for any two users u 1 , u 2 ∈ U , and S ⊆ M ,
[Equation]
where D u 1 and D u 2 are the datasets of users u 1 and u 2 respectively.
User level LDP is a stronger privacy guarantee than subject level DP.
More formally,
Theorem A.1.
User level ( ε , δ )-local differential privacy entails ( ε , δ )-subject level differential privacy.
Proof.
The deﬁnition of user level local DP can be easily reframed as a group differential privacy instance, where groups are as large as the entire dataset of each user.
More speciﬁcally, Equationis the deﬁnition of ( ε , δ )-group differential privacy for groups of size g = max ( | D u 1 | , | D u 2 | ) .
Beyond this observation, a simple application of Theorem B.3 proves ( ε , δ )-subject level differential privacy.
We now present a new user level ( ε , δ )-LDP algorithm called UserLDP .
The underlying intuition behind this algorithm is to let the user locally inject enough noise to make its entire signal indistinguishable from any other user’s signal.
In every training round, each federation user enforces user level LDP independently of the federation and any other users in the federation.
The federation server simply averages parameter updates received from users and broadcasts the new averaged parameters back to the users.
UserLDP ’s pseudo code appears in Algorithm 2.
UserLDP appears signiﬁcantly similar to DP-SGD .
The key difference is that while DP-SGD computes noise proportional to the gradient contribution of any single data item in a mini-batch, UserLDP computes noise proportional to the gradient contribution of the entire mini-batch, thus obfuscating the entire signal from the mini-batch.
To guarantee
DP, we need to ﬁrst cap the sensitivity of each user
u i ’s contribution to parameter updates.
To that end, we focus on change affected by any mini-batch b trained at u i .
Lemma A.2.
For every mini-batch b of a sampled user u i ’s training round in UserLDP , the sensitivity S b of the computed parameter gradient is bounded by C ; i.e. S b ≤ | C | .
Proof.
The gradient clipping step of UserDPSGD() forces the bound on the gradient: (cid:107) g ( S ) (cid:107) 2 ≤ C .
Thus the sensitivity of parameter gradient S b ≤ | C | .
Theorem A.3.
UserLDP enforces user level ( ε , δ )-local differential privacy for each mini-batch b .
Proof.
For any two users u 1 and u 2 , the parameter gradient for each mini-batch is identically bounded by C .
The Gaussian noise added per mini-batch parameter gradient is also drawn from the same distribution N (0 , σ 2 C 2 I ) scaled to the gradient bound C .
Over T mini-batches, the cumulative parameter gradient is bounded by TC .
Thus the aggregate sensitivity of parameter gradients over a training round ( T mini-batches) for both u 1 and u 2 is TC .
For every training round, we use the moments accountant technique [1] to compute the correct noise scale σ given a privacy budget of ε , and a DP failure probability of δ .
The resulting parameter updates from both u 1 and u 2 received by the federation server in a training round are effectively locally randomized [14].
Parameter updates from both u 1 and u 2 , as observed by the federation server, can each be broken down into a signal bounded by | TC | (we can ignore the scaling factor of the learning rate η since it is identical for all users), and the cumulative noise from the distribution N (0 , Tσ 2 C 2 I ) (by linear composition of random variables with identical Gaussian distributions).
More precisely, let C be the change in parameters affected by any user u i .
Then
[Equation]
[Equation]
Also note that a dataset containing just u 1 is adjacent to a dataset containing just u 2 since they differ from each other in just one data item.
Thus we can apply the classic proof of ( ε , δ )-DP for the
Gaussian mechanism [10] Theorem A.1 to show that
[Equation]
We can further extend this guarantee over multiple training rounds with a σ computed correctly using the moments accountant DP-composition technique.
Intuitively, the parameter updates M u 1 and M u 2 computed by any single mini-batch at users uand u 2 respectively, can be viewed as locally randomized responses [30] that satisfy Equation 4.
Applying standard DP composition results [1, 11] to Theorem A.3, and combining it with Theorem
A.1 locally proves subject level DP guarantee for UserLDP over individual training rounds.
B Locally Enforced Group Level Differential Privacy ( LocalGroupDP )
While user level LDP is a stronger guarantee than subject level privacy, LDP in general is known to induce excessive noise in the training process, leading to signiﬁcant utility degradation in the trained model [8, 14].
Intuitively, user level LDP is guaranteeing privacy at a coarser granularity (user level) as opposed to granularity of individual data subjects.
As a result, we need to ﬁnd better alternatives that more precisely calibrate noise proportional to a data subject’s inﬂuence on training.
A direct method to attain that is by obfuscating the effects of the group of data items belonging to the same subject.
We can apply formalism of group differential privacy [10] to achieve this group level obfuscation.
Formally (from [10]),
Theorem B.1.
Any ( ε, δ )-differentially private randomized algorithm
A is ( gε, ge ( g − 1) ε δ )- differentially private for groups of size g .
That is, given two g -adjacent datasets D and D (cid:48) , and
R ∈ M , where M is the output space domain,
[Equation]
where D and D (cid:48) are g -adjacent if they differ from each other in g data items.
Parameters
Set of n users U
= u i u 2 , ...
u n ; D i , dataset of user u i ; M , the model to be trained; θ , model M users U s
1 LocalGroupDP( u i ): for t = 1 to T do
S = random sample of B data items from
D i for s i ∈ S do
Compute gradients: g ( s i ) =
(cid:79) L ( θ, s i )
Clip gradients: ¯ g ( s i ) =
Clip ( g ( s i ) , C )
Z = LrgGrpCnt ( S ) ˜
g s = 1 B ( (cid:80) i ¯ g ( s i ) + N (0 , σ 2 Z C 2 I ))
θ = θ − η ˜ g s return M 13 Server Loop: for r = 1 to R do U s = sample s users from U for u i ∈
U s do θ i =
LocalGroupDP ( u i ) θ
= 1 s (cid:80) i θ i Send M to all users in U Clearly, group DP incurs a big linear penalty on the privacy loss ε , and an even bigger penalty in the failure probability ( ge ( g − 1) ε δ ).
Nevertheless, if g is restricted to a small value (e.g. 2 ) the group DP penalty may be acceptable.
Theorem B.1 is a bi-directional implication.
So it can be restated as follows:
Corollary B.2.
Any ( E , ∆ )-group differentially private algorithm A , for a group size of g , is ( E /g, ∆ / ( ge ( g − 1) E g ) -differentially private.
In the FL setting, subject level DP immediately follows from group DP for every sampled mini-batch of data items at every federation user.
Let S be a sampled mini-batch of data items at a user u i , and
M be the domain space of the ML model being trained in the FL setting.
Theorem B.3.
Let training algorithm
A g : S → M be group differentially private for groups of size g , and l be the largest number of data items belonging to any single subject in S .
If l ≤ g , then
A g is subject level differentially private.
Composition of group DP guarantees over multiple mini-batches and training rounds also follows established DP composition results [1, 11, 22].
For instance, the moments accountant method by Abadi et al.
[1] shows that given an ( ε , δ )-DP gradient computation for a single mini-batch, the full training algorithm, which consists of T mini-batches and a mini-batch sampling fraction of q , is ( O ( qε √ T ) , δ ) -differentially private.
Theorem B.1 implies that the same algorithm is ( O ( gqε √ T, ge ( g − 1) ε δ ) -differentially private for a group of size g .
We now present our new FL training algorithm, LocalGroupDP , that guarantees group DP.
We make a critical assumption in LocalGroupDP : Each user can determine the subject for any of its data items.
Absent this assumption, the user may need to make the worst case assumption that all data items used to train the model belong to the same subject.
On the other hand, these algorithms are strictly local , and do not require that the identity of the subjects be resolved across users.
LocalGroupDP ( Algorithm 3) enforces subject level privacy locally at each user.
Like prior work [1, 20, 24], we enforce DP in LocalGroupDP by adding carefully calibrated Gaussian noise in each mini-batch’s gradients.
Each user clips gradients for each data item in a mini-batch to a clipping threshold C prescribed by the federation server.
The clipped gradients are subsequently averaged over the mini-batch.
The clipping step bounds the sensitivity of each mini-batch’s gradients to C .
To enforce group DP, LocalGroupDP also locally tracks the item count of the subject with the largest number of items in the sampled mini-batch.
This count determines the group size needed to enforce group DP for that mini-batch.
This group size, Z in Algorithm 3, helps determine the noise scale σ Z , given the target privacy parameters ( E , ∆) over the entire training round.
More speciﬁcally, we use the moments accountant method and Corollary B.2 to calculate σ for ε = E /Z , and δ = ∆ / ( Ze ( Z − 1) E Z ) .
Note that the value of Z can vary between mini-batches, due to which we represent the noise scale as σ Z in the pseudo code.
σ Z is computed using the moments accountant method.
The rest of the parameters to calculate σ Z – E , ∆ , total number of mini-batches ( T.R ), and sampling fraction ( B/total dataset size ) – remain the same throughout the training process.
LocalGroupDP enforces ( E /Z, ∆ / ( Ze ( Z − 1) E Z ) -differential privacy, which by Corollary B.2 implies ( E , ∆ )-group differential privacy, hence subject level DP by Theorem B.3.
C Utility Loss
Our formal analysis leverages a long line of former work on differentially private empirical risk minimization (ERM) [2, 3, 7, 8, 12, 15, 24, 25, 26, 28].
In particular, we extend the notation of and heavily base our formal analysis on work by Bassily et al.
[2], applying it to subject level DP in general, with specializations for our individual algorithms.
Let Z denote the data domain, and D denote a data distribution over Z .
We assume a L -Lipschitz convex loss function (cid:96) :
R d × Z → R that maps a parameter vector w ∈ W , where W ⊂ R
d is a convex parameter space, and a data point z ∈ Z , to a real value.
Deﬁnition C.1 ( α -Uniform Stability [2, 5]) .
Let α > 0 .
A (randomized) algorithm
A : Z n → W is α -uniformly stable (w.r.t loss (cid:96) :
W × Z → R ) if for any pair S, S (cid:48)
∈
Z n differing in at most one data point, we have
[Equation]
Deﬁnition C.2 ( ( k, α ) -Uniform Stability) .
Let α > 0 .
A (randomized) algorithm
A : Z n → W is said to be ( k, α ) -uniformly
stable (w.r.t loss (cid:96) :
W × Z → R ) if for any pair S, S (cid:48)
∈
Z n differing in at most k data points, we have
[Equation]
We use ( k, α ) -uniform stability to represent the effect of a data subject with cardinality k in the dataset.
Thus algorithm A is ( k, βk ) -uniformly
stable if
[Equation]
Lemma C.1.
A (randomized) algorithm
A : Z n → W is ( k, βk ) -uniformly
stable iff it is βkuniformly stable.
Proof.
Consider sets S 0 , S 1 , S 2 , ..., S k ⊂ Z such that S i
=
S
i − 1 ∪ { x i } , for all 1 ≤ i ≤ k , where x
i ∈ Z .
In other words, S i contains a single additional data point than S
i − 1 .
Assume that A : Z n → W is ( k, βk )
-uniformly stable.
Then we have
[Equation]
By i.i.d.
and symmetry assumptions, we get ∀ i ∈ 1 , 2 , ..., k
[Equation]
For the other direction of the iff we use the same sets S 0 , S 1 , S 2 , ..., S k , and assume ∀ i ∈ 1 , 2 , ..., k
[Equation]
Hence,
[Equation]
C.1 General Utility Loss for Subject Level Privacy
Given the parameter vector w ∈ W , dataset S = s 1 , s 2 , ..., s n , and loss function (cid:96) , we deﬁne the empirical loss of w as ˆ L ( w ; S ) (cid:44) 1 n (cid:80) ni
=1 (cid:96) ( w , s i ) , and the excess empirical loss of w as ∆ ˆ L ( w ; S ) (cid:44) ˆ L ( w ; S ) − min ˜ w ∈ W ˆ L ( ˜ w ; S ) .
Similarly we deﬁne the population loss of w ∈ W w.r.t.
loss (cid:96) and a distribution D over Z as L ( w ; D ) (cid:44) E
z
∼D [ (cid:96) ( w , z )] .
The excess population loss of w is deﬁned as ∆ L ( w ; D ) (cid:44) L ( w ; D ) − min ˜ w ∈ W L ( ˜ w ; D ) .
Lemma C.2 (from [2]) .
Let A : Z n → W be a βk -uniformly stable algorithm w.r.t.
loss (cid:96) :
W × Z → R .
Let D be any distribution over Z , and let S ∼ D n .
Then,
[Equation]
Let A be a L -Lipschitz convex function that uses dataset S to generate an approximate minimizer ˆ w S ∈ W for L ( . ; D ) .
Thus the accuracy of A is measured in terms of expected excess population loss
[Equation]
Lemma C.3.
Let A SDP be a L -Lipschitz randomized algorithm that guarantees subject level ( ε, δ ) -
DP.
Let T be the number of training iterations, m the minibatch size per training step, and η the learning rate.
Then, A SDP is ( κ, α ) -uniformly stable, where κ is the expected number of data items for any subject s p appearing in A SDP ’s training dataset, and
α =
L 2 ( T +1) η n .
Proof.
Consider dataset S comprising data items of n s subjects s 1 , ..., s p − 1 , s p , s p +1 , ..., s n s , and dataset S (cid:48) comprising data items of n s − 1 subjects s 1 , ..., s p − 1 , s p +1 , ..., s n s ; i.e. S and S (cid:48) differ from each other by a single data subject s p .
Let number of data items per subject | s i | > 0 .
Let w 0 , w 1 , ... w T and w (cid:48) 0 , w (cid:48) 1 , ..., w (cid:48)
T be the parameter values of A SDP corresponding to T training steps taken over input datasets S and S (cid:48) respectively.
Let ξ t (cid:44) w t − w (cid:48) t for any t ∈
[ T ] .
Assume random sampling with replacement for a minibatch of data items.
Let r be the number of data items in a sampled minibatch of size m that belong to subject s p .
Then, by the nonexpansiveness property of the gradient update step, we have
[Equation]
Note that r is a binomial random variable.
Thus the expected value of r , i.e. E [ r ] = m κn , where κ = E
[ | s p | ] .
Thus κ depends on the underlying data distribution D .
For instance, if D is a uniform distribution, κ = nn s , where n s is the number subjects in S .
Assuming (cid:107) ξ 0 (cid:107) = 0 , taking expectation and using the induction hypothesis, we get
[Equation]
Now let ¯ w T
=
(cid:80) Tt =1 w t and ¯ w (cid:48) T = (cid:80)
Tt =1 w (cid:48) t .
Since (cid:96) is L -Lipschitz, for every z ∈
Z we get
[Equation]
n
Note that the above bound is a scaled version (by κ ) of the recently shown bound for item level
DP [2].
Thus, intuitively in our case, the smaller the number of data items per subject in a dataset, the closer our bound is to that of item level DP.
Our bound is identical to the item level DP bound in the extreme case where each subject has just one data item in the dataset.
From Lemma C.3, Equation 8 and Equation 9, and substituting k = κ and β =
L 2 ( T +1) ηκ n in Equation 8, we get
Theorem C.4.
Let A SDP be a L -Lipschitz randomized algorithm that guarantees subject level ( ε, δ ) -DP.
Then its excess population loss is bounded by
[Equation]
Interestingly, the above inequality appears to be identical to the excess population loss bound of work by Bassily et al.
on item level DP [2].
However, only the third RHS term is identical, and the ﬁrst two RHS terms evaluate to different quantities for all of our algorithms as we show below.
C.2 Utility Loss for LocalGroupDP and UserLDP
We now formally show how LocalGroupDP ampliﬁes the Gaussian noise that factors directly into excess population loss ∆ L .
Lemma C.5.
Let W be the M -bounded convex parameter space for LocalGroupDP , and S ∈ Z n be the input (training) dataset.
Let ( ε, δ ) be the subject level DP parameters for LocalGroupDP , q be the minibatch sampling ratio, and d the model dimensionality.
Then, for any η > 0 , the excess empirical loss of LocalGroupDP is bounded by
[Equation]
Proof.
From the classic analysis of gradient descent on convex-Lipschitz functions [2, 23], we get
[Equation]
where the last term on the right hand side of the inequality is the additional empirical error due to that privacy enforcing noise.
By Theorem 1 from [1], the term σ is lower bounded by
[Equation]
for item level DP.
Extending the bound to group level DP, for groups of size k , by substituting ( ε, δ ) with ( kε, ke ( k − 1) ε δ ) gives us
[Equation]
We get the theorem’s inequality by substituting σ as above.
Combining Lemma C.5 with Theorem C.4 gives us
Theorem C.6.
The excess population loss of A LocalGroupDP is satisﬁed by
[Equation]
n
Note that the noise term ampliﬁes quadratically with group size k , which leads to rapid utility degradation with increasing group size.
The excess population loss measure for UserLDP can be obtained by simply replacing the group size term k to the size of the minibatch m , which clearly leads to signiﬁcantly greater noise ampliﬁcation.
Formally,
Theorem C.7.
The excess population loss of A UserLDP is satisﬁed by
[Equation]
n
C.3 Utility Loss for HiGradAvgDP
Recall that unlike LocalGroupDP , HiGradAvgDP does not scale up the noise to the group size of a subject in a minibatch.
It instead scales down the gradients of all data items of the subject to a single data item’s gradient bounds (established by the clipping threshold).
As a result, the noise ampliﬁcation we showed for LocalGroupDP does not exist for HiGradAvgDP .
However, scaling down the gradient signal of a subject does indeed affect HiGradAvgDP ’s utility.
To show the effect formally we go back to the classic analysis of gradient descent for convex-Lipschitz functions, Lemma 14 .in
Let w ∗ = argmin w ∈ W L ( w , S ) .
Given that ˆ L is a convex L -Lipschitz function, from [23] we have
[Equation]
Consider w t +1 =
w t + η v t , where v t = (cid:53)
ˆ L ( w t ) , and η is the learning rate.
Lemma C.8.
Consider algorithm A − HiGradAvgDP that performs the same steps as HiGradAvgDP except for the noise injection step (at line 12 of Algorithm 3 ).
Let ˆ L − HiGradAvgDP ( w ; S ) be the LLipschitz continuous empirical loss function, and W be the M -bounded convex parameter space for A − HiGradAvgDP .
If k is the expected number of data items per subject in a sampled minibatch, then
[Equation]
Proof.
Consider
[Equation]
Summing the equality over t and collapsing the ﬁrst term on the right hand side gives us
T (cid:88)
[Equation]
assuming w 1 = 0 .
Since W is M bounded and ˆ L − HiGradAvgDP is L -Lipschitz, combining the above with Equation [Equation] we get
[Equation]
Now reintroducing the noise in HiGradAvgDP (at line 12 in Algorithm 3 ), with ˆ L HiGradAvgDP ( w ; S ) as the L -Lipschitz continuous loss function of HiGradAvgDP , we get
[Equation]
where the last term of the right hand side is the additional empirical error due to the privacy enforcing noise [2].
Combining the above inequality with Theorem C.4 we get
Theorem C.9.
The excess population loss of A HiGradAvgDP is satisﬁed by
[Equation]
where σ takes a value satisfying Equation 10.
Note that the noise term for HiGradAvgDP is identical to that of the DP-SGD algorithm’s noise term that enforces item level privacy guarantee [1].
However, the ﬁrst term on the right hand side of the inequality scales linearly with k , the expected number of data items per subject.
Thus we should expect some utility loss compared to DP-SGD for k > 1 .
At k = 1 HiGradAvgDP performs identically to DP-SGD .
D Additional Proofs
Proof for Lemma 3.1: Clipping gradients for each data item belonging to subject a ∈ S before averaging the clipped gradients over the S Sa ensures that the averaged gradients’ L2-norm is bounded by C .
Hence S S ≤ | C | .
Proof for Theorem 3.2: Bounding subject sensitivity S S ≤ | C | and scaling the Gaussian noise to that sensitivity bound clearly results in ( ε , δ )-DP guarantee in gradient computation for each subject of each mini-batch S .
The mini-batch wide averaging of gradients ˜
g S done using the mini-batch size B is justiﬁed since the per subject gradients’ average can be restated as aggregation of scaled down gradients for data items corresponding to a subject a ; i.e. g ( S Sa )
= (cid:80) i ¯ g ( s i ) |S
Sa | .
This gives us
B distinct gradient quantities for the data items in the sampled mini-batch S , and averaging these quantities requires the term B in the denominator of the expression that computes ˜ g S .
We can apply the moments accountant method for privacy budget composition to extend the subject level ( ε , δ )-
DP guarantee over T mini-batches in a training round, aggregated over R training rounds.
Thus
HiGradAvgDP locally enforces subject level ( ε , δ )-differential privacy.
Proof for Theorem 4.1: Assume two distinct users u 1 and u 2 in a federation that host private data items of subject s .
Let ε 1 and ε 2 be the respective subject privacy losses incurred by the two users during a training round.
It is straightforward to see that, in the worst case, data items of s at users u 1 and u 2 can affect disjoint parameters in M .
Thus parameter averaging done by F g simply results in summation and scaling of these disjoint parameter updates.
As a result, the privacy losses, ε 1 and ε 2 incurred by uand u 2 respectively are retained to their entirety by F g .
In other words, privacy losses incurred for subject s at users u 1 and u 2 compose sequentially.
Proof for Theorem 4.2: The proof of training round constraints on horizontal composition can be broken down into two cases: First, each user in the federation locally trains for exactly T minibatches per training round, with exactly the same mini-batch sampling fraction q .
Since horizontal composition is equivalent to sequential composition in the worst case, the moments accountant method shows us that the resulting algorithm will be ( O ( qε √ TRs ) , δ ) -differentially private.
To compensate for the √ s factor scaling of the privacy loss, F can be executed for R √ s training rounds, yielding a ( O ( qε √ TR ) , δ ) -differentially private algorithm.
In the second case, each user u i may train for a unique number of mini-batches per training round, with a unique mini-batch sampling fraction dictated by u
i ’s private dataset.
Let T 1 , T 2 , ..., T s , and q 1 , q 2 , ..., q s be the number of mini-batches per training round and mini-batch sampling fraction for the sampled users u 1 , u 2 , ..., u s respectively.
All our algorithms 2, 3, and 1 locally enforce subject level ( O ( q i ε √ T i R ) , δ )-DP at each user
u i .
Privacy enforcement is done independantly at each federation user u i .
Furthermore, note that the privacy loss is uniformly apportioned among training rounds.
Let E
= O
( q i ε √ T i R ) .
Note that
E is identical for each user u i in the federation.
Thus if E is the total privacy loss budget over R training rounds, a sampled user incurs ε r = E
/R privacy loss in a single training round r .
Similarly, each of the s sampled users in round r incurs identical privacy loss ε r despite having different minibatches per training round T i and mini-batch sampling fractions q i .
As noted earlier, these privacy losses compose horizontally (sequentially) via F g over s users, leading to privacy loss ampliﬁcation by a factor of √ s as per the moments accountant method.
To compensate for this privacy loss ampliﬁcation, F can be executed for R √ s training rounds.
E Empirical Evaluation in More Detail
We implemented all our algorithms UserLDP , LocalGroupDP , and HiGradAvgDP , and a version of the DP-SGD algorithm by Abadi et al.
[1] that enforces item level DP in the FL setting ( LocalItemDP ).
We also compare these algorithms with a FL training algorithm, FedAvg [16], that does not enforce any privacy guarantees.
All our algorithms are implemented in our distributed FL framework built on distributed PyTorch.
We focus our evaluation on Cross-Silo FL [13], which we believe is the most appropriate setting for the subject level privacy problem.
We use the FEMNIST and Shakespeare datasets [6] for our evaluation.
In FEMNIST, the hand-written numbers and letters can be divided based on authors, which ordinarily serve as federation users in FL experiments by most researchers.
In Shakespeare, each character in the Shakespeare plays serves as a federation user.
In our experiments however, the FEMNIST authors and Shakespeare play characters are treated as data subjects.
To emulate the cross-silo FL setting, we report evaluation on a 16-user federation.
We use the CNN model on FEMNIST appearing in the LEAF benchmark suite [6] as our target model to train.
More speciﬁcally, the model consists of two convolution layers interleaved with
ReLU activations and maxpooling, followed by two fully connected layers before a ﬁnal log softmax layer.
For the Shakespeare dataset we use a stacked LSTM model with two linear layers at the end.
We use 80% of the training data for training, and 20% for validation.
Test data comes separately in FEMNIST and Shakespeare.
Training and testing was done on a local GPU cluster comprisingnodes, each containing 8 Nvidia Tesla V100 GPUs.
We extensively tuned the hyperparameters of mini-batch size B , number of training rounds T , gradient clipping threshold C , and learning rate η .
The ﬁnal hyperparameters for FEMNIST were:
B = 512 , T = 100 , C = 0 .
001 , and learning rates η of 0 .
001 and 0 .
01 for the non-private and private FL algorithms respectively.
Shakespeare hyperparameters were: B = 100 , T = 200 ,
C = 0 .
00001 , and learning rates η of 0 . 0002 and 0 .
01 for the non-private and private FL algorithms.
In our implementations of all our algorithms UserLDP , LocalGroupDP , and HiGradAvgDP , we used the privacy loss horizontal composition accounting technique that reduces the number of training rounds by √ s , where s is the number of sampled users per training round.
We experimented with the alternative approach that scales up the number of minibatches by s to calculate a larger noise scale σ , but this approach consistently yielded worse model utility than our ﬁrst approach.
Hence here we report only the performance of our ﬁrst approach.
FEMNIST
[Figure]
E.1 FEMNIST and Shakespeare Performance
We ﬁrst conduct an experiment that reports average test accuracy and loss at the end of each training round, over a total of 100 and 200 training rounds for FEMNIST and Shakespeare respectively.
The
FEMNIST dataset contains 3500 subjects, and the Shakespeare dataset contains 660 subjects.
In both datasets each subject comprises hundreds of data items.
Each subject’s data items are uniformly distributed among the 16 federation users.
Figure 2 shows performance of the models trained using our algorithms.
FedAvg performs the best since it does not incur any DP enforcement penalties.
Item level privacy enforcement in LocalItemDP results in performance degradation of 8% for FEMNIST and 22% for Shakespeare.
The utility cost of user level LDP in UserLDP is quite clear from the ﬁgure.
This cost is also reﬂected in the relatively high observed loss for the respective model.
LocalGroupDP performs signiﬁcantly better than UserLDP , but worse than LocalItemDP , by 15% on FEMNIST, and 18% on Shakespeare.
The reason for LocalGroupDP ’s worse performance is clear from Figure 2(c) and (g): the group size for a mini-batch tends to be dominated by 3 on both FEMNIST and Shakespeare, which cuts the privacy budget for these mini-batches by a factor of 3 , leading to greater Gaussian noise, which in turn leads to model performance degradation.
HiGradAvgDP performs competitively with LocalItemDP for the 25 and 50 rounds it is trained for on FEMNIST and Shakespeare respectively.
Figure 2 (d) and (h) show that instances of sampling multiple data items corresponding to the same subject in a single mini-batch are relatively low – the number of distinct subjects sampled per mini-batch of 512 for FEMNIST averages to 475 , and per mini-batch of 100 for Shakespeare averages to 86 .
As a result, HiGradAvgDP incurs insigniﬁcant performance degradation for both datasets.
However, the training round restriction does result in degradation of the ﬁnal model produced by HiGradAvgDP compared to LocalItemDP :
For
FEMNIST, HiGradAvgDP gives 75 .
24% prediction accuracy after 25 rounds compared to 77 .
96% accuracy after 100 rounds with LocalItemDP .
For Shakespeare, HiGradAvgDP gives 41 .
58% model accuracy after 50 rounds compared to 45 .
91% accuracy with LocalItemDP after 200 rounds.
E.2 Effect of Subject Data Distribution
While evaluation of our algorithms using a uniform distribution of subject data among federation users is a good starting point, often times the data distribution is non-uniform in real world settings.
To emulate varying subject data distributions, we conduct experiments on the FEMNIST dataset where subject data is distributed among federation users according to the power distribution
[Equation]
[Figure] Figure 3 shows performance of the models trained using our algorithms over varying subject data distributions of FEMNIST.
As expected, different data distributions clearly do not signiﬁcantly affect
FedAvg , LocalItemDP , and User-Local-SGD .
However, performance of the model trained using
LocalGroupDP degrades substantially as the unevenness of data distribution increases, resulting in test accuracy under 50% for α = 16 .
This degradation is singularly attributable to growth in subject group size per mini-batch – the average group size per mini-batch ranges from 3 when α = 2 to 6 when α = 16 .
This increase in group size signiﬁcantly reduces the privacy budget leading to increase in Gaussian noise that restricts test accuracy.
On the other hand, HiGradAvgDP appears to be much more resilient to non-uniform subject data distributions among federation users – test accuracy drops by just about 5% from α = 1 ( 75 . 84% accuracy) to α = 16 ( 71 . 89% accuracy).
The corresponding subjects per minibatch observed in our experiments goes from an average of 475 to 310 respectively (not reported in detail due to space constraints)
[Figure]
E.3 Small-FEMNIST Performance
[Figure]
F r e q u e n c y
[Figure]
HiGradAvgDP appears to generally perform well when the number of subjects in the federation is sufﬁciently large.
To study the effects of fewer subjects in a federation we experimented with a trimmed down version of FEMNIST, called Small-FEMNIST, that contains just the ﬁrst 350 subjects in the aggregate dataset.
For these experiments we reduced mini-batch size to 256 .
Figureshows the performance of our algorithms on Small-FEMNIST.
We note a substantial drop in the performance of LocalGroupDP , which can be explained by the increase in subject group size (to an average of 4 ) as can be seen in Figure 4 (c).
There is also a noticable drop in performance of
HiGradAvgDP , which is attributable to a decrease in distinct subjects occuring per mini-batch (from
Figure 4 (d)).
This drop in number of subjects adversely affects performance of models trained using the algorithms that enforce subject level privacy.