LAMP: Extracting Text from Gradients with Language Model Priors
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recent work shows that sensitive user data can be reconstructed from gradientupdates, breaking the key privacy promise of federated learning.
While successwas demonstrated primarily on image data, these methods do not directly transferto other domains such as text.
In this work, we propose LAMP, a novel attacktailored to textual data, that successfully reconstructs original text from gradients.
Our attack is based on two key insights: (i) modelling prior text probability via anauxiliary language model, guiding the search towards more natural text, and (ii)alternating continuous and discrete optimization which minimizes reconstructionloss on embeddings while avoiding local minima via discrete text transformations.
Our experiments demonstrate that LAMP is signiﬁcantly more effective than priorwork: it reconstructs 5x more bigrams and 23% longer subsequences on average.
Moreover, we are ﬁrst to recover inputs from batch sizes larger than 1 for textualmodels.
These ﬁndings indicate that gradient updates of models operating ontextual data leak more information than previously thought.
Introduction
Federated learning [21] is a widely adopted framework for training machine learning models in adecentralized way.
Conceptually, the method allows training of highly accurate models while offeringa level of data privacy as the raw data never leaves the client machine.
However, recent work hasshown that in practice, client data can be reconstructed from updates sent from the client to theserver during training [25, 40, 38].
Such reconstructions typically start from a randomly sampledinput and modify it so that gradient updates obtained with this input match the gradients which theclient sent to the server.
And while there has been much work on reconstruction attacks in computervision, there has comparatively been little work in the text domain, despite the fact that some of themost prominent applications of federated learning involve learning over textual data, e g., next-wordprediction on mobile phones [27].
A key component behind the successful attacks in vision has beenthe use of image priors such as total variation [7].
These priors guide the reconstruction towardsnatural images more likely to correspond to client data.
However, the use of priors has so far beenmissing from attacks on text [40, 3], limiting their ability to reconstruct real client data.
This Work: private text reconstruction via priors
In this work, we propose LAMP, a newreconstruction attack which leverages language model priors to extract private text from gradients.
The overview of our attack is shown in Fig.
The attacker receives the transformer network and agradient ∇ θ L which the client has computed on their private data ( x ∗ , y ∗ ) .
The attack is initiatedby sampling token embeddings for a candidate text from a Gaussian distribution.
Then, at eachstep, we alternate between continuous (shown in blue) and discrete optimization (shown in green).
Yellow boxes show our reconstruction after each iteration.
First, the continuous part minimizes thereconstruction loss the gradients the observed client gradients, together with embeddings regularization.
However, only the continuous part is not sufﬁcient as thegradient descent can get stuck in a local optimum due to its inability to make discrete changes to thesentence.
We address this issue by introducing a discrete step – here, we generate a list of candidatesentences using several transformations on the sequence of tokens (e.g., moving a token to anotherposition) and then select a candidate that minimizes the combined reconstruction loss and perplexity,which measures the likelihood of observing the text in a natural distribution.
We use GPT-2 [26]as an auxiliary language model to measure the perplexity of each candidate (however, our method allows using other models).
We compute our ﬁnal reconstruction by setting each embedding to its nearest neighbor from the
Key to the effectiveness of our reconstruction attack is the use of a language model prior combinedwith a search that alternates continuous and discrete optimization steps.
Our experimental evaluationdemonstrates the effectiveness of this approach – LAMP is able to extract text from state-of-the-arttransformer models on several common datasets, reconstructing up to 5 times more bigrams thanprior work.
Moreover, we are the ﬁrst to perform text reconstruction in more complex settings suchas batch sizes larger than 1, ﬁne-tuned models, and defended models.
Overall, across all settings wedemonstrate LAMP is effective in reconstructing large portions of private text.
Main
Contributions
Our main contributions are:
• A novel attack for recovering input text from gradients by leveraging an auxiliary language
model to guide the search towards natural text combined with a search procedure which
alternates continuous and discrete optimization.
• An extensive experimental evaluation demonstrating that LAMP is highly effective and thatit can reconstruct signiﬁcantly more private text than prior work.
• The ﬁrst thorough experimental evaluation of text attacks in more complex settings such as
larger batch sizes, ﬁne-tuned models and defended models.
Related Work
Federated learning [21, 16] has attracted substantial interest [14] due to its ability to train deeplearning models in a decentralized way, such that individual user data is not shared during training.
Instead, individual clients calculate local gradient updates on their private data and share only theupdates with a centralized server that aggregates updates from all clients [21].
The underlyingassumption here is that user data cannot be recovered from gradient updates, allowing clients totrain a joint model on their private data.
Recently, several works [25, 40, 38, 39] demonstrated thatgradients can still leak information, thus invalidating the fundamental privacy assumption.
Moreover,recent work achieved near-perfect image reconstruction from gradients [7, 37, 12].
Interestingly,prior work showed that an auxiliary model [12] or prior information [2] can signiﬁcantly improvereconstruction quality.
Finally, Huang et al.
[10] noticed that gradient leakage attacks often makestrong assumptions, namely that batch normalization statistics and ground truth labels are known.
In our work, we do not assume knowledge of batch normalization statistics and as we focus on binaryclassiﬁcation tasks, we can simply enumerate all possible labels during reconstruction.
Despite substantial progress on image reconstruction, attacks in other domains remain challenging,as the techniques used for images rely extensively on domain speciﬁc knowledge.
In the domainof text, in particular, where federated learning is often applied [29], only a handful of works exist[40, 3, 19].
DLG [40] was ﬁrst to attempt reconstruction from gradients coming from a transformer;
TAG [3] extended DLG by adding an L 1 term to the reconstruction loss; ﬁnally, unlike TAG and
DLG which are optimization-based techniques, APRIL [19] recently demonstrated an exact gradientleakage technique applicable to transformer networks.
However, APRIL assumes batch size of 1 andlearnable positional embeddings, which makes it easy to defend against.
Another attack on NLPis [6], but they use stronger assumption that server can send malicious updates to the clients.
Finally, there have been several works attempting to protect against gradient leakage.
Works basedon heuristics [31, 28] lack privacy guarantees and have been shown ineffective against strongerattacks [2], while those based on differential privacy do train models with formal guarantees [1], buttypically hurt the accuracy of the trained models as they require adding noise to the gradients.
Weremark that we also evaluate our attack on defended networks and demonstrate its effectiveness.
Background
In this section, we introduce the background necessary to understand our work.
3.1 Federated Learning
In federated learning, C clients aim to jointly optimize a neural network f with parameters θ on theirprivate data.
At iteration k , the parameters θ k are sent to all clients, where each client c executesa gradient update ∇ θ k L ( x ∗ c , y ∗ c ) on a sample ( x ∗ c , y ∗ c ) from their dataset ( X c , Y c ) .
The updates aresent back to the server and aggregated.
While in Sec.
5 we experiment with both FedSGD [21] and
FedAvg [17] client updates , throughout the text we assume that clients use FedSGD updates:
[Equation]
Gradient Leakage Attacks A gradient leakage attack is an attack executed by the server (or aparty which compromised it) that tries to obtain the private data ( x ∗ c , y ∗ c ) of a client using the gradientupdates ∇ θ k L ( x ∗ c , y ∗ c ) sent to the server.
Gradient leakage attacks usually assume honest-but-curiousservers which are not allowed to modify the federated training protocol outlined above.
A commonapproach to gradient leakage attacks, adopted by Zhu et al.
[40], Zhao et al.
[38], Deng et al.
[3] aswell as our work, is to obtain the private data by solving the optimization problem:
[Equation]
where δ is some distance measure and ( x c , y c ) denotes dummy data optimized using gradient descentto have similar gradients ∇ θ k L ( x c , y c ) to true data ( x ∗ c , y ∗ c ) .
Common choices for δ are L 2 [40],
L 1 [3] and cosine distances [7].
When the true label y ∗ c is known, the problem simpliﬁes to
[Equation]
which was shown [38, 10] to be simpler to solve with gradient descent approaches.
3.2 Transformer Networks
In this paper, we focus on the problem of gradient leakage of text on transformers [32].
Given someinput text, the ﬁrst step is to tokenize it into tokens from some ﬁxed vocabulary of size V .
Each tokenis then converted to a 1 -hot vector denoted t 1 , t 2 , . .
. , t n ∈ R V , where n is the number of tokensin the text.
The tokens are then converted to embedding vectors x 1 , x 2 , . . .
, x n ∈ R d , where d is achosen embedding size, by multiplying by a trained embedding matrix W embed ∈ R V × d [4].
The rowsof W embed represent the embeddings of the tokens and we denote them as e 1 , e 2 . . .
, e V ∈ R d .
In addition to tokens, their positions in the sequence are also embedded using the positional embeddingmatrix W pos ∈ R P × d , where P is the longest allowed token sequence.
The resulting positionalembeddings are denoted p 1 , p 2 , . .
. , p n .
For notational simplicity, we denote e =
e 1 , e 2 . . .
, e V ,x = x 1 , x 2 , . . .
, x n and p = p 1 , p 2 , . .
. , p n .
We use the token-wise sum of the embeddings x and pas an input to a sequence of self-attention layers [32].
The ﬁnal classiﬁcation output is given by theﬁrst output of the last self-attention layer that undergoes a ﬁnal linear layer, followed by a tanh .
3.3 Calculating Perplexity on Pretrained Models
In this work, we rely on large pretrained language models, such as GPT-2 [26], to assess the qualityof the text produced by the continuous part of our optimization.
Such models are typically trainedto calculate the probability P ( t n | t 1 , t 2 , ..., t n − 1 ) of inserting a token t n from the vocabulary oftokens to the end of the sequence of tokens t 1 , t 2 , ..., t n − 1 .
Therefore, such models can be leveragedto calculate the likelihood of a sequence of tokens t 1 , t 2 , ..., t n , as follows:
[Equation]
One can use the likelihood, or the closely-related negative log-likelihood, as a measure of the qualityof a produced sequence.
However, the likelihood depends on the length of the sequence, as probabilitydecreases with length.
To this end, we use the perplexity measure [11], deﬁned as:
[Equation]
l = log P ( t l +1 | t 1 , t 2 , ..., t l ) .
In the discrete part of our optimization, we rely on this measure to assess the quality of reconstructedsequences produced by the continuous part.
Extracting Text with LAMP
In this section we describe the details of our attack which alternates between continuous optimizationusing gradient descent, presented in Sec. 4.2, and discrete optimization using language models toguide the search towards more natural text reconstruction, presented in Sec. 4.3.
4.1 Notation
We denote the attacked neural network and its parameters with f and θ , respectively.
Further, wedenote the client token sequence and its label as ( t ∗ , y ∗ ) , and our reconstructions as ( t , y ) .
For eachtoken t ∗ i in t ∗ and t i in t , we denote their embeddings on f with x ∗ i ∈ R d and x i ∈ R d , respectively.
Moreover, for each token in our vocabulary, we denote the embedding on f with e i ∈ R d .
We collectthe individual embeddings x
i , x ∗ i , and e i into the matrices x ∈ R d × n , x ∗ ∈ R d × n and e ∈ R d × V ,where n is the number of tokens in t ∗ and V is the size of the vocabulary.
4.2 Continuous Optimization
We now describe the continuous part of our attack (blue in Fig. 1).
Throughout the paper, we assumeknowledge of the ground truth label y ∗ of the client token sequence we aim to reconstruct, meaningy =
y ∗ .
This assumption is not a signiﬁcant restriction as we mainly focus on binary classiﬁcation,with batch sizes such that trying all possible combinations of labels is feasible.
Moreover, priorwork [8, 37] has demonstrated that labels can easily be recovered for basic network architectures,which can be adapted for transformers in future work.
We initialize our reconstruction candidate bysampling embeddings from a Gaussian and pick the one with the smallest reconstruction loss.
Reconstruction Loss A key component of our attack is a loss measuring how close the reconstructed gradient is to the true gradient.
Assuming an l -layer network, where θ i denotes the parametersof layer i , an option is to use the combination of L 2 and L 1 loss proposed by Deng et al.
L tag ( x ) = l (cid:88)
[Equation]
where α tag is a hyperparameter.
Another option is to use the cosine reconstruction loss proposedby Geiping et al.
[7] in the image domain:
[Equation]
Naturally, LAMP can also be instantiated using any other loss.
Interestingly, we ﬁnd that there is noloss that is universally better, and the effectiveness is dataset dependent.
Intuitively, L 1 loss is lesssensitive to outliers, while cosine loss is independent of the gradient norm so it works well for smallgradients.
Thus, we set the gradient loss L grad to either L tag or L cos , depending on the setting.
Embedding Regularization In the process of optimizing the reconstruction loss, we observe theresulting embedding vectors x i often steadily grow in length.
We believe this behavior is due to theself-attention layers in transformer networks that rely predominantly on dot product operations.
Asa result, the optimization process focuses on optimizing the direction of individual embeddings x i ,disregarding their length.
To address this, we propose an embedding length regularization term:
[Equation]
The regularizer forces the mean length of the embeddings of the reconstructed sequence to be closeto the mean length of the embeddings in the vocabulary.
The ﬁnal gradient reconstruction erroroptimized in LAMP is given by:
[Equation]
where α reg is a regularization weighting factor.
Optimization We summarize how described components work together in the setting of continuousoptimization.
To reconstruct the token sequence t ∗ , we ﬁrst randomly initialize a sequence of dummytoken embeddings x = x 0 x 1 . . .
x n , with x i ∈ R d .
Following prior work on text reconstruction fromgradients [3, 40], we apply gradient descent on x to minimize the reconstruction loss L rec ( x ) .
To thisend, a second-order derivative needs to be computed, as L rec ( x ) depends on the network gradient atx .
Similar to prior work [3, 40], we achieve this using Pytorch’s[24] automatic differentiation.
4.3 Discrete Optimization
Next, we describe the discrete part of our optimization (green in Fig. 1).
While continuous optimization can often successfully recover token embeddings close to the original, they can be in thewrong order, depending on how much positional embeddings inﬂuence the output.
For example,sequences corresponding to sentences “weather is nice.”
and “nice weather is.”
might result in asimilar reconstruction loss, though the ﬁrst sequence has a higher likelihood of being natural text.
Toaddress this issue, we perform several discrete sequence transformations, and choose the one withboth a low reconstruction loss and a low perplexity under the auxiliary language model.
Generating Candidates Given the current sequence x = x 1 x 2 ...x n , we generate candidates forthe new sequence x (cid:48) using one of the following transformations:
• Swap : We select two random positions i and j in the sequence, and swap thetokens x i and x j at these two positions to obtain a new candidate sequence
x (cid:48)
=
x 1 x 2 . . .
x i − 1 x j x
i
+1 . . .
x j − 1 x i x j +1 . . .
x n .
• MoveToken : We select two random positions i and j in the sequence, andmove the token x
i after the position j in the sequence, thus obtaining x (cid:48)
=
x 1 x 2 . . .
x i − 1 x
i
+1 . . .
x j − 1 x j x i x j +1 . . .
x n .
• MoveSubseq : We select a subsequence of tokens between random positions i and j (wherei < j ), and move the entire subsequence after some random position p .
The new sequenceis now x (cid:48) =
x 1 x 2 . . .
x i − 1 x j +1 . . .
x p x i . . .
x j x p +1 . . .
x n .•
MovePreﬁx : We select some random preﬁx of the sequence up to token i and move it to theend of the sequence.
The modiﬁed sequence then is x (cid:48)
=
x i +1 . . .
x n x 1 x 2 . . .
x i .
Next, we use a language model to check if generated candidates improve over the current sequence.
Selecting Candidates via Language Model We accept the new sequence x (cid:48) if it improves thecombination of the reconstruction loss and perplexity over the current one:
[Equation]
1: x ( k ) ∼ N (0 , I ) , where k
= 1 , 2 , ..., n init 2: x ← arg min x ( k ) L rec ( x ( k ) ) 3: for i = 1 to it do 4: for j = 1 to n c do 5: x ← x − λ ∇ x L rec ( x ) 6: end for 7: for j = 1 to n d do 8: x (cid:48) ← T RANSFORM ( x ) 9: t ← P ROJECT T O V
OCABULARY ( x ) 10: t (cid:48) ← P ROJECT T O V OCABULARY
( x (cid:48) ) 11: if L rec ( x (cid:48) )
+ α lm L lm ( t (cid:48) ) <
L rec ( x ) + α lm L lm ( t ) then 12: x best ← x (cid:48) 13: end if 14: end for 15: x ← x best 16: end for 17: return P ROJECT T
O V OCABULARY ( x )
Here t and t (cid:48) denote sequences of tokens obtainedby mapping each embedding of x and x (cid:48) to thenearest neighbor in the vocabulary according tothe cosine distance.
The term L rec is the reconstruction loss introduced in Sec.
4.2, while L lmdenotes the perplexity measured by an auxiliarylanguage model, such as GPT-2.
The parameterα lm determines the trade-off between L rec and L lm :if it is too low then the attack will not utilize thelanguage model, and if it is too high then the attackwill disregard the reconstruction loss and only focus on the perplexity.
Going back to our example,assume that our reconstruction equals the secondsequence “nice weather is.”.
Then, at some point,we might use the MoveToken transformation tomove the word “nice“ behind the word “is” whichwould presumably keep the reconstruction losssimilar, but drastically improve perplexity.
4.4 Complete Reconstruction Attack
We describe our end-to-end attack in Algorithm 1.
We initialize the reconstruction x by sampling froma Gaussian distribution n init times, and choose the sample with minimal reconstruction loss as ourinitial reconstruction.
Then, at each step we alternate between continuous and discrete optimization.
We ﬁrst perform n c steps of continuous optimization to minimize the reconstruction loss (Line 5,see Sec. 4.2).
Then, we perform n d steps of discrete optimization to minimize the combination ofreconstruction loss and perplexity (Line 8, see Sec. 4.3).
Finally, in Line 17 we project the continuousembeddings x to the nearest tokens according to the cosine similarity of the embeddings.
Experimental Evaluation
We now discuss our experimental results, demonstrating the effectiveness of LAMP compared to priorwork in a wide range of settings.
We ﬁrst describe our experimental setup, followed by a presentationof our main results, an extensive ablation study and additional evaluation on defended networks.
Datasets Prior work [3] demonstrated that text length is a key factor for the success of reconstructionfrom gradients.
To this end, in our experiments we consider three binary classiﬁcation datasets ofincreasing complexity: CoLA [34] and SST-2 [30] from GLUE [33] with typical sequence lengthsbetween 5 and 9 words, and 3 and 13 words, respectively, and RottenTomatoes [23] with typicalsequence lengths between 14 and 27 words.
The CoLA dataset contains English sentences fromlanguage books annotated with binary labels describing if the sentences are grammatically correct,while SST-2 and RottenTomatoes contain movie reviews annotated with a binary sentiment.
For allexperiments, we evaluate the methods on 100 random sequences from the respective training sets.
We remark that attacking in binary classiﬁcation settings is a more difﬁcult task than the maskingevaluated in prior work [40], where the attacker can utilize strictly more information.
Models Our experiments are performed on different target models based on the BERT [4] architecture.
The main model we consider is BERT BASE , which has 12 layers, 768 hidden units, 3072feed-forward ﬁlter size, and 12 attention heads.
To illustrate the generality of our approach withrespect to model size, we additionally consider smaller model TinyBERT 6 from Jiao et al.
[13] with6 layers, 768 hidden units, feed-forward ﬁlter size of 3072 and 12 attention heads.
All models weretaken from Hugging Face [36].
The BERT BASE was pretrained on Wikipedia [5] and BookCorpus [41]datasets, while TinyBERT 6 was distilled from BERT BASE .
We perform our main experiments onpretrained models, as this is the most common setting for training classiﬁcation models from text [22].
For the auxiliary language model we use the pre-trained GPT-2 given by Guo et al.
[9], trained on thesame tokenizer used to pre-train our target BERT models.
Metrics Following TAG [3], we measure the success of our methods based on the ROUGE familyof metrics [18].
In particular, we report the aggregated F-scores on ROUGE-1, ROUGE-2 and
ROUGE-L, which represent the percentage of recovered unigrams, recovered bigrams and the ratioof the length of the longest matching subsequence to the length of whole sequences.
When evaluatingbatch sizes greater than 1 , we exclude the padding tokens, used to pad shorter sequences, from thereconstruction and the ROUGE metric computation.
Experimental Setup We run all of our experiments on a single NVIDIA RTX
Ti GPU with11 GB of RAM.
In all settings we consider, we compare our method with baselines DLG [40] and
TAG [3] discussed in Sec. 2.
As TAG does not have public code, we use our own implementation, andremark that the results obtained using our implementation are similar or better than those reported in Deng et al.
We consider two variants of our tool, LAMP Cos and LAMP L 2 + L 1 , that use the L cosand L tag gradient matching losses for the continuous optimization.
We run our attack with it = 30 andn _ c = 75 and n _ d = 200 and stop the optimization early once we reach a total of 2000 continuousoptimization steps. We run DLG and TAG for 2500 optimization steps.
We use Adam [15] for thecontinuous optimization with a learning rate decay factor γ applied every 50 steps in all methods.
We picked the hyperparameters for TAG, LAMP Cos and LAMP L 2 + L 1 , separately on CoLA and
RottenTomatoes using grid search on BERT BASE and applied them to all networks.
For CoLA, weﬁrst evaluated 50 combinations on 10 randomly selected (in a stratiﬁed way with respect to length)sequences from the training set (after removing the 100 test sequences).
Then, we further evaluatedthe best 10 combinations on different 20 sequences from the training set.
For RottenTomatoes, wepicked the hyperparameters from the same 10 best combinations and evaluated them on the same 20additional sequences.
As the optimal hyperparameters for RottenTomatoes exactly matched the oneson CoLA, we also used the same hyperparameters on SST-2. Further details are provided in App.
D.
The resulting hyperparameters for LAMP Cos are α lm = 0 .
2 , α reg = 1 . 0 , λ = 0 .
01 , γ = 0 . 89 .
Incontrast, the best hyperparameters for LAMP L 2 + L 1 are α tag
= 0 .
01 , α lm = 60 , α reg = 25 , λ = 0 .
01 ,γ = 0 .
89 , as the loss L tag is on a different order of magnitude compared to L cos .
Finally, TAG’s besthyperparameters are α tag
= 0 .
01 , λ = 0 .
1 , γ = 1 .
(no decay).
For our methods, we additionallyapplied a two step initialization procedure.
We ﬁrst initialized the embedding vectors with 500random samples from a standard Gaussian distribution and picked the best one according to L grad ( x ) .
We then computed 500 permutations on the best initialization and chose the best one in the same way.
Main Experiments We evaluate the two variants of LAMP against DLG [40] and TAG [3] on
BERT BASE and TinyBERT 6 .
Additionally, on BERT BASE we experiment on two more complexsettings.
First, unlike prior work, we also evaluate batch sizes > 1 to investigate whether we canreconstruct some of the sequences in this more challenging setting.
Second, we evaluate attacksafter the model has already been ﬁne-tuned for 2 epochs on each task (following Devlin et al.
[4]),as Balunovi ´ c et al.
[2] showed that in the vision domain it is signiﬁcantly more difﬁcult to attackalready trained networks.
For both baselines and our attacks, for simplicity we assume the lengths ofsequences are known, as otherwise an adversary can simply run the attack for all possible lengths.
The results are shown in Table 1.
From the ROUGE-1 metric, we can observe that we recovermore tokens than the baselines in all settings, except on RottenTomatoes with batch size 2, where
TAG is slightly better.
Moreover, the main advantage of LAMP is that the order of tokens in thereconstructed sequences matches the order in target sequences much more closely, as evidenced bythe large increase in ROUGE-2 (5 × on CoLA with batch size 1 ).
This observation is further backedby the ROUGE-L metric that shows we are able to reconstruct up to 23% longer subsequences onaverage compared to the baselines.
These results conﬁrm our intuition that using GPT-2 to guide thesearch allows us to reconstruct sequences that are a much closer match to the original sequences.
Wepoint out that Table 1 reafﬁrms the observations ﬁrst made in Deng et al.
[3], that DLG is consistentlyworse in all metrics compared to both TAG and LAMP and that the much longer sequences in
RottenTomatoes still pose challenges to good reconstruction.
In addition to demonstrated gains on batch size 1 , our experiments show that for larger batch sizes wecan also reconstruct signiﬁcant portions of text (see experiments on CoLA and SST-2).
To the bestof our knowledge, we are the ﬁrst to show this, suggesting that gradient leakage can be a realistic
[Table] [Table]
security threat in practice. Comparing the results for LAMP L 2 + L 1 and LAMP Cos , we observe that 300 L cos is better than L tag in almost all metrics on batch size 1 , across models, but the trend reversesas batch size is increased.
We also examined variability in the results: LAMP
Cos with 10 randomseeds for BERT BASE on CoLA with B=1 produces R-1, R-2 and R-L of 88 .
2 ± 1 .
± 2 . 37 ,75 . 0 ± 1 . 21 , respectively, which is reasonable.
Further, our results indicate that even smaller and ﬁne-tuned models leak signiﬁcant amount of clientinformation.
In particular, TinyBERT 6 is shown to be even more vulnerable than BERT BASE and
BERT BASE -FT is shown to be only few percent worse in reconstruction compared to BERT BASE ,which is surprising given the prior image domain results.
This shows that smaller models can notresolve the privacy issue, despite previous suggestions in Deng et al.
Finally, we show sample sequence reconstructions from both LAMP and the TAG baseline on CoLAwith B = 1 in Table 2.
In Table 2, with green we denote correctly reconstructed bigrams, andwith yellow correct unigrams.
We can observe that our reconstruction is more coherent, and thatit qualitatively outperforms the baseline.
In App.
B, we further show the convergence rate of ouroptimization procedure compared to the baselines on an example sequence, suggesting that LAMPcan converge faster than TAG and DLG.
Ablation Studies
In the next experiment, we perform ablation studies to examine the inﬂuence ofeach proposed component of our method.
We compare the following variants of LAMP: (i) withcosine loss, (ii) with L 2 loss, (iii) with L 1 + L 2 loss, (iv) without the language model ( α lm = 0 ),(v) without embedding regularization ( α reg = 0 ), (vi) without discrete transformations ( n d = 0 ).
For this experiment, we use the CoLA dataset and BERT BASE with B = 1 .
We show the resultsin Table 3.
We observe that LAMP achieves good results with both losses, though cosine is generallybetter for batch size 1 .
More importantly, dropping any of the proposed features makes ROUGE-1and ROUGE-2 signiﬁcantly worse.
We note the most signiﬁcant drop in ROUGE-2 reconstructionquality happens when using transformations without using the language model (LAMP α lm =0 ), whichperforms even worse than doing no transformations (LAMP NoDiscrete ) at all.
This suggests that the use
[Table] [Table]
of the language model is crucial to obtaining good results.
The experiments also conﬁrm usefulnessof other features such as embedding regularization.
Attacking the Defended Networks So far, all experiments assumed that clients have not defendedagainst data leakage.
Following work on vision attacks [40, 35], we now consider the defense ofadding Gaussian noise to gradients (with additional clipping this would correspond to DP-SGD [1]).
Note that, as usual, there is a trade-off between privacy and accuracy: adding more noise will leadto better privacy, but make accuracy worse.
We measure the performance of the ﬁne-tuned modelson CoLA using the MCC metric [20] for which higher values are better.
We ﬁne-tuned for 2 epochswith different Gaussian noise levels σ and obtained the MCC scores depicted in Table 4.
We didnot explore noises > 0 .
due to the signiﬁcant drop in MCC from 0 .
557 for the undefended modelto 0 .
We show the results of our experiments on these defended networks in Table 4.
While allmethods’ reconstruction metrics degrade, as expected, we see that most text is still recoverable forthe chosen noise levels.
Moreover, our method is still signiﬁcantly better than the baselines, and thusshows the importance of evaluating defenses with strong reconstruction attacks.
In App.
C.3 we alsoshow that LAMP is useful against a defense which masks some percentage of gradients.
Attacking FedAvg So far, we have only considered attacking the FedSGD algorithm.
In thisexperiment, we apply our attack on the commonly used FedAvg [17] algorithm.
As NLP models areoften ﬁne-tuned using small learning rates (2e-5 to 5e-5 in the original BERT paper), we ﬁnd that
FedAvg reconstruction performance is close to FedSGD performance with batch size multiplied bythe number of FedAvg steps.
We experimented with attacking FedAvg with 4 steps using B = 1 perstep, lr = 5 e-
5 on CoLA and BERT BASE with LAMP L 2 + L 1 .
We obtained R-1, R-2 and R-L of 66 .
5 ,21 . 0 , 55 . 1 , respectively, comparable to the reported results on FedSGD with B = 4 .
Conclusion
In this paper, we presented LAMP, a new method for reconstructing private text data from gradients byleveraging language model priors and alternating discrete and continuous optimization.
Our extensiveexperimental evaluation showed that LAMP consistently outperforms prior work on datasets ofvarying complexity and models of different sizes.
Further, we established that LAMP is able toreconstruct private data in a number of challenging settings, including bigger batch sizes, noisedefended gradients, and ﬁne-tuned models.
Our work highlights that private text data is not sufﬁcientlyprotected by federated learning algorithms and that more work is needed to alleviate this issue.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We demonstrate the effectivenessof existing defenses against our attack in Sec. 5, further we outline that our work does
not deal with reconstructing labels which is left for future work, as described in Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] As
described in intro Sec. 1, our work is closely related to existing work on gradients
leakage attacks which are capable of breaking the privacy promise of federated learning
e g. Zhao et al.
[38], Geiping et al
[7], Yin et al.
[37], Deng et al.
We refer the
reader to these papers for comprehensive discussion of the societal implications of
breaking federated privacy.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report [Table] [Table]
[Table]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[No] We have reported the type ofresources used only.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[No] We use standard datasets and cite the [Table]
authors instead.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[No] We use standard datasets.
We refer the reader
[Table]
to the original authors of the dataset for this discussion.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]