LAMP: Extracting Text from Gradients with Language Model Priors
Abstract
Recent work shows that sensitive user data can be reconstructed from gradient updates, breaking the key privacy promise of federated learning.
While success was demonstrated primarily on image data, these methods do not directly transfer to other domains such as text.
In this work, we propose LAMP, a novel attack tailored to textual data, that successfully reconstructs original text from gradients.
Our attack is based on two key insights: (i) modelling prior text probability via an auxiliary language model, guiding the search towards more natural text, and (ii) alternating continuous and discrete optimization which minimizes reconstruction loss on embeddings while avoiding local minima via discrete text transformations.
Our experiments demonstrate that LAMP is signiﬁcantly more effective than prior work: it reconstructs 5x more bigrams and 23% longer subsequences on average.
Moreover, we are ﬁrst to recover inputs from batch sizes larger than 1 for textual models.
These ﬁndings indicate that gradient updates of models operating on textual data leak more information than previously thought.
Introduction
Federated learning [24] is a widely adopted framework for training machine learning models in a decentralized way.
Conceptually, the method allows training of highly accurate models while offering a level of data privacy as the raw data never leaves the client machine.
However, recent work has shown that in practice, client data can be reconstructed from updates sent from the client to the server during training [28, 43, 41].
Such reconstructions typically start from a randomly sampled input and modify it so that gradient updates obtained with this input match the gradients which the client sent to the server.
And while there has been much work on reconstruction attacks in computer vision, there has comparatively been little work in the text domain, despite the fact that some of the most prominent applications of federated learning involve learning over textual data, e.g., next-word prediction on mobile phones [30].
A key component behind the successful attacks in vision has been the use of image priors such as total variation [7].
These priors guide the reconstruction towards natural images more likely to correspond to client data.
However, the use of priors has so far been missing from attacks on text [43, 3], limiting their ability to reconstruct real client data.
This Work: private text reconstruction via priors
In this work, we propose LAMP, a new reconstruction attack which leverages language model priors to extract private text from gradients.
The overview of our attack is shown in Fig.
The attacker receives the transformer network and a gradient ∇ θ L which the client has computed on their private data ( x ∗ , y ∗ ) .
The attack is initiated by sampling token embeddings for a candidate text from a Gaussian distribution.
Then, at each ∗ Equal contribution.
Key to the effectiveness of our reconstruction attack is the use of a language model prior combined with a search that alternates continuous and discrete optimization steps.
Our experimental evaluation demonstrates the effectiveness of this approach – LAMP is able to extract text from state-of-the-art transformer models on several common datasets, reconstructing up to 5 times more bigrams than prior work.
Moreover, we are the ﬁrst to perform text reconstruction in more complex settings such as batch sizes larger than 1, ﬁne-tuned models, and defended models.
Overall, across all settings we demonstrate LAMP is effective in reconstructing large portions of private text.
Main Contributions Our main contributions are:
• A novel attack for recovering input text from gradients by leveraging an auxiliary language model to guide the search towards natural text combined with a search procedure which alternates continuous and discrete optimization.
• An implementation of the attack and its extensive experimental evaluation demonstrating that LAMP is highly effective and that it can reconstruct signiﬁcantly more private text than prior work. We make our code publicly available at https://github.com/eth-sri/lamp .
• The ﬁrst thorough experimental evaluation of text attacks in more complex settings such as larger batch sizes, ﬁne-tuned models and defended models.
Related Work
Federated learning [24, 18] has attracted substantial interest [16] due to its ability to train deep learning models in a decentralized way, such that individual user data is not shared during training.
Instead, individual clients calculate local gradient updates on their private data and share only the updates with a centralized server that aggregates updates from all clients [24].
The underlying assumption here is that user data cannot be recovered from gradient updates, allowing clients to train a joint model on their private data.
Recently, several works [28, 43, 41, 42] demonstrated that gradients can still leak information, thus invalidating the fundamental privacy assumption.
Moreover, recent work achieved near-perfect image reconstruction from gradients [7, 40, 14].
Interestingly, prior work showed that an auxiliary model [14] or prior information [2] can signiﬁcantly improve reconstruction quality.
Finally, Huang et al.
[12] noticed that gradient leakage attacks often make strong assumptions, namely that batch normalization statistics and ground truth labels are known.
In our work, we do not assume knowledge of batch normalization statistics and as we focus on binary classiﬁcation tasks, we can simply enumerate all possible labels during reconstruction.
Despite substantial progress on image reconstruction, attacks in other domains remain challenging, as the techniques used for images rely extensively on domain speciﬁc knowledge.
In the domain of text, in particular, where federated learning is often applied [32], only a handful of works exist [43, 3, 22].
DLG [43] was ﬁrst to attempt reconstruction from gradients coming from a transformer;
TAG [3] extended DLG by adding an L 1 term to the reconstruction loss; ﬁnally, unlike TAG and
DLG which are optimization-based techniques, APRIL [22] recently demonstrated an exact gradient leakage technique applicable to transformer networks.
However, APRIL assumes batch size of 1 and learnable positional embeddings, which makes it easy to defend against.
Another attack on NLP is [6], but they use stronger assumption that server can send malicious updates to the clients.
Furthermore, there is a concurrent work [11] on reconstructing text from transformers, but it is limited to the case when token embeddings are trained with the network.
Finally, there have been several works attempting to protect against gradient leakage.
Works based on heuristics [34, 31] lack privacy guarantees and have been shown ineffective against stronger attacks [2], while those based on differential privacy do train models with formal guarantees [1], but typically hurt the accuracy of the trained models as they require adding noise to the gradients.
We remark that we also evaluate our attack on defended networks and demonstrate its effectiveness.
Background
In this section, we introduce the background necessary to understand our work.
3.1 Federated Learning
In federated learning, C clients aim to jointly optimize a neural network f with parameters θ on their private data.
At iteration k , the parameters θ k are sent to all clients, where each client c executes a gradient update ∇ θ k L ( x ∗ c , y ∗ c ) on a sample ( x ∗ c , y ∗ c ) from their dataset ( X c , Y c ) .
The updates are sent back to the server and aggregated.
While in Sec.
5 we experiment with both FedSGD [24] and
FedAvg [19] client updates, throughout the text we assume that clients use FedSGD updates:
[Equation]
Gradient Leakage Attacks A gradient leakage attack is an attack executed by the server (or a party which compromised it) that tries to obtain the private data ( x ∗ c , y ∗ c ) of a client using the gradient updates ∇ θ k L ( x ∗ c , y ∗ c ) sent to the server.
Gradient leakage attacks usually assume honest-but-curious servers which are not allowed to modify the federated training protocol outlined above.
A common approach to gradient leakage attacks, adopted by Zhu et al.
[43], Zhao et al.
[41], Deng et al.
[3] as well as our work, is to obtain the private data by solving the optimization problem:
[Equation]
where δ is some distance measure and ( x c , y c ) denotes dummy data optimized using gradient descent to have similar gradients ∇ θ k L ( x c , y c ) to true data ( x ∗ c , y ∗ c ) .
Common choices for δ are L 2 [43],
L 1 [3] and cosine distances [7].
When the true label y ∗ c is known, the problem simpliﬁes to
[Equation]
which was shown [41, 12] to be simpler to solve with gradient descent approaches.
3.2 Transformer Networks
In this paper, we focus on the problem of gradient leakage of text on transformers [35].
Given some input text, the ﬁrst step is to tokenize it into tokens from some ﬁxed vocabulary of size V .
Each token is then converted to a 1 -hot vector denoted t 1 , t 2 , . .
. , t n ∈ R V , where n is the number of tokens in the text.
The tokens are then converted to embedding vectors x 1 , x 2 , . . .
, x n ∈ R d , where d is a chosen embedding size, by multiplying by a trained embedding matrix W embed ∈ R V × d [4].
The rows of W embed represent the embeddings of the tokens and we denote them as e 1 , e 2 , . . .
, e V ∈ R d .
In addition to tokens, their positions in the sequence are also embedded using the positional embedding matrix W pos ∈ R P × d , where P is the longest allowed token sequence.
The resulting positional embeddings are denoted p 1 , p 2 , . . .
, p n .
For notational simplicity, we denote e =
e 1 , e 2 , . .
, e V , x = x 1 , x 2 , . . .
, x n and p = p 1 , p 2 , . .
. , p n .
We use the token-wise sum of the embeddings x and p as an input to a sequence of self-attention layers [35].
The ﬁnal classiﬁcation output is given by the ﬁrst output of the last self-attention layer that undergoes a ﬁnal linear layer, followed by a tanh .
3.3 Calculating Perplexity on Pretrained Models
In this work, we rely on large pretrained language models, such as GPT-2 [29], to assess the quality of the text produced by the continuous part of our optimization.
Such models are typically trained to calculate the probability P ( t n | t 1 , t 2 , ..., t n − 1 ) of inserting a token t n from the vocabulary of tokens to the end of the sequence of tokens t 1 , t 2 , ..., t n − 1 .
Therefore, such models can be leveraged to calculate the likelihood of a sequence of tokens t 1 , t 2 , ..., t n , as follows:
[Equation]
One can use the likelihood, or the closely-related negative log-likelihood, as a measure of the quality of a produced sequence.
However, the likelihood depends on the length of the sequence, as probability decreases with length.
To this end, we use the perplexity measure [13], deﬁned as:
[Equation]
In the discrete part of our optimization, we rely on this measure to assess the quality of reconstructed sequences produced by the continuous part.
Extracting Text with LAMP
In this section we describe the details of our attack which alternates between continuous optimization using gradient descent, presented in Sec. 4.2, and discrete optimization using language models to guide the search towards more natural text reconstruction, presented in Sec. 4.3.
4.1 Notation
We denote the attacked neural network and its parameters with f and θ , respectively.
Further, we denote the client token sequence and its label as ( t ∗ , y ∗ ) , and our reconstructions as ( t , y ) .
For each token t ∗ i in t ∗ and t i in t , we denote their embeddings on f with x ∗ i ∈ R d and x i ∈ R d , respectively.
Moreover, for each token in our vocabulary, we denote the embedding on f with e i ∈ R d .
We collect the individual embeddings x i , x ∗ i , and e i into the matrices x ∈ R d × n , x ∗ ∈ R d × n and e ∈ R d × V , where n is the number of tokens in t ∗ and V is the size of the vocabulary.
4.2 Continuous Optimization
We now describe the continuous part of our attack (blue in Fig. 1).
Throughout the paper, we assume knowledge of the ground truth label y ∗ of the client token sequence we aim to reconstruct, meaning y =
y ∗ .
This assumption is not a signiﬁcant restriction as we mainly focus on binary classiﬁcation, with batch sizes such that trying all possible combinations of labels is feasible.
Moreover, prior work [9, 40] has demonstrated that labels can easily be recovered for basic network architectures, which can be adapted for transformers in future work.
We initialize our reconstruction candidate by sampling embeddings from a Gaussian and pick the one with the smallest reconstruction loss.
Reconstruction Loss A key component of our attack is a loss measuring how close the reconstructed gradient is to the true gradient.
Assuming an l -layer network, where θ i denotes the parameters of layer i , an option is to use the combination of L 2 and L 1 loss proposed by Deng et al.
L tag ( x ) = l (cid:88) [Equation] =
[Equation]
where α tag is a hyperparameter.
Another option is to use the cosine reconstruction loss proposed by Geiping et al.
[7] in the image domain:
[Equation]
Naturally, LAMP can also be instantiated using any other loss.
Interestingly, we ﬁnd that there is no loss that is universally better, and the effectiveness is dataset dependent.
Intuitively, L 1 loss is less sensitive to outliers, while cosine loss is independent of the gradient norm so it works well for small gradients.
Thus, we set the gradient loss L grad to either L tag or L cos , depending on the setting.
Embedding Regularization In the process of optimizing the reconstruction loss, we observe the resulting embedding vectors x i often steadily grow in length.
We believe this behavior is due to the self-attention layers in transformer networks that rely predominantly on dot product operations.
As a result, the optimization process focuses on optimizing the direction of individual embeddings x i , disregarding their length.
To address this, we propose an embedding length regularization term:
[Equation]
2 − 1 V
V (cid:88) j = (cid:107) e j (cid:107)  . The regularizer forces the mean length of the embeddings of the reconstructed sequence to be close to the mean length of the embeddings in the vocabulary.
The ﬁnal gradient reconstruction error optimized in LAMP is given by:
[Equation]
where α reg is a regularization weighting factor.
Optimization We summarize how described components work together in the setting of continuous optimization.
To reconstruct the token sequence t ∗ , we ﬁrst randomly initialize a sequence of dummy token embeddings x = x 0 x 1 . . .
x n , with x i ∈ R d .
Following prior work on text reconstruction from gradients [3, 43], we apply gradient descent on x to minimize the reconstruction loss L rec ( x ) .
To this end, a second-order derivative needs to be computed, as L rec ( x ) depends on the network gradient at x .
Similar to prior work [3, 43], we achieve this using Pytorch’s[27] automatic differentiation.
4.3 Discrete Optimization
Next, we describe the discrete part of our optimization (green in Fig. 1).
While continuous optimization can often successfully recover token embeddings close to the original, they can be in the wrong order, depending on how much positional embeddings inﬂuence the output.
For example, sequences corresponding to sentences “weather is nice.”
and “nice weather is.”
might result in a similar reconstruction loss, though the ﬁrst sequence has a higher likelihood of being natural text.
To address this issue, we perform several discrete sequence transformations, and choose the one with both a low reconstruction loss and a low perplexity under the auxiliary language model.
Generating Candidates Given the current sequence x = x 1 x 2 ...x n , we generate candidates for the new sequence x (cid:48) using one of the following transformations:
• Swap : We select two random positions i and j in the sequence, and swap the tokens x i and x j at these two positions to obtain a new candidate sequence
x (cid:48)
=
x 1 x 2 . . .
x i − 1 x j x
i
+1 . . .
x j − 1 x i x j +1 . . .
x n .
• MoveToken : We select two random positions i and j in the sequence, and move the token x i after the position j in the sequence, thus obtaining x (cid:48)
=
x 1 x 2 . . .
x i − 1 x
i
+1 . . .
x j − 1 x j x i x j +1 . . .
x n .
• MoveSubseq : We select a subsequence of tokens between random positions i and j (where i < j ), and move the entire subsequence after some random position p .
The new sequence is now x (cid:48) =
x 1 x 2 . . .
x i − 1 x j +1 . . .
x p x i . . .
x j x p +1 . . .
x n .
• MovePreﬁx : We select some random preﬁx of the sequence up to token i and move it to the end of the sequence.
The modiﬁed sequence then is x (cid:48)
=
x i +1 . . .
x n x 1 x 2 . . .
x i .
Next, we use a language model to check if generated candidates improve over the current sequence.
Selecting Candidates via Language Model We accept the new sequence x (cid:48) if it improves the combination of the reconstruction loss and perplexity over the current one:
[Equation]
1: x ( k ) ∼ N (0 , I ) , where k
= 1 , ..., n init 2: x ← arg min x ( k ) L rec ( x ( k ) ) 3: for i = 1 to it do 4: for j = 1 to n c do 5: x ← x − λ ∇ x L rec ( x ) 6: end for 7: x best ← x 8: t best ← P ROJECT T O V OCAB ( x best ) 9: for j = 1 to n d do 10: x (cid:48) ← T RANSFORM ( x ) 11: t (cid:48)
← P ROJECT T O V OCAB ( x (cid:48) ) 12: if L rec ( x best ) + α lm L lm ( t best ) >
L rec ( x (cid:48) )
+ α lm L lm ( t (cid:48) )
then 13: x best ← x (cid:48) 14: t best ← t (cid:48) 15: end if 16: end for 17: x ← x best 18: end for 19: return P ROJECT T O V OCAB ( x )
Here t and t (cid:48) denote sequences of tokens obtained by mapping each embedding of x and x (cid:48) to the nearest neighbor in the vocabulary according to the cosine distance.
The term L rec is the reconstruction loss introduced in Sec.
4.2, while L lm denotes the perplexity measured by an auxiliary language model, such as GPT-2.
The parameter α lm determines the trade-off between L rec and L lm : if it is too low then the attack will not utilize the language model, and if it is too high then the attack will disregard the reconstruction loss and only focus on the perplexity.
Going back to our example, assume that our reconstruction equals the second sequence “nice weather is.”.
Then, at some point, we might use the MoveToken transformation to move the word “nice“ behind the word “is” which would presumably keep the reconstruction loss similar, but drastically improve perplexity.
4.4 Complete Reconstruction Attack
We describe our end-to-end attack in Algorithm 1.
We initialize the reconstruction x by sampling from a Gaussian distribution n init times, and choose the sample with minimal reconstruction loss as our initial reconstruction.
Then, at each step we alternate between continuous and discrete optimization.
We ﬁrst perform n c steps of continuous optimization to minimize the reconstruction loss (Line 4-6, see Sec. 4.2).
Then, we perform n d steps of discrete optimization to minimize the combination of reconstruction loss and perplexity (Line 9-16, see Sec. 4.3).
Finally, in Line 19 we project the continuous embeddings x to their respective nearest tokens in cosine similarity.
Experimental Evaluation
We now discuss our experimental results, demonstrating the effectiveness of LAMP compared to prior work in a wide range of settings.
We ﬁrst describe our experimental setup, followed by a presentation of our main results, an extensive ablation study and additional evaluation of reconstructions created for different batch sizes and on defended networks.
Datasets Prior work [3] demonstrated that text length is a key factor for the success of reconstruction from gradients.
To this end, in our experiments we consider three binary classiﬁcation datasets of increasing complexity: CoLA [37] and SST-2 [33] from GLUE [36] with typical sequence lengths between 5 and 9 words, and 3 and 13 words, respectively, and RottenTomatoes [26] with typical sequence lengths between 14 and 27 words.
The CoLA dataset contains English sentences from language books annotated with binary labels describing if the sentences are grammatically correct, while SST-2 and RottenTomatoes contain movie reviews annotated with a binary sentiment.
For all experiments, we evaluate the methods on 100 random sequences from the respective training sets.
We remark that attacking in binary classiﬁcation settings is a more difﬁcult task than the masking evaluated in prior work [43], where the attacker can utilize strictly more information.
Models Our experiments are performed on different target models based on the BERT [4] architecture.
The main model we consider is BERT BASE , which has 12 layers, 768 hidden units,feed-forward ﬁlter size, and 12 attention heads.
To illustrate the generality of our approach with respect to model size, we additionally consider a larger model BERT LARGE , which has 24 layers,hidden units, 4096 feed-forward ﬁlter size, and 16 attention heads and a smaller model TinyBERTfrom Jiao et al.
[15] with 6 layers, 768 hidden units, feed-forward ﬁlter size of 3072 and 12 attention heads.
All models were taken from Hugging Face [39].
The BERT BASE and BERT LARGE were pretrained on Wikipedia [5] and BookCorpus [44] datasets, while TinyBERT 6 was distilled from BERT BASE .
We perform our main experiments on pretrained models, as this is the most common setting for training classiﬁcation models from text [25].
For the auxiliary language model we use the pre-trained GPT-2 given by Guo et al.
[10], trained on the same tokenizer used to pre-train our target BERT models.
Metrics Following TAG [3], we measure the success of our methods based on the ROUGE family of metrics [20].
In particular, we report the aggregated F-scores on ROUGE-1, ROUGE-2 and
ROUGE-L, which measure the recovered unigrams, recovered bigrams and the ratio of the length of the longest matching subsequence to the length of whole sequences.
When evaluating batch sizes greater than 1 , we exclude the padding tokens, used to pad shorter sequences, from the reconstruction and the ROUGE metric computation.
Experimental Setup We run all of our experiments on a single NVIDIA RTX [-
-]2080 Ti GPU with 11 GB of RAM, except for the experiments on BERT LARGE for which we used a single NVIDIA
RTX 3090 Ti GPU with 24 GB of RAM instead.
In all settings we consider, we compare our method with baselines DLG [43] and TAG [3] discussed in Sec. 2.
As TAG does not have public code, we use our own implementation, and remark that the results obtained using our implementation are similar or better than those reported in Deng et al.
We consider two variants of our tool, LAMP Cos and LAMP L 2 + L 1 , that use the L cos and L tag gradient matching losses for the continuous optimization.
For the BERT BASE and TinyBERT 6 experiments, we run our attack with it = 30 , n c = 75 and n d = 200 , and stop the optimization early once we reach a total of 2000 continuous optimization steps.
For the BERT LARGE model, whose number of parameters make the optimization harder, we use it = 25 and n c = 200 instead, resulting in 5000 continuous optimization steps.
We run DLG and TAG for 10000 optimization steps on BERT LARGE and 2500 on all other models.
For the continuous optimization, we use Adam [17] with a learning rate decay factor γ applied everysteps for all methods and experiments, except for BERT LARGE ones where, following Geiping et al. [8], we use AdamW [21] and linear learning rate decay schedule applied every step.
We picked the hyperparameters for TAG, LAMP Cos and LAMP L 2 + L 1 , separately on CoLA and RottenTomatoes using grid search on BERT BASE and applied them to all networks.
As the optimal hyperparameters for
RottenTomatoes exactly matched the ones on CoLA, we used the same hyperparameters on SST-2, as well.
The resulting hyperparameters for LAMP Cos are α lm = 0 .
2 , α reg = 1 . 0 , λ = 0 .
01 , γ = 0 .
In contrast, the best hyperparameters for LAMP L 2 + L 1 are α tag
= 0 .
01 , α lm = 60 , α reg = 25 , λ = 0 .
01 , γ = 0 .
89 , as the loss L tag is on a different order of magnitude compared to L cos .
Finally, TAG’s best hyperparameters are α tag
= 0 .
01 , λ = 0 .
1 , γ = 1 .
(no decay), which we also use for DLG (with α tag = 0 . 0 ). To account for the different optimizer used in BERT LARGE experiments, we tuned the learning rate λ for all methods separately in this setting, resulting in λ = 0 . 1 for DLG, and λ = 0 .
01 for TAG, LAMP
Cos , and LAMP L 2 + L 1 .
For our methods, we additionally applied a two step initialization procedure.
We ﬁrst initialized the embedding vectors with 500 random samples from a standard Gaussian distribution and picked the best one according to L grad ( x ) .
We then computedpermutations on the best initialization and chose the best one in the same way. We investigate the effect of this procedure in App. C.3. Further details on our experimental setup are shown in App.
D. Main Experiments We evaluate the two variants of LAMP against DLG [43] and TAG [3] on
BERT BASE , BERT LARGE , and TinyBERT 6 .
Additionally, we evaluate attacks after BERT BASE has already been ﬁne-tuned for 2 epochs on each task (following Devlin et al.
[4]), as Balunovi´c et al. [Table]
showed that in the vision domain it is signiﬁcantly more difﬁcult to attack already trained networks.
For both baselines and our attacks, for simplicity we assume the lengths of sequences are known, as otherwise an adversary can simply run the attack for all possible lengths.
The results are shown in Table 1.
From the ROUGE-1 metric, we can observe that we recover more tokens than the baselines in all settings.
Moreover, the main advantage of LAMP is that the order of tokens in the reconstructed sequences matches the order in target sequences much more closely, as evidenced by the large increase in ROUGE-2 (5 × on CoLA).
This observation is further backed by the ROUGE-L metric that shows we are able to reconstruct up to 23% longer subsequences on the BERT BASE model on average compared to the baselines.
These results conﬁrm our intuition that using GPTto guide the search allows us to reconstruct sequences that are a much closer match to the original sequences.
We point out that Table 1 reafﬁrms the observations ﬁrst made in Deng et al.
[3], that DLG is consistently worse in all metrics compared to both TAG and LAMP and that the much longer sequences in RottenTomatoes still pose challenges to good reconstruction.
Further, our results show that even smaller and ﬁne-tuned models leak signiﬁcant amount of client information. In particular, TinyBERT 6 is even more vulnerable than BERT BASE and BERT BASE -FT is shown to be only slightly worse in reconstruction compared to BERT BASE , which is surprising given the prior image domain results.
This shows that smaller models can not resolve the privacy issue, despite previous suggestions in Deng et al. Additionally, our BERT LARGE experiments reafﬁrm the observation in Deng et al.
[3] that the model is highly vulnerable to all attacks.
Additionally, we examine the variability of our LAMP Cos method with respect to random initialization.
To this end, we ran the BERT BASE experiment on CoLA with 10 random seeds, which produced R-1, R-2 and R-L of 88 .
2 ± 1 .
± 2 . 37 , 75 . 0 ± 1 . 21 , respectively, which suggests that our results are consistent.
Further, we assess the variability with respect to sentence choice in App.
C.1. Unlike prior work, we also evaluate the different attacks on updates computed on batch sizes > 1 on the BERT BASE model to investigate whether we can reconstruct some of the sequences in this more challenging setting. The results are shown in Table 2.
Similarly to the results in Table 1, we observe that we obtain better results than the baselines on all ROUGE metrics in all experiments, except on RottenTomatoes with batch size 2, where TAG obtains slightly better ROGUE-1.
Our experiments show that for larger batch sizes we can also reconstruct signiﬁcant portions of text (see experiments on CoLA and SST-2). To the best of our knowledge, we are the ﬁrst to show this, suggesting that gradient leakage can be a realistic security threat in practice. Comparing the results for LAMP L 2 + Land LAMP Cos , we observe that L cos is better than L tag in almost all metrics on batch size 1 , across models, but the trend reverses as batch size is increased.
Finally, we show sample sequence reconstructions from both LAMP and the TAG baseline on CoLA with B = 1 in Table 3.
In Table 3, with green we denote correctly reconstructed bigrams, and with yellow correct unigrams.
We can observe that our reconstruction is more coherent, and that [Table] [Table] it qualitatively outperforms the baseline.
In App.
B, we further show the convergence rate of our optimization procedure compared to the baselines on an example sequence [Table] suggesting that LAMP can converge faster than TAG and DLG [Table]
Ablation Studies
In the next experiment [Table] we perform ablation studies to examine the inﬂuence of each proposed component of our method.
We compare the following variants of LAMP: (i) with cosine loss, (ii) with L 1 + L 2 loss, (iii) with L 2 loss, (iv) without the language model ( α lm = 0 ), (v) without embedding regularization ( α reg = 0 ), (vi) without alternating of the discrete and continuous optimization steps — executing it ∗ n c continuous optimization steps ﬁrst, followed by it discrete optimizations with n d steps each, (vii) without discrete transformations ( n d = 0 ).
For this experiment, we use the CoLA dataset and BERT BASE with B = 1 .
We show the results in Table 4.
We observe that LAMP achieves good results with both losses, though cosine is generally better for batch size 1 .
More importantly, dropping any of the proposed features makes ROUGE-1 and ROUGEsigniﬁcantly worse.
We note the most signiﬁcant drop in ROUGE-2 reconstruction quality happens when using transformations without using the language model (LAMP α lm =0 ), which performs even worse than doing no transformations (LAMP NoDiscrete ) at all.
This suggests that the use of the language model is crucial to obtaining good results.
Further, we observe that our proposed scheme for alternating the continuous and discrete optimization steps is important, as doing the discrete optimization at the end (LAMP DiscreteAtEnd ) for the same number of steps results in reconstructions only marginally better (in ROUGE-2) compared to the reconstructions obtained without any discrete optimization (LAMP NoDiscrete ).
The experiments also conﬁrm usefulness of other features such as embedding regularization. [Table] [Table]
Attacking the Defended Networks So far, all experiments assumed that clients have not defended against data leakage.
Following work on vision attacks [43, 38], we now consider the defense of adding Gaussian noise to gradients (with additional clipping this would correspond to DP-SGD [1]).
Note that, as usual, there is a trade-off between privacy and accuracy: adding more noise will lead to better privacy, but make accuracy worse.
We measure the performance of the ﬁne-tuned models on CoLA using the MCC metric [23] for which higher values are better.
We ﬁne-tuned for 2 epochs with different Gaussian noise levels σ and obtained the MCC scores depicted in Table 5.
We did not explore noises > 0 .
due to the signiﬁcant drop in MCC from 0 .
557 for the undefended model to 0 .
We show the results of our experiments on these defended networks in Table 5.
While all methods’ reconstruction metrics degrade, as expected, we see that most text is still recoverable for the chosen noise levels.
Moreover, our method is still signiﬁcantly better than the baselines, and thus shows the importance of evaluating defenses with strong reconstruction attacks.
In App.
C.2 we also show that LAMP is useful against a defense which masks some percentage of gradients.
Attacking FedAvg So far, we have only considered attacking the FedSGD algorithm.
In this experiment, we apply our attack on the commonly used FedAvg [19] algorithm.
As NLP models are often ﬁne-tuned using small learning rates (2e-5 to 5e-5 in the original BERT paper), we ﬁnd that
FedAvg reconstruction performance is close to FedSGD performance with batch size multiplied by the number of FedAvg steps.
We experimented with attacking FedAvg with 4 steps using B = 1 per step, lr = 5 e- 5 on CoLA and BERT BASE with LAMP L 2 + L 1 .
We obtained R-1, R-2 and R-L of 66 .
5 , 21 . 0 , 55 . 1 , respectively, comparable to the reported results on FedSGD with B = 4 .
Conclusion
In this paper, we presented LAMP, a new method for reconstructing private text data from gradients by leveraging language model priors and alternating discrete and continuous optimization.
Our extensive experimental evaluation showed that LAMP consistently outperforms prior work on datasets of varying complexity and models of different sizes.
Further, we established that LAMP is able to reconstruct private data in a number of challenging settings, including bigger batch sizes, noisedefended gradients, and ﬁne-tuned models.
Our work highlights that private text data is not sufﬁciently protected by federated learning algorithms and that more work is needed to alleviate this issue.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We demonstrate the effectiveness of existing defenses against our attack in Sec. 5, further we outline that our work does not deal with reconstructing labels which is left for future work, as described in Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We provide a discussion in App.
F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We have reported the type of resources used in the Sec. We have reported the total amount of compute in App. E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] We use standard datasets and cite the authors instead.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] We use standard datasets.
We refer the reader to the original authors of the dataset for this discussion.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material A Discussion A.1 Threat Model Discussion In this section, we further discuss the threat model chosen by LAMP and compare it to the related work.
To make our attack as generic as possible, we relax common assumptions that have been exploited to reconstruct client’s data in the literature before. In particular, LAMP is applicable even if:
• The model’s word embeddings are not ﬁne-tuned. As the gradients of the word embedding vectors are non-zero for words that are contained in the client’s training data and zero otherwise, revealing the gradients to the server will allow it to easily obtain the client sequence up to reorder. This constitutes a serious breach of clients’ privacy and, thus, we assume the word embeddings are not trainable. • The model’s positional embeddings are not ﬁne-tuned.
Similarly to the word-embedding gradients, Lu et al.
[22] have recently demonstrated that for batch size B = 1 positionalembedding gradients can leak client’s full sequence.
To this end, we assume models without trainable positional embeddings.
• The model’s transformer blocks contain no bias terms.
Lu et al.
[22] have also shown that the popular attack by Phong et al.
[28] can be applied on the bias terms of transformer blocks to leak the client’s data.
To this end, we assume models without transformer block biases.
• The transformer model is ﬁne-tuned on a classiﬁcation task.
As language modeling tasks are usually self-supervised, they often feed the same data to the model both as inputs and outputs.
Based on this observation, Fowl et al.
[6] have recently shown that label reconstruction algorithms can be used to obtain the client’s word counts.
We thus assume the more challenging binary classiﬁcation setting.
• The server is honest-but-curious, i.e., it aims to learn as much as possible about clients’ data from gradients but does not tamper with the learning protocol.
While prior work has shown that a malicious server can force a client to leak much more data [6], this is orthogonal to our work.
We focus on the honest-server setting instead,
which is the harder setting to attack.
We note that the assumptions we make for our transformer networks can result in a small amount of accuracy loss on the ﬁnal ﬁne-tuned model, but preserve the client’s data privacy much better.
We emphasize that while LAMP focuses on attacks in the harder setting, it is also applicable to the simpler settings without modiﬁcation.
A.2 Improvements over Prior Work
In this section, we outline the differences between LAMP and TAG and we discuss how these differences help LAMP to signiﬁcantly improve its text data reconstruction from gradients compared to TAG.
A major difference between the two methods is the introduction of our discrete optimization that takes advantage of a GPT-2 language model to help reconstruct the token order better than TAG.
Our discrete optimization step is novel in several ways:
• It is based on a set of discrete transformations that ﬁx common token reordering problems arising from the continuous reconstruction.
• We take advantage of the perplexity computed by the existing language models such as GPT-2 to evaluate the quality of different discrete transformations.
[Table]
[Table]
• Finally, our discrete optimization is alternated with the continuous, allowing for both take advantage of the result of the other which ultimately results in better token order reconstruction (See our ablation in Sec. 5)
The other major difference between our method and existing work is the choice of the error function
L rec used in the continuous part of the optimization.
Our choice of reconstruction loss results in better reconstruction of individual tokens and thus increases R-1.
In particular, we show that the cosine error function, previously applied in the image domain, can often outperform the error function suggested by TAG for text reconstruction and introduce a regularization term L reg that helps the continuous optimization to converge faster to more accurate embeddings using prior knowledge about the vector sizes of embeddings.
B Detailed Text Reconstruction Example
In Table 6, we show the intermediate steps of text reconstruction for a real example taken from our experiments presented in Table 3.
We can observe that LAMP Cos reaches convergence signiﬁcantly faster than the TAG baseline, and that after only 500 iterations most words are already reconstructed by our method.
C Additional Experiments C.1 Dependency of Experimental Results to the Chosen Sentences
Throughout this paper, we conducted our experiments on the same 100 sequences randomly chosen from the test portion of the datasets we attack.
In this experiment, we show that our results are consistent when different sets of 100 sequences are used.
To achieve this, we ran the BERT BASE
CoLA experiment with B=1 on additional 10 different sets of 100 randomly chosen sentences from the COLA test set.
We report the mean ± one standard deviation of the resulting R-1, R-2 and R-L metrics averaged across the 10 sets in Table 7.
We see that the results are consistent with our original ﬁndings.
[Table]
[Table]
C.2 Attacking Gradient Masking Defense
We experimented with a defense which zeroes out a percentage of elements in the gradient vector.
In Table 8 we vary the percentage and report MCC, R-1 and R-2.
While zeroing out most gradients weakens the attack, it also reduces utility (MCC) of the model.
C.3 Dependance on the Number of Initializations
In this section
[Table]
we investigate the inﬂuence of our proposed initialization on the reconstructions of
LAMP Cos by comparing a single random initialization ( n init = 1 ) with using our two-step initialization procedure with n init =
500 on the BERT BASE model and batch size of 1 .
The results are shown in
Table 9.
We observe that the two-step initialization scheme consistently improves individual token recovery (measured in terms of R-1) but may in some cases slightly degrade token ordering results (measured in terms of R-2).
Even though we used two-step initialization in the paper (it is strictly better on one dataset and non-comparable to single random initialization on the remaining datasets), it is indeed sometimes possible to get slightly better R-2 results with the latter.
D Additional Experimental Details
As we explain in Sec. 5, we choose the hyperparameters of our methods using a grid search approach on the CoLA and RottenTomatoes datasets.
For CoLA, we ﬁrst evaluated 50 hyperparameter combinations on 10 randomly selected (in a stratiﬁed way with respect to length) sequences from the training set (after removing the 100 test sequences).
Then, we further evaluated the bestcombinations on different 20 sequences from the training set.
For RottenTomatoes, we picked the hyperparameters from the same 10 best combinations and evaluated them on the sameadditional sequences.
For both LAMP Cos and the baselines we investigated the following ranges for the hyperparameters: α lm ∈ [0 . 05 , 0 . 2] , α reg ∈ [0 .
01 , 1] , λ ∈ [0 .
001 , 0 . 5] , γ ∈ [0 . 8 , 1] , and α tag ∈ [10 − 5 , 10 2 ] .
For LAMP L 2 + L 1 , we consider α lm ∈ [30 , 240] and α reg ∈ [10 , 100] , as the scale of the loss values is orders of magnitude larger than in LAMP
Cos .
We experimentally found that our algorithm is robust with respect to the exact values of n c and n d , provided that they are sufﬁciently large.
To this end, we select n d = 200 because we found that the selected token order from therandom transformations is close to the optimal one according to L rec ( x ) +
α lm L lm ( t ) for the sentence lengths present in our datasets.
Similarly, we set n c = 75 ( n c = 200 for BERT LARGE ) which allows our continuous optimization to signiﬁcantly change the embeddings before applying the next discrete optimization step in the process.
Finally, we also observed the performance of our algorithm is robust with respect to n init so we set it to 500 throughout the experiments.
We note that compared to TAG and DLG, the only additional hyperparameters we have to search over are α reg and α lm which makes the grid search feasible for our methods.
To account for the different optimizer used in BERT LARGE experiments, we tuned the learning rate λ for all methods separately in this setting by evaluating each method on 5 different learning rates in the range [0 . 01 , 0 . 1] on 10 randomly selected sentences from the CoLA dataset.
We applied the chosen values of λ to all 3 datasets.
Additionally, following Geiping et al.
[8] we clip the gradient magnitudes (cid:107)∇ x L rec ( x ) (cid:107) 2 for our BERT LARGE experiments to 1 . 0 for DLG and TAG and 0 .
5 for
LAMP Cos and LAMP L 2 + L 1 .
E Total Runtime of the Experiments
The BERT LARGE model experiments in Table 1 were the most computationally expensive to execute.
They took between 50 hours per experiment for the LAMP Cos and LAMP L 2 + L 1 methods andhours for TAG, which executes two times more continuous optimization steps.
Our experiments on the rest of the networks for both the baselines and our methods on batch size 1 ( B = 1 ) all took between 8 and 16 hours to execute on a single GPU with our methods being up to 2x slower due to the additional computational cost of our discrete optimization.
Additionally, our experiments on batch size 4 ( B = 4 ) took between 8 and 36 hours to execute on a single GPU with our methods being up to 4x slower due to the additional computational cost of our discrete optimization.
F Potential Negative Societal Impact of This Work
Our work is closely related to the existing works on gradients leakage attacks (See Sec. 2) which are capable of breaking the privacy promise of federated learning e.g. Zhao et al.
[41], Geiping et al.
[7], Yin et al.
[40], Deng et al.
Similar to these works, LAMP can be used to compromise the privacy of client data in real-world federated learning setups, especially when no defenses are used by the clients.
Our attack emphasizes that text data, which is commonly used in federated settings [30], is highly vulnerable to gradient leakage attacks, similarly to data in other domains, and that when federated learning is applied in practice extra steps need to be taken to mitigate the potential risks.
Further, in line with the related work, we study a range of possible mitigations to our attack in
Tables 1, 5 and 8 in the paper, thus promoting possible practical federated learning implementations that will be less vulnerable including those defended with Gaussian noise and gradient pruning and those using bigger batch sizes.