LAMP: Extracting Text from Gradients with Language Model Priors
Abstract
Recent work shows that sensitive user data can be reconstructed from gradient updates, breaking the key privacy promise of federated learning.
While success was demonstrated primarily on image data, these methods do not directly transfer to other domains such as text.
In this work, we propose LAMP, a novel attack tailored to textual data, that successfully reconstructs original text from gradients.
Our attack is based on two key insights: (i) modeling prior text probability with an auxiliary language model, guiding the search towards more natural text, and (ii) alternating continuous and discrete optimization, which minimizes reconstruction loss on embeddings, while avoiding local minima by applying discrete text transformations.
Our experiments demonstrate that LAMP is signiﬁcantly more effective than prior work: it reconstructs 5x more bigrams and 23% longer subsequences on average.
Moreover, we are the ﬁrst to recover inputs from batch sizes larger than 1 for textual models.
These ﬁndings indicate that gradient updates of models operating on textual data leak more information than previously thought.
Introduction
Federated learning [24] (FL) is a widely adopted framework for training machine learning models in a decentralized way.
Conceptually, FL aims to enable training of highly accurate models without compromising client data privacy, as the raw data never leaves client machines.
However, recent work [28, 43, 41] has shown that the server can in fact recover the client data, by applying a reconstruction attack on the gradient updates sent from the client during training.
Such attacks typically start from a randomly sampled input and modify it such that its corresponding gradients match the gradient update originally sent by the client.
While most works focus on reconstruction attacks in computer vision, there has comparatively been little work in the text domain, despite the fact that some of the most prominent applications of FL involve learning over textual data, e.g., next-word prediction on mobile phones [30].
A key component of successful attacks in vision has been the use of image priors such as total variation [7].
These priors guide the reconstruction towards natural images, which are more likely to correspond to client data.
However, the use of priors has so far been missing from attacks on text [43, 3], limiting their ability to reconstruct real client data.
This work: private text reconstruction with priors In this work, we propose LAMP, a new reconstruction attack which leverages language model priors to extract private text from gradients.
The overview of our attack is given in Fig.
The attacker has access to a snapshot of the transformer network being trained in a federated manner (e.g., BERT), and a gradient ∇ θ L ( x ∗ , y ∗ ) which the client has computed on that snapshot, using their private data.
The attack starts by sampling token ∗ Equal contribution.
Key component of our reconstruction attack is the use of a language model prior combined with a search that alternates continuous and discrete optimization steps.
Our experimental evaluation demonstrates the effectiveness of this approach—LAMP is able to extract text from state-of-the-art transformer models on several common datasets, reconstructing up to 5 times more bigrams than prior work.
Moreover, we are the ﬁrst to perform text reconstruction in more complex settings such as batch sizes larger than 1, ﬁne-tuned models, and defended models.
Overall, across all settings we demonstrate that LAMP is effective in reconstructing large portions of private text.
Main contributions Our main contributions are:
• LAMP, a novel attack for recovering input text from gradients, which leverages an auxiliary language model to guide the search towards natural text, and a search procedure which alternates continuous and discrete optimization.
• An implementation of LAMP and its extensive experimental evaluation, demonstrating that it can reconstruct signiﬁcantly more private text than prior work.
We make our code publicly available at https://github.com/eth-sri/lamp .
• The ﬁrst thorough experimental evaluation of text attacks in more complex settings such as larger batch sizes, ﬁne-tuned models and defended models.
Related Work
Federated learning [24, 18] has attracted substantial interest [16] due to its ability to train deep learning models in a decentralized way, such that individual user data is not shared during training.
Instead, individual clients calculate local gradient updates on their private data, and share them with a centralized server, which aggregates them to update the model [24].
The underlying assumption is that user data cannot be recovered from gradient updates.
Recently, several works [28, 43, 41, 42] demonstrated that gradients can in fact still leak information, invalidating the fundamental privacy assumption.
Moreover, recent work achieved near-perfect image reconstruction from gradients [7, 40, 14].
Interestingly, prior work showed that an auxiliary model [14] or prior information [2] can signiﬁcantly improve reconstruction quality.
Finally, Huang et al.
[12] noticed that gradient leakage attacks often make strong assumptions, namely that batch normalization statistics and ground truth labels are known.
In our work, we do not assume knowledge of batch normalization statistics and as we focus on binary classiﬁcation tasks, we can simply enumerate all possible labels.
Despite substantial progress on image reconstruction, attacks in other domains remain challenging, as the techniques used for images rely extensively on domain speciﬁc knowledge.
In the domain of text, in particular, where federated learning is often applied [32], only a handful of works exist [43, 3, 22].
DLG [43] was ﬁrst to attempt reconstruction from gradients coming from a transformer;
TAG [3] extended DLG by adding an L 1 term to the reconstruction loss; ﬁnally, unlike TAG and
DLG which are optimization-based techniques, APRIL [22] recently demonstrated an exact gradient leakage technique applicable to transformer networks.
However, APRIL assumes batch size of 1 and learnable positional embeddings, which makes it simple to defend against.
Another attack on NLP is given in Fowl et al.
[6], but they use stronger assumption that the server can send malicious updates to the clients.
Furthermore, there is a concurrent work [11] on reconstructing text from transformers, but it is limited to the case when token embeddings are trained together with the network.
Finally, there have been several works attempting to protect against gradient leakage.
Works based on heuristics [34, 31] lack privacy guarantees and have been shown ineffective against stronger attacks [2], while those based on differential privacy do train models with formal guarantees [1], but typically hurt the accuracy of the trained models as they require adding noise to the gradients.
We remark that we also evaluate LAMP on defended networks.
Background
In this section, we introduce the background necessary to understand our work.
3.1 Federated Learning
In federated learning, C clients aim to jointly optimize a neural network f with parameters θ on their private data.
At iteration k , the parameters θ k are sent to all clients, where each client c executes a gradient update ∇ θ k L ( x ∗ c , y ∗ c ) on a sample ( x ∗ c , y ∗ c ) from their dataset ( X c , Y c ) .
The updates are sent back to the server and aggregated.
While in Sec.
5 we experiment with both FedSGD [24] and
FedAvg [19] client updates, throughout the text we assume that clients use FedSGD updates:
[Equation]
Gradient leakage attacks A gradient leakage attack is an attack executed by the server (or a party which compromised it) that tries to obtain the private data ( x ∗ c , y ∗ c ) of a client using the gradient updates ∇ θ k L ( x ∗ c , y ∗ c ) sent to the server.
Gradient leakage attacks usually assume honest-but-curious servers which are not allowed to modify the federated training protocol outlined above.
A common approach, adopted by Zhu et al.
[43], Zhao et al.
[41], Deng et al.
[3] as well as our work, is to obtain the private data by solving the optimization problem:
[Equation]
where δ is some distance measure and ( x c , y c ) denotes dummy data optimized using gradient descent to have similar gradients ∇ θ k L ( x c , y c ) to true data ( x ∗ c , y ∗ c ) .
Common choices for δ are L 2 [43],
L 1 [3] and cosine distances [7].
When the true label y ∗ c is known, the problem reduces to
[Equation]
which was shown [41, 12] to be simpler to solve with gradient descent approaches.
3.2 Transformer Networks
In this paper, we focus on the problem of gradient leakage of text on transformers [35].
Given some input text, the ﬁrst step is to tokenize it into tokens from some ﬁxed vocabulary of size V .
Each token is then converted to a 1 -hot vector denoted t 1 , t 2 , . .
. , t n ∈ R V , where n is the number of tokens in the text.
The tokens are then converted to embedding vectors x 1 , x 2 , . . .
, x n ∈ R d , where d is a chosen embedding size, by multiplying by a trained embedding matrix W embed ∈ R V × d [4].
The rows of W embed represent the embeddings of the tokens, and we denote them as e 1 , e 2 , . . .
, e V ∈ R d .
In addition to tokens, their positions in the sequence are also encoded using the positional embedding matrix W pos ∈ R P × d , where P is the longest allowed token sequence.
The resulting positional embeddings are denoted p 1 , p 2 , . . .
, p n .
For notational simplicity, we denote e =
e 1 , e 2 , . .
, e V , x = x 1 , x 2 , . . .
, x n and p = p 1 , p 2 , . .
. , p n .
We use the token-wise sum of the embeddings x and p as an input to a sequence of self-attention layers [35].
The ﬁnal classiﬁcation output is given by the ﬁrst output of the last self-attention layer that undergoes a ﬁnal linear layer, followed by a tanh .
3.3 Calculating Perplexity on Pretrained Models
In this work, we rely on large pretrained language models, such as GPT-2 [29], to assess the quality of the text produced by the continuous part of our optimization.
Such models are typically trained to calculate the probability P ( t n | t 1 , t 2 , ..., t n − 1 ) of inserting a token t n from the vocabulary of tokens to the end of the sequence of tokens t 1 , t 2 , ..., t n − 1 .
Therefore, such models can be leveraged to calculate the likelihood of a sequence of tokens t 1 , t 2 , ..., t n , as follows:
[Equation]
One can use the likelihood, or the closely-related negative log-likelihood, as a measure of the quality of a produced sequence.
However, the likelihood depends on the length of the sequence, as probability decreases with length.
To this end, we use the perplexity measure [13], deﬁned as:
[Equation]
In the discrete part of our optimization, we rely on this measure to assess the quality of reconstructed sequences produced by the continuous part.
Extracting Text with LAMP
In this section we describe the details of our attack which alternates between continuous optimization using gradient descent, presented in Sec. 4.2, and discrete optimization using language models to guide the search towards more natural text reconstruction, presented in Sec. 4.3.
4.1 Notation
We denote the attacked neural network and its parameters with f and θ , respectively.
Further, we denote the client token sequence and its label as ( t ∗ , y ∗ ) , and our reconstructions as ( t , y ) .
For each token t ∗ i in t ∗ and t i in t , we denote their embeddings with x ∗ i ∈ R d and x i ∈ R d , respectively.
Moreover, for each token in our vocabulary, we denote the embedding with e i ∈ R d .
We collect the individual embeddings x i , x ∗ i , and e i into the matrices x ∈ R d × n , x ∗ ∈ R d × n and e ∈ R d × V , where n is the number of tokens in t ∗ and V is the size of the vocabulary.
4.2 Continuous Optimization
We now describe the continuous part of our attack (blue in Fig. 1).
Throughout the paper, we assume knowledge of the ground truth label y ∗ of the client token sequence we aim to reconstruct, meaning y =
y ∗ .
This assumption is not a signiﬁcant restriction as we mainly focus on binary classiﬁcation, with batch sizes such that trying all possible combinations of labels is feasible.
Moreover, prior work [9, 40] has demonstrated that labels can easily be recovered for basic network architectures, which can be adapted for transformers in future work.
We initialize our reconstruction candidate by sampling embeddings from a Gaussian and pick the one with the smallest reconstruction loss.
Reconstruction loss A key component of our attack is a loss measuring how close the reconstructed gradient is to the true gradient.
Assuming an l -layer network, where θ i denotes the parameters of layer i , an option is to use the combination of L 2 and L 1 loss proposed by Deng et al.
L
[Equation]
=
[Equation]
where α tag is a hyperparameter.
Another option is to use the cosine reconstruction loss proposed by Geiping et al.
[7] in the image domain:
[Equation]
Naturally, LAMP can also be instantiated using any other loss.
Interestingly, we ﬁnd that there is no loss that is universally better, and the effectiveness is dataset dependent.
Intuitively, L 1 loss is less sensitive to outliers, while cosine loss is independent of the gradient norm, so it works well for small gradients.
Thus, we set the gradient loss L grad to either L tag or L cos , depending on the setting.
Embedding regularization In the process of optimizing the reconstruction loss, we observe the resulting embedding vectors x i often steadily grow in length.
We believe this behavior is due to the self-attention layers in transformer networks that rely predominantly on dot product operations.
As a result, the optimization process focuses on optimizing the direction of individual embeddings x i , disregarding their length.
To address this, we propose an embedding length regularization term:
[Equation]
The regularizer forces the mean length of the embeddings of the reconstructed sequence to be close to the mean length of the embeddings in the vocabulary.
The ﬁnal gradient reconstruction error optimized in LAMP is given by:
[Equation]
where α reg is a regularization weighting factor.
Optimization We summarize how described components work together in the setting of continuous optimization.
To reconstruct the token sequence t ∗ , we ﬁrst randomly initialize a sequence of dummy token embeddings x = x 0 x 1 . . .
x n , with x i ∈ R d .
Following prior work on text reconstruction from gradients [3, 43], we apply gradient descent on x to minimize the reconstruction loss L rec ( x ) .
To this end, a second-order derivative needs to be computed, as L rec ( x ) depends on the network gradient at x .
Similar to prior work [3, 43], we achieve this using automatic differentiation in Pytorch [27].
4.3 Discrete Optimization
Next, we describe the discrete part of our optimization (green in Fig. 1).
While continuous optimization can often successfully recover token embeddings close to the original, they can be in the wrong order, depending on how much positional embeddings inﬂuence the output.
For example, reconstructions corresponding to sentences “weather is nice.”
and “nice weather is.”
might result in a similar reconstruction loss, though the ﬁrst reconstruction has a higher likelihood of being natural text.
To address this issue, we perform several discrete sequence transformations, and choose the one with both a low reconstruction loss and a low perplexity under the auxiliary language model.
Generating candidates Given the current reconstruction x = x 1 x 2 ... x n , we generate candidates for the new reconstruction x (cid:48) using one of the following transformations:
• Swap : We select two positions i and j in the sequence uniformly at random, and swap the tokens x i and x j at these two positions to obtain a new candidate sequence x (cid:48)
= x 1 x 2 . . .
x i − 1 x j x
i
+1 . . .
x j − 1 x i x j +1 . . .
x n .
• MoveToken :
Similarly, we select two positions i and j in the sequence uniformly at random, and move the token x i after the position j in the sequence, thus obtaining x (cid:48)
=
x 1 x 2 . . .
x i − 1 x
i
+1 . . .
x j − 1 x j x i x j +1 . . .
x n .
• MoveSubseq : We select three positions i , j and p (where i < j ) uniformly at random, and move the subsequence of tokens between i and j after position p .
The new sequence is thus x (cid:48)
=
x 1 x 2 . . .
x i − 1 x j +1 . . .
x p x i . . .
x j x p +1 . . .
x n .
• MovePreﬁx : We select a position i uniformly at random, and move the preﬁx of the sequence ending at position i to the end of the sequence.
The modiﬁed sequence then is x (cid:48)
=
x i +1 . . .
x n x 1 x 2 . . .
x i .
Next, we use a language model to check if generated candidates improve over the current sequence.
Using a language model to select candidates We accept the new reconstruction x (cid:48) if it improves the combination of the reconstruction loss and perplexity:
[Equation]
1: x ( k ) ∼ N (0 , I ) , where k
= 1 , ..., n init 2: x ← arg min x ( k ) L rec ( x ( k ) ) 3: for i = 1 to it do 4: for j = 1 to n c do 5: x ← x − λ ∇ x L rec ( x ) 6: end for 7:
x best ← x 8: t best ← P ROJECT T O V OCAB ( x best ) 9:
L best ← L rec ( x best ) + α lm L lm ( t best ) 10: for j = 1 to n d do 11: x (cid:48) ← T RANSFORM ( x ) 12: t (cid:48) ← P ROJECT T O V OCAB ( x (cid:48) ) 13:
L (cid:48)
← L rec ( x (cid:48) )
+ α lm L lm ( t (cid:48) ) 14: if L (cid:48) <
L best then 15: x best , t best , L best ← x (cid:48) , t (cid:48) , L (cid:48) 16: end if 17: end for 18: x ← x best 19: end for 20: return P ROJECT T O V OCAB ( x )
Here t and t (cid:48) denote sequences of tokens obtained by mapping each embedding of x and x (cid:48) to the nearest neighbor in the vocabulary according to the cosine distance.
The term L rec is the reconstruction loss introduced in Sec.
4.2, while L lm denotes the perplexity measured by an auxiliary language model, such as GPT-2.
The parameter α lm determines the trade-off between L rec and L lm : if it is too low then the attack will not utilize the language model, and if it is too high then the attack will disregard the reconstruction loss and only focus on the perplexity.
Going back to our example, assume that our reconstruction equals the second sequence “nice weather is.”.
Then, at some point, we might use the MoveToken transformation to move the word “nice“ behind the word “is” which would presumably keep the reconstruction loss similar, but drastically improve perplexity.
4.4 Complete Reconstruction Attack
We present our end-to-end attack in Algorithm 1.
We initialize the reconstruction x by sampling from a Gaussian distribution n init times, and choose the sample with minimal reconstruction loss as our initial reconstruction.
Then, at each step we alternate between continuous and discrete optimization.
We ﬁrst perform n c steps of continuous optimization to minimize the reconstruction loss (Lines 4-6, see Sec. 4.2).
Then, we perform n d steps of discrete optimization to minimize the combination of reconstruction loss and perplexity (Lines 10-17, see Sec. 4.3).
Finally, in Line 20 we project the continuous embeddings x to respective nearest tokens, according to cosine similarity.
Experimental Evaluation
We now discuss our experimental results, demonstrating the effectiveness of LAMP compared to prior work in a wide range of settings.
We present reconstruction results on several datasets, architectures, and batch sizes, together with the additional ablation study and evaluation of different defenses and training methods.
Datasets Prior work [3] has demonstrated that text length is a key factor for the success of reconstruction from gradients.
To this end, in our experiments we consider three binary classiﬁcation datasets of increasing complexity: CoLA [37] and SST-2 [33] from GLUE [36] with typical sequence lengths between 5 and 9 words, and 3 and 13 words, respectively, and RottenTomatoes [26] with typical sequence lengths between 14 and 27 words.
The CoLA dataset contains English sentences from language books annotated with binary labels describing if the sentences are grammatically correct, while SST-2 and RottenTomatoes contain movie reviews annotated with a binary sentiment.
For all experiments, we evaluate the methods on 100 random sequences from the respective training sets.
We remark that attacking in binary classiﬁcation setting is a more difﬁcult task than in the masking setting considered by prior work [43], where the attacker can utilize strictly more information.
Models Our experiments are performed on different target models based on the BERT [4] architecture.
The main model we consider is BERT BASE , which has 12 layers, 768 hidden units,feed-forward ﬁlter size, and 12 attention heads.
To illustrate the generality of our approach with respect to model size, we additionally consider a larger model BERT LARGE , which has 24 layers, 1024 hidden units, 4096 feed-forward ﬁlter size, and 16 attention heads as well as a smaller model
TinyBERT 6 from Jiao et al.
[15] with 6 layers, 768 hidden units, feed-forward ﬁlter size of 3072 and 12 attention heads.
All models were taken from Hugging Face [39].
The BERT BASE and BERT LARGE were pretrained on Wikipedia [5] and BookCorpus [44] datasets, while TinyBERT 6 was distilled from BERT BASE .
We perform our main experiments on pretrained models, as this is the most common setting for training classiﬁcation models from text [25].
For the auxiliary language model we use the pretrained GPT-2 provided by Guo et al.
[10], trained on the same tokenizer used to pretrain our target BERT models.
Metrics Following TAG [3], we measure the success of our methods based on the ROUGE family of metrics [20].
In particular, we report the aggregated F-scores on ROUGE-1, ROUGE-2 and
ROUGE-L, which measure the recovered unigrams, recovered bigrams and the ratio of the length of the longest matching subsequence to the length of whole sequence.
When evaluating batch sizes greater than 1 , we exclude the padding tokens, used to pad shorter sequences, from the reconstruction and the ROUGE metric computation.
Experimental setup In all settings we consider, we compare our method with baselines DLG [43] and TAG [3] discussed in Sec.
As TAG does not have public code, we use our own implementation, and remark that the results obtained using our implementation are similar or better than those reported in Deng et al.
We consider two variants of our attack, LAMP Cos and LAMP L 2 + L 1 , that use the L cos and L tag gradient matching losses for the continuous optimization.
For the BERT BASE and TinyBERT 6 experiments, we run our attack with it = 30 , n c = 75 and n d = 200 , and stop the optimization early once we reach a total of 2000 continuous optimization steps.
For the
BERT LARGE model, whose number of parameters make the optimization harder, we use it = 25 and n c = 200 instead, resulting in 5000 continuous optimization steps.
We run DLG and TAG for 10optimization steps on BERT LARGE and 2500 on all other models.
For the continuous optimization, we use Adam [17] with a learning rate decay factor γ applied every 50 steps for all methods and experiments, except for BERT LARGE ones where, following Geiping et al.
[8], we use AdamW [21] and linear learning rate decay schedule applied every step.
We picked the hyperparameters for
TAG, LAMP Cos and LAMP L 2 + L 1 , separately on CoLA and RottenTomatoes using grid search on
BERT BASE and applied them to all networks.
As the optimal hyperparameters for RottenTomatoes exactly matched the ones on CoLA, we used the same hyperparameters on SST-2, as well.
To account for the different optimizer used for BERT LARGE models, we further tuned the learning rate λ for
BERT LARGE experiments separately, keeping the other hyperparameters ﬁxed.
Additionally, for our methods we applied a two-step initialization procedure.
We ﬁrst initialized the embedding vectors with 500 random samples from a standard Gaussian distribution and picked the best one according to L grad ( x ) .
We then computed 500 permutations on the best initialization and chose the best one in the same way.
The effect of this procedure is investigated in App.
C.3.
Further details on our experimental setup are shown in App.
D.
Main experiments We evaluate the two variants of LAMP against DLG [43] and TAG [3] on
BERT BASE , BERT LARGE , and TinyBERT 6 .
Additionally, we evaluate attacks after BERT BASE has already been ﬁne-tuned for 2 epochs on each task (following Devlin et al.
[4]), as Balunovi´c et al.
[2] showed that in the vision domain it is signiﬁcantly more difﬁcult to attack already trained networks.
For both baselines and our attacks, for simplicity we assume the lengths of sequences are known, as otherwise an adversary can simply run the attack for all possible lengths.
In the ﬁrst experiment we consider setting where batch size is equal to 1.
The results are shown in Table 1.
From the ROUGE-
[Table]
metric, we can observe that we recover more tokens than the baselines in all settings.
Moreover, the main advantage of LAMP is that the order of tokens in the reconstructed sequences matches the order in target sequences much more closely, as evidenced by the large increase in ROUGE-2 (5 × on
CoLA).
This observation is further backed by the ROUGE-L metric that shows we are on average able to reconstruct up to 23% longer subsequences on the BERT BASE model compared to the baselines.
These results conﬁrm our intuition that guiding the search with GPT-2 allows us to reconstruct sequences that are a much closer match to the original sequences.
We point out that Table 1 reafﬁrms the observations ﬁrst made in Deng et al.
[3], that DLG is consistently worse in all metrics compared to both TAG and LAMP, and that the signiﬁcantly longer sequences in RottenTomatoes still pose challenges to good reconstruction.
Our results show that smaller and ﬁne-tuned models also leak signiﬁcant amount of client information.
In particular, TinyBERT 6 is even more vulnerable than BERT BASE and BERT BASE -FT is shown to be only slightly worse in reconstruction compared to BERT BASE , which is surprising given the prior image domain results.
This shows that smaller models can not resolve the privacy issue, despite previous suggestions in Deng et al.
Additionally, our BERT LARGE experiments reafﬁrm the observation in Deng et al.
[3] that the model is highly vulnerable to all attacks.
Further, we examine the variability of our LAMP Cos method with respect to random initialization.
To this end, we ran the BERT BASE experiment on CoLA with 10 random seeds, which produced R-1,
R-2 and R-L of 88 .
2 ± 1 .
± 2 . 37 , 75 . 0 ± 1 . 21 , respectively, which suggests that our results are consistent.
Further, we assess the variability with respect to sentence choice in App.
C.1.
Larger batch sizes Unlike prior work, we also evaluate the different attacks on updates computed on batch sizes greater than 1 on the BERT BASE model to investigate whether we can reconstruct some sequences in this more challenging setting.
The results are shown in Table 2.
Similarly to the results in Table 1, we observe that we obtain better results than the baselines on all ROUGE metrics in all experiments, except on RottenTomatoes with batch size 2, where TAG obtains slightly better
ROGUE-1.
Our experiments show that for larger batch sizes we can also reconstruct signiﬁcant portions of text (see experiments on CoLA and SST-2).
To the best of our knowledge, we are the ﬁrst to show this, suggesting that gradient leakage can be a realistic security threat in practice.
Comparing the results for LAMP L 2 + L 1 and LAMP Cos , we observe that L cos is better than L tag in almost all metrics on batch size 1 , across models, but the trend reverses as batch size is increased.
Sample reconstructions We show sample sequence reconstructions from both LAMP and the TAG baseline on CoLA with B = 1 in Table 3, marking the correctly reconstructed bigrams with green and correct unigrams with yellow.
We can observe that our reconstruction is more coherent, and that it qualitatively outperforms the baseline.
In App.
B, we show the convergence rate of our method compared to the baselines on an example sequence, suggesting that LAMP can often converges faster.
[Table] [Table]
Ablation studies
In the next experiment
[Table]
we perform ablation studies to examine the inﬂuence of each proposed component of our method.
We compare the following variants of LAMP: (i) with cosine loss, (ii) with L 1 + L 2 loss, (iii) with L 2 loss, (iv) without the language model ( α lm = 0 ), (v) without embedding regularization ( α reg = 0 )
[Table]
(vi) without alternating of the discrete and continuous optimization steps—executing it · n c continuous optimization steps ﬁrst, followed by it discrete optimizations with n d steps each, (vii) without discrete transformations ( n d = 0 ).
For this experiment, we use the CoLA dataset and BERT BASE with B = 1 .
We show the results in Table 4.
We observe that LAMP achieves good results with both losses
[Table]
though cosine is generally better for batch size 1 .
More importantly, dropping any of the proposed features makes ROUGE-1 and ROUGEsigniﬁcantly worse.
We note the most signiﬁcant drop in ROUGE-2 reconstruction quality happens when using transformations without using the language model (LAMP α lm =0 ), which performs even worse than doing no transformations (LAMP NoDiscrete ) at all.
This suggests that the use of the language model is crucial to obtaining good results.
Further, we observe that our proposed scheme for alternating the continuous and discrete optimization steps is important, as doing the discrete optimization at the end (LAMP DiscreteAtEnd ) for the same number of steps results in reconstructions only marginally better (in ROUGE-2) compared to the reconstructions obtained without any discrete optimization (LAMP NoDiscrete ).
The experiments also conﬁrm usefulness of other features such as embedding regularization.
Attacking defended networks So far, all experiments assumed that clients have not defended against data leakage.
Following work on vision attacks [43, 38], we now consider the defense of adding Gaussian noise to gradients (with additional clipping this would correspond to DP-SGD [1]).
Note that, as usual, there is a trade-off between privacy and accuracy: adding more noise will lead
[Table] [Table]
to better privacy, but make accuracy worse.
We measure the performance of the ﬁne-tuned models on CoLA using the MCC metric [23] for which higher values are better.
The ﬁne-tuning was done for 2 epochs with different Gaussian noise levels σ , and we obtained the MCC scores depicted in
Table 5.
We did not explore noises > 0 .
due to the signiﬁcant drop in MCC from 0 .
557 for the undefended model to 0 .
The results of our experiments on these defended networks are presented in Table 5.
While all methods’ reconstruction metrics degrade, as expected, we see that most text is still recoverable for the chosen noise levels.
Moreover, our method still outperforms the baselines, and thus shows the importance of evaluating defenses with strong reconstruction attacks.
In App.
C. we show that LAMP is also useful against a defense which masks some percentage of gradients.
Attacking FedAvg So far, we have only considered attacking the FedSGD algorithm.
In this experiment, we apply our attack on the commonly used FedAvg [19] algorithm.
As NLP models are often ﬁne-tuned using small learning rates (2e-5 to 5e-5 in the original BERT paper), we ﬁnd that
FedAvg reconstruction performance is close to FedSGD performance with batch size multiplied by the number of FedAvg steps.
We experimented with attacking FedAvg with 4 steps using B = 1 per step, lr = 5 e- 5 on CoLA and BERT BASE with LAMP L 2 + L 1 .
We obtained R-1, R-2 and R-L of 66 .
5 , 21 . 0 , 55 . 1 , respectively, comparable to the reported results on FedSGD with B = 4 .
Conclusion
In this paper, we presented LAMP, a new method for reconstructing private text data from gradients by leveraging language model priors and alternating discrete and continuous optimization.
Our extensive experimental evaluation showed that LAMP consistently outperforms prior work on datasets of varying complexity and models of different sizes.
Further, we established that LAMP is able to reconstruct private data in a number of challenging settings, including bigger batch sizes, noisedefended gradients, and ﬁne-tuned models.
Our work highlights that private text data is not sufﬁciently protected by federated learning algorithms and that more work is needed to alleviate this issue.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We demonstrate the effectiveness of existing defenses against our attack in Sec. 5, further we outline that our work does not deal with reconstructing labels which is left for future work, as described in Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We provide a discussion in App.
F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We have reported the type of resources used in the Sec.
We have reported the total amount of compute in App.
E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] We use standard datasets and cite the authors instead.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] We use standard datasets.
We refer the reader to the original authors of the dataset for this discussion.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-1 , γ = 1 . 
-]