Recovering Private Text in Federated Learning of Language Models
Abstract
Federated learning allows distributed users to collaboratively train a model while keeping each user’s data private.
Recently, a growing body of work has demonstrated that an eavesdropping attacker can effectively recover image data from gradients transmitted during federated learning.
However, little progress has been made in recovering text data.
In this paper, we present a novel attack method FILM for federated learning of language models (LMs). For the ﬁrst time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences.
Unlike image-recovery methods that are optimized to match gradients, we take a distinct approach that ﬁrst identiﬁes a set of words from gradients and then directly reconstructs sentences based on beam search and a prior-based reordering strategy.
We conduct the FILM attack on several large-scale datasets and show that it can successfully reconstruct single sentences with high ﬁdelity for large batch sizes and even multiple sentences if applied iteratively. We evaluate three defense methods: gradient pruning, DPSGD, and a simple approach to freeze word embeddings that we propose. We show that both gradient pruning and DPSGD lead to a signiﬁcant drop in utility.
However, if we ﬁne-tune a public pre-trained LM on private text without updating word embeddings, it can effectively defend the attack with minimal data utility loss. Together, we hope that our results can encourage the community to rethink the privacy concerns of LM training and its standard practices in the future.
Introduction
Federated learning (McMahan et al., 2017) is a method to allow multiple participants to collaboratively train a global model without exchanging their private data.
At each step of training, a central server transmits model parameters to every participating client.
Each client then computes a model update (i.e., gradients) using its local data and sends it to the server.
Finally, the server aggregates all updates— typically by averaging them—and updates the model.
Federated learning is actively being considered for privacy-sensitive applications such as virtual mobile keyboards and analysis of electronic health records in hospitals (Li et al., 2020a).
[Figure]
However, recent studies (Zhu et al., 2019; Geiping et al., 2020; Pustozerova & Mayer, 2020; Lyu et al., 2020; Yin et al., 2021; Zhu & Blaschko, 2021; Huang et al., 2021; Gupta et al., 2021) show that an honest-but-curious eavesdropper in federated learning can recover private data of the client (Fig. 1).
However, success in recovering text data remains limited to unrealistically small batch sizes (see Sec. 2 for more details).
In this work, we study federated learning of neural language models (LMs) and present a novel attack called FILM ( F ederated I nversion Attack for L anguage M odels), which can recover private text data from information transmitted during training.
For the ﬁrst time, we demonstrate that an attacker can successfully recover sentences from large training batches of up to 128 sentences, making it practical and alarming in real-world scenarios.
We focus on language models for two reasons: (1)
Transformer-based (Vaswani et al., 2017) language models have become the backbone of modern NLP systems (Devlin et al., 2019; Radford et al., 2018, 2019; Brown et al., 2020) and are quickly adapted to a number of privacy-sensitive domains (Alsentzer et al., 2019;
Huang et al., 2019; Li et al., 2020b; Liu & Miller, 2020; Kraljevic et al., 2021); (2) We identify that training LMs is at a higher risk of information leakage by nature because the attacker can possibly leverage the memorization capability of LMs during federated learning.
Our FILM attack deviates from previous gradient-matching approaches for image data that directly optimize in high-dimensional, continuous space, which we ﬁnd to be ineffective for discrete text inputs and highly sensitive to initializations (Sec. 6).
Instead, we ﬁrst recover a set of words from the word embeddings’ gradients (Melis et al., 2019), and this step provides a set of words that appear in any sentences of the training batch without the frequency information 3 .
We then develop a simple and effective strategy using beam search (Reddy et al., 1977; Russell & Norvig, 2010), which attempts to reconstruct one sentence from the set of words.
This step is crucial as it takes advantage of prior knowledge encoded in pre-trained language models and memorization of training data during federated training.
We also further design a token-reordering approach that leverages both language prior and gradient information to reﬁne the recovered sentence.
Finally, we show that we can recover multiple sentences from the same training batch by iteratively applying the same procedure.
We evaluate FILM on two language modeling datasets: WikiText-103 (Merity et al., 2017) and Enron
Email (Klimt & Yang, 2004), based on a GPT-2 model (Radford et al., 2019), using either pre-trained or randomly-initialized weights.
We analyze the attack performance with different batch sizes, the number of training data points, and the number of training epochs.
Our experiments demonstrate high-ﬁdelity recovery of a single sentence feasible, and recovery of signiﬁcant portions of sentences for training batches of up to 128 sentences.
Furthermore, we ﬁnd that recovery of multiple sentences from the same batch ( 1 / 3 for a batch size of 16) is possible through repeated applications of FILM.
For defense, we ﬁrst evaluate two previously proposed defense methods, gradient pruning (Zhu et al., 2019) and Differentially Private Stochastic Gradient Descent (DPSGD) (Abadi et al., 2016) against the proposed attack and ﬁnd that both of them suffer substantial utility loss. We then propose a simple method to freeze the word embeddings of the model during training to prevent the critical ﬁrst stage of our attack and ﬁnd that (1) if we train an LM from the scratch on the private text, it will cause more utility loss; (2) however, if we start from a public LM (e.g., pre-trained GPT-2), it can effectively defend the FILM attack with minimal drop in utility.
Together, our work presents a strong attack on federated learning of LM training, and raises privacy concerns in current practices. We hope our new (and simple) proposal of freezing word embeddings from a public LM encourages reconsideration of standard practices for training LMs on privacy-sensitive domains.
Related work
Data recovery from gradients.
Gradients in federated learning have been shown to leak information of private data.
Prior works reconstruct private training images from gradients by treating the reconstruction as an optimization problem (Zhu et al., 2019; Zhao et al., 2020; Geiping et al., 2020;
Enthoven & Al-Ars, 2021; Yin et al., 2021; Zhu & Blaschko, 2021; Jin et al., 2021; Jeon et al., 2021). They iteratively manipulate reconstructed images to yield similar gradients as observed gradients computed on the private images, using regularization terms based on image priors such as total variation (Zhu et al., 2019; Geiping et al., 2020) and batch normalization statistics (Yin et al., 2021).
The ﬁrst attempt to recover text from gradients is Zhu et al.
(2019), which brieﬂy presented leakage results in masked language modeling (Devlin et al., 2019).
This attack matches target gradients with continuous representations and maps them back to words that are closest in the embedding matrix.
Deng et al.
(2021) improve the method by adding a regularization term that prioritizes gradient matching in layers closer to the input data.
Both approaches only work with a batch size of 1 (Sec. 6).
Recently, Dimitrov et al. (2022) propose a method to minimize the loss of gradient matching and the probability of prior text computed by an auxiliary language model and show the recovery of a batch of 4 sentences with binary classiﬁcation tasks with a BERT model. In addition, Boenisch et al.
Fowl et al.
(2022) recover text data in federated learning under a stronger threat model: the server is malicious and can manipulate the training model’s weights to enable easier text reconstruction.
Our work builds on the insight from Melis et al.
(2019) that show the information leakage from the embedding layer by observing non-zero entries in the gradients of the word embedding matrix.
Different from their attack which only infers a person or location from a shallow GRU model (Cho et al., 2014), we propose a much stronger attack that aims at recovering full sentences from larger language models. We include a high-level comparison of key differences between our approach and prior approaches in Table 6 in the appendix.
Memorization in language models.
Our method is also inspired by the work related to memorization of training data in language models.
Recent studies (Carlini et al., 2019; Thakkar et al., 2021;
Song & Raghunathan, 2020; Zanella-Béguelin et al., 2020; Carlini et al., 2021; Bender et al., 2021) notice that large language models can memorize their training dataset and thus can be prompted to output speciﬁc sensitive information.
While these previously demonstrated memorization attacks try to recover a small subset of sensitive text from a large training corpora, our attack aims at recovering sentences from every mini-batch in federated learning.
We stress that our attack is signiﬁcantly more powerful than the previous works studying memorization.
Defenses against gradient inversion attacks for text data. Cryptographic approaches such as encrypting gradients (Bonawitz et al., 2016) or encrypting the data and model (Phong et al., 2018) can guarantee secure training in a federated learning setting against gradient inversion attacks. However, practical deployment of these approaches often slows down model training and requires special setup. A few efﬁcient methods are proposed to defend gradient inversion attacks are for text data.
Zhu et al. (2019) mention brieﬂy adding differentially private noise to the gradients (Abadi et al., 2016), or setting gradients of small magnitudes to zero (i.e., gradient pruning), but these defenses typically hurt the accuracy of the trained models (Li et al., 2021; Yu et al., 2021). Huang et al. (2020b) propose
InstaHide that encodes each input training image to the neural network using a random pixel-wise mask and the MixUp data augmentation (Zhang et al., 2018).
The idea is later extended to language understanding tasks (Huang et al., 2020a): instead of ‘hiding’ the input word embeddings, they perform MixUp on [CLS] tokens of BERT models.
In this section, we describe relevant background of language models, federated learning, as well as the threat model of our proposed attack method.
3.1 Language Modeling
Language modeling is a core task in natural language processing and has been used as a building block of state-of-the-art pipelines (Radford et al., 2018; Brown et al., 2020).
Given a sequence of n tokens x = { x 1 , x 2 , ..., x n } , the language modeling task is to estimate the probability distribution of IP ( x ) :
[Equation]
Modern neural language models are parameterized by RNNs (Mikolov et al., 2010) or Transformers (Vaswani et al., 2017), which consist of millions or billions of parameters (denoted θ ).
A language model ﬁrst maps x to a sequence of vectors via a word embedding matrix W ∈ R |V|× d , where V is the vocabulary and d is the hidden dimension.
After computing the hidden representation h i conditioned on x 1 , . . .
, x i − 1 , the model predicts the probability of the next token as:
[Equation]
3.2 Federated Learning
Federated learning (FL) (McMahan et al., 2017) is a communication protocol for training a shared machine learning model on decentralized data.
An FL system usually involves N clients: c 1 , c 2 , · · · , c N
① Let ② The
Let the
The mouse
Let dogs …
Let the dogs
The mouse chases
Let dogs lie
Let the dogs lie.
The mouse chases the cat.
Let dogs lie sleeping.
“Let”, “The”, “cat”, “chases”, “dogs”,“lie”, “Mouse”, “sleeping”, “the”, “.” … … … … … … (
[Figure] Model
Gradient (2) Beam Search for Reconstruction
The cat chases the mouse.
[Figure] [Figure]
and a central server s , where the clients wish to collaboratively train a neural network f θ with their private data D 1 , D 2 , · · · , D N , under the coordination of the server. The training process optimizes θ (model parameters) using a loss function L and runs for T iterations. This work focuses on federated learning of language models, where clients can start from a randomly-initialized neural network, or a pre-trained public model (Hard et al., 2018).
At each iteration t ∈ [ T ] , an individual client c i , i ∈ [ N ] computes ∇ θ t L θ t ( B i ) , the gradients of the current model parameters θ t on a randomlysampled private data batch B i ⊂ D i , |B i | = b , and shares the gradients with the server s . The server then aggregates gradient updates from all clients and updates the model:
[Equation] [Equation]
where η is the learning rate. Finally, the updated parameters θ t +1 are broadcast to individual clients.
3.3 Threat Model
Adversary’s capabilities.
We consider an honest-but-curious adversary who is eavesdropping on the communication between the central server s and
an arbitrary client c
i , i ∈ [ N ] in the federated training of a language model parameterized by θ , as described above.
An adversary in this scenario has white-box access to 1) the gradients ∇ θ t L θ t ( B i ) sent by the client c i , and 2) the model parameters θ t (Fig. 1), including the vocabulary V and the embedding matrix W .
Note that the adversary can inspect these information at any training iterations t ∈ [ T ] .
Adversary’s objective.
The adversary aims to recover at least one sentence from the private training data batch B i based on the information they observe, as recovering a single sentence is already sufﬁcient to break the privacy guarantee of federated learning. Since each training batch consists of randomly selected sentences from the private training dataset, if the adversary could recover at least one sentence from each training batch during the whole training procedure, they would be able to recover a considerable fraction of the original training dataset.
Additionally, the attacker may also apply the attack on a single batch multiple times to recover more sentences from the batch (Sec. 4.5).
The strength of an attack will be measured by the similarity between the recovered sentence and its corresponding private sentence in the original batch.
Proposed Attack: FILM 4.1 Overview
In this section, we present a novel attack method FILM, which can construct sentences from any training batches ( |B i | ≤ 128 ).
Our method consists of three steps: (1) Inspired by Melis et al.
(2019), we ﬁrst recover a set of words that may appear in a private batch ( B i ) from gradients (Sec. 4.2).
While their goal is to identify individual words from gradients, our approach is able to recover the order of words in a sentence. (2) We attempt searching for full sentences using beam search using the set of words, based on the probability distribution IP θ t ( x ) , which leverages either the language prior from a pre-trained language model ( t = 0 ), or the memorization capability of the LM during training ( t > 0 )
(Sec. 4.3).(3)
Finally, we design a prior-based scoring function and demonstrate that we can further improve the quality of recovered sentences by performing a simple and lightweight reordering step (
We show that the above pipeline is capable of recovering single sentences from large training batches successfully.
In Sec. 4.5, we discuss that we can iterate the above procedure to recover multiple sentences from the same training batch, which makes our attack even stronger.
In the following, we describe our attack method in detail.
To better illustrate our attack’s functionality, we provide a simpliﬁed step-by-step example that recovers a batch of 2 simple sentences in Fig.
4.2 Bag-of-Words Extraction
[-Input: r ✓ t L ✓ t ( B i ) , gradients of the language model on the private batch of sentences. Output: bag of words B : {“Let", “The", “cat", “chases", “dogs", “lie", “mouse", “sleeping", “the", “."} .
-][Table]
Our ﬁrst step is to extract a bag of words from ∇ θ t L θ t ( B i ) , the gradients observed by an eavesdropper at the t -th iteration of training.
We follow the method outlined by Melis et al.
and recover a set of words B = { w 1 , w 2 , . .
w m } by considering the non-zero rows of the word embedding gradients ∇ W t L θ t ( B i ) .
Additionally, we are also able to recover the maximum length of sentences in the batch by analyzing the non-zero rows of position embeddings gradients ∇ P t L θ t ( B i ) .
6 Note that this step only determines the set of words but not their frequency .
In practice, we ﬁnd that augmenting our beam search with an n-gram penalty (Sec. 4.3) can compensate for this lack of information.
4.3 Beam Search for Sentence Reconstruction
[Table]
The second stage of our attack is to recover one sentence based on the extracted bag of words B and the learned probability distribution of text IP θ t ( x ) at training step t .
Since it is intractable to generate text directly using IP θ t ( x ) , we use the beam search (Reddy et al., 1977; Russell & Norvig, 2010), which is a greedy auto-regressive algorithm to generate sequences in a step-by-step manner with a trained language model.
We provide a detailed algorithm in Appendix A.
To initiate the beam search, we ﬁrst ﬁnd a set of words that can be chosen as starting tokens of sentences.
A straightforward method is to select words that begin with a capital letter from B .
For example, we would select “Let" and “My" as possible prompts for the illustrated batch.
Then at each beam search step, we keep the top k possible beam states, ﬁnd the k most likely next words to each beam by using the language model (see procedure T OP ( · ) in Appendix A, Algorithm 1), and keeps the top k new beam states.
The beam tree continues until the generated sentence reaches a speciﬁed length (we use 40 words), and outputs the text that has the overall highest probability (see Fig. 2).
As the bag of words B does not contain information on the frequency of each word in the sentence, and beam search is prone to generating repetitive content (Holtzman et al., 2020), we also include a penalty for repeated n -grams (Vijayakumar et al., 2016) in a sentence (see line 3 in procedure T OP ( · )).
Noticeably, we compare the performance of our attack with an oracle scenario where the attacker knows the frequency of each word in Fig. 5, and ﬁnd that adding an n-gram penalty ( n = 2 ) can achieve performance comparable without frequency information.
4.4 Prior-Guided Token Reordering
Input: a recovered sentence from Sec. 4.3, “The mouse chases the cat.
" Output: a reordered sentence, “The cat chases the mouse."
By construction, beam search is a greedy method that generates sentences from left to right.
Despite that it generally produces ﬂuent sentences by maximizing IP θ t ( x ) , it does not leverage the gradients information which is the main signal in previous image attack methods. We also notice that beam search may recover the general structure of the sentence, while failing to recover the order of speciﬁc phrases or words.
Thus, the ﬁnal stage of our attack is to reorder tokens in the extracted sentence from the previous step, with the objective to improve its quality measured by a prior-based scoring function.
The scoring function leverages both the perplexity and the gradient norm.
Formally, given a sentence x , its prior score S θ ( x ) is deﬁned as:
[Equation]
where log IP θ ( x ) is the log-likelihood of the sentence (Eq. 1) according to the model parameters θ , and β is a hyperparameter that controls the importance of gradient norm.
Appendix B.8 provides empirical analysis that supports the design of this scoring function.
We conduct two additional steps to reorder both phrases and tokens:
Phrase-wise reordering.
We ﬁrst preprocess the sentence x by removing its redundant segments 7 .
We then adjust the preprocessed sentence by reordering the phrases inside it.
At each iteration, we generate multiple candidates by ﬁrst cutting the sentence at p positions using the strategy in Malkin et al.
(2021), and then permuting the phrases between the cuts to form a new sentence.
We then select the best candidate measured by S θ ( · ) .
This stage terminates after 200 steps.
Token-wise reordering.
We then reorder individual tokens.
At each reordering step, we generate candidates using the following token-wise operations: 1) randomly swaps two tokens in the sentence, or 2) randomly deletes a token from the sentence, or 3) randomly inserts a token from the bag of words (recovered by Sec. 4.2) into the sentence.
We then select the best candidate measured by S ( · ) and move to the next iteration.
This stage terminates after 200 steps.
4.5 Recovering Multiple Sentences
The previous steps only consider the recovery of a single sentence from a training batch.
In this step, we describe an extension of our attack which is able to support recovery of multiple sentences from the same batch.
We ﬁrst perform beam search as described in Sec. 4.3.
We store the results and repeat the beam search again, except this time applying an additional n -gram penalty based on the results of the previous search.
Over many repetitions of this procedure, we are able to build a set of candidate recoveries for the original batch. Finally, we use the reordering method described in
Sec 4.4 to improve each candidate.
Defending Against the FILM Attack
We evaluate previously proposed defense methods, gradient pruning (Zhu et al., 2019) and DPSGD (Abadi et al., 2016) and ﬁnd that both are less effective in defending our attack (Sec. 6.3). To defend against the FILM attack, we propose a defense method which simply freezes the word embeddings of the model during training.
Our key insight is to focus on preventing the ﬁrst step of the attack, i.e., the recovery of the bag-of-words from word embedding gradients.
If the attack fails at this step, the later stages of the attack are no better than beam search over the entire vocabulary as with the prior attack (Carlini et al., 2021).
Similarly, previously proposed gradient inversion attacks (Zhu et al., 2019; Fowl et al., 2022) for smaller batch sizes also depend on this step.
By freezing the word embeddings during training, it is able to completely prevent the recovery of bag-of-words.
In other words, once we prevent the transmission of gradients of word embeddings during training, it is no longer possible to recover the bag-of-words. We consider the settings for both 1) training an LM from the scratch; 2) continuing training from a public, pre-trained LM (e.g., GPT-2) on the private text, when freezing the word embeddings. As we later show in Sec. 6.3, the ﬁrst setting causes more utility loss (because the word embeddings are randomly initialized) while the latter setting provides the best privacy-utility trade-off.
Since updating word embeddings is a common practice in training LMs, we suggest researchers and practitioners consider training from a public LM and freezing word embeddings in privacy-sensitive applications.
Experiments 6.1 Setup
Model and datasets.
We evaluate the proposed attack with the GPT-2 base (117M parameters) model (Radford et al., 2019) on two language modeling datasets, including WikiText-103 (Merity et al., 2017) and the Enron Email dataset (Klimt & Yang, 2004).
Both datasets are publicly available for research uses.
We choose WikiText-103 because it is commonly used in language modeling research, and Enron Email because it consists of private email messages that contain abundant private information such as individuals’ names, addresses, and even passwords.
Note that the GPT-2 model is not trained on any Wikipedia data, and very unlikely on the Enron Email data.
We evaluate our attack on a subset of WikiText-103 and Enron Email.
After preprocessing (more details provided in Appendix B), we have 203,456 sentences from the WikiText-103 dataset and 31,797 sentences from the Enron Email dataset.
Training and attack settings.
Following previous studies (Zhu et al., 2019;
Geiping et al., 2020;
Yin et al., 2021; Huang et al., 2021) for gradient inversion attacks, our evaluation considers a federated
[Table] [Table] learning setting with a single server and a single client. Unless otherwise noted, we train the model on these sentences for 90 , 000 iterations using an initial learning rate of 1 × 10 − 5 , with a linearly decayed learning rate scheduler.
All models were trained using early stopping, i.e., models were trained until the loss of the model on the evaluation set increased.
We note that the running time of our algorithm is quite fast, and we can recover a single sentence in under a minute using an Nvidia 2080TI GPU.
Evaluation metrics.
We use the following metrics to evaluate the attack performance: (a)
ROUGE (Lin, 2004) is a set of metrics for evaluating summarization of texts as well as machine translation.
We use ROUGE-1/2/L F-Scores to evaluate the similarity between the recovered and the original sentences, following (Deng et al., 2021; Dimitrov et al., 2022).
More speciﬁcally, ROUGE1/2 refer to the overlap of unigrams and bigrams between the recovered and the original text respectively.
ROUGE-L measures the longest matching subsequence.
For ablation studies we only show ROUGE-L as it is more representative of signiﬁcant leakage than ROUGE-1 or ROUGE-2.
(b) We also propose to use named entity recovery ratio (NERR) as the percentage of original named entities that can be perfectly recovered.
Since named entities usually contain sensitive information (e.g., names, addresses, dates, or events), NERR measures how well the attacker recovers such information, ranging from a complete mismatch (NERR = 0) to a perfect recovery (NERR = 1).
6.2 Performance of Our Attack
Scaling to large training batches.
We ﬁrst compare the performance with respect to different batch sizes.
In this setting, we start from a pre-trained GPT-2 model and ﬁne-tune it on both datasets respectively (WikiText-103 and Enron Email).
Fig.
3 shows the performance of our attack using different batch sizes with the WikiText-103 dataset and the Enron Email dataset.
We observe that for a batch size of 1, all four scores (ROUGE-1, ROUGE-2, ROUGE-L, NERR) are close to 1, indicating a near-perfect recovery.
We observe that as the batch size increases, the quality of the recovery by ROUGE-1, ROUGE-2, and NERR start to decrease.
We attribute this drop in performance to the bigger set of words in the same training batch, as well as the growing search space of possible word orderings.
We also observe that the Enron Email dataset is more susceptible to attack.
Table 1 shows that the approach of Zhu et al.
(2019) fails to recover the original sentence for batch size b = 1 .
We ﬁnd that the gradient matching method in Zhu et al.
(2019) is sensitive to initialization and it can perform better recovery when using a good initialization (see Appendix B.3). Deng et al.
[Figure]
{+R O U G E1 
 [Figure] [Figure]+}
{+R O U G E -(a) WikiText1 
+}
[Figure]
{+R O U G E - L 1 
+}
[Figure]
{+N E RR 1 
+}
[Figure] [Figure] [Figure]
O U G E -
[Figure]
FILM (Ours) Zhu improves over (Zhu et al., 2019) by prioritizing the matching of gradients of layers that are closer to the input.
However, Fig.
3 shows the NERR and ROUGE scores sharply degrade for larger batch sizes. On the contrary, our approach is able to recover a large portion of the original sentences on both datasets even when the batch size is 128. This demonstrates the superior effectiveness of leveraging the knowledge encoded in pre-trained language models and its memorization ability to recover sentences, compared to directly performing optimization in a high-dimensional continuous space.
Attack performance at different training iterations. We evaluate the attack performance of our approach on the models trained at different numbers of training iterations t , using the WikiTextdataset.
Speciﬁcally, we vary the number of training iterations t ∈ { 0 , 10000 , . . .
, 90000 } and ﬁx the batch size for training b = 16 and dataset size N = 200 , 000 . The batch size for attack is 1 .
Previous gradient matching attacks on image data suggests that well-trained models are harder to attack (Geiping et al., 2020) due to the shrinkage of gradient norm as training progresses.
However, as shown in Fig.
4 (see qualitative examples in Table 11), we ﬁnd that well-trained models are more vulnerable to our approach because the model is able to memorize data during training.
Pre-trained vs randomly-initialized models.
We additionally compare the attack performance for training that starts from a pre-trained model (GPT-2 in our case) and a randomly-initialized model (we use the same GPT-2 architecture but re-initialize all the weights randomly).
We ﬁnd that starting from a pre-trained model is more susceptible to attack than training from scratch as shown in Fig.
However, the gap in performance gradually decreases as the number of iterations increases. As shown in Fig. 4, a pre-trained model which has not been ﬁne-tuned on client data can recover sentences with an average ROUGE-L score of 0.41, indicating that prior knowledge encoded in pre-trained models may help recover sentences.
When the model is continued to train on Wikitext103, the average score increases to 0.53, an almost 30% increase.
Our results suggest that leveraging the memorization in the model yields a more powerful attack than only using the language prior encoded in the pre-trained model.
We note that we would expect stronger attack performance (especially for smaller training epochs) when attacking largercapacity models, as stronger models can memorize training instances easily.
We report more results for t = 0 with different batch sizes in Table 12 in Appendix B.5.
[Figure]
Recovering multiple sentences by iterative attack.
We further investigate a variant of our attack in which we apply the beam search for multiple repetitions.
Each repetition applies an n -gram penalty based on the results of previous generations.
We consider a sentence to have strong leakage if the longest recovered subsequence is > 25% of the original sentence (i.e., ROUGE-L score > 0.25). Iter. Original Sentence [Figure]
Sentence 1st In 1988 it was made into a feature ﬁlm Katinka, di- rected by Max von Sydow, starring Tammi Øst as Katinka. After the end of the year, the ﬁlm was made into a fea- ture ﬁlm, and it was directed by Max von Sydow and published by Warner Bros.
2nd
He was killed in a mysterious plane crash on 29 March 1959, while en route to Bangui.
Fox, in an attempt to put an end to the ﬁre, was killed in a plane crash while en route to Guadalcanal, Puerto
Rico, on April 6, 2015, while on a 10th
The next year, Edwards published
[Table]
[Table]
[Table]
Table 2 shows a qualitative example of this iterative process.
Table 3 shows that for a batch size of 16, an attacker who performs 100 repetitions of beam search is able to recover roughly 34% of the original batch (roughly 5 sentences) with ROUGE-L score over 0.
(recall). However, since 100 repetitions results indifferent generated sentences, an attacker would still need a method that picks good recoveries out of them 9 .
We estimate the difﬁculty of choosing good sentences by using the ratio of good sentences to total generated sentences (precision) [Table] 6.3 Evaluation of Defenses
We evaluate three defense methods: gradient pruning (Zhu et al., 2019) and Differentially Private
Stochastic Gradient Descent (DPSGD) (Abadi et al., 2016), and our method of freezing embeddings.
The evaluations use the WikiText-103 dataset and the GPT-2 model, with batch size = 16. Given S ∗ , the set of tokens recovered from the attack and S , the original set of tokens in the private batch. Two metrics for performance of the attack are: Precision: | S ∗ ∩ S | / | S ∗ | , which is the fraction of original tokens in the recovered set.
Recall: | S ∗ ∩ S | / | S | , which is the fraction of recovered original tokens in all original tokens. We measure the impacts of defenses on model utility by computing the average perplexity of the model across the test set for each dataset. For evaluating the defense of freezing word embeddings during training, precision and recall are both 0. Token reconstruction under gradient pruning (Zhu et al., 2019).
Gradient pruning zeros out the fraction of gradient entries with low magnitude.
However, the embedding gradients of existing tokens in the batch are still non-zero unless the prune ratio p is extremely high.
Thus, the attack strategy remains the same as the vanilla attack: we retrieve the tokens whose embedding gradients are non-zero. As shown in Table 4.a, the attack always returns existing tokens in the private batch (i.e. precision is always 1). The recall decreases as the prune ratio increases, because some entries of word embedding gradients get completely zeroed out and thus the corresponding tokens cannot be retrieved by the attack. However, the attacker can still recover a considerable amount of tokens (i.e.
>90%) even with a prune ratio as high as 0.9999. Token reconstruction under DPSGD (Abadi et al., 2016). The strategy to launch the attack under DPSGD is trickier, as the gradients become noisy and the previous heuristics of returning nonzero gradient entries no longer hold. We come up with a new attack strategy which involves using a threshold τ = σ √ 2 log d to discriminate noisy gradients with pure noise, where d is the embedding dimension (i.e. 768) and σ is the noise scale of DPSGD.
For each token, we check the maximum magnitude of its embedding gradients: if the value is larger than τ , then the token may be included in the original batch with high probability.
As shown in Table 4.b, the attack performance drops when the epsilon of DPSGD decreases (at the cost of perplexity), because the relative scale between the noise magnitudes and the original gradient magnitude increases.
Token reconstruction under freezing embeddings.
Freezing embeddings stops the attacker’s access to the bag-of-words, and therefore results in precision and recall both being 0 .
However, it is nots considered a standard practice in training language models, as it may lead to worse model utility. We compare the perplexity of GPT-2 models (1) with frozen embeddings or unfrozen embeddings and (2) from scratch (i.e., from a randomly initialized model) or from pre-trained (i.e., initialized by GPT-2 parameters) in Table 5. We ﬁnd that freezing word embeddings in training from scratch results in a signiﬁcant loss of model utility, but is negligible in the setting of using a pre-trained model. From Scratch
From Pretrained
Unfrozen Frozen Unfrozen Frozen Wikitext27.  Analysis We ﬁnally discuss the impact of different parameters of our FILM attack in Fig.
5 and Fig.
We also study the effect of different training parameters and present the results in Appendix B.7.
Beam size.
As discussed in Sec. 4.3, beam size (denoted by k ) is a hyper-parameter of beam search which controls the number of beams active at any time.
As shown in Fig.
5 (left), the attack performance grows as the beam size is increased.
This result aligns with our intuition using a larger beam size corresponds with consideration of more sentences during search.
Considering that the computational resources required for the attack also increases with a larger beam size, we ﬁnd that a beam size of around 32 achieves the best trade-off in terms of attack performance and efﬁciency.
N-gram penalty.
The n -gram penalty controls for what size of n -gram the penalty is applied.
In
Fig.
5 (right), we observe that having no penalty performs signiﬁcantly worse than applying a penalty on repeated 2-grams.
We note that performance for n = 2 (i.e., penalizing repeated 2-grams) is the closest to the “oracle” case, where the frequency of each word is known from the bag of words.
Gradient norm’s coefﬁcient β in prior-based scoring.
As discussed in Sec. 4.4, the priorbased token recording for a sentence x uses a scoring function S ( x )
=
Perplexity ( x ) + β ·
Gradient Norm ( x ) , where β is a hyper-parameter that controls the importance of the gradient norm term.
Our results (see Fig.
(left)) show that the ﬁne-tuning stage achieves a trade-off between fully perplexity-guided (i.e., β = 0 ) and almost fully gradient norm-guided (i.e., β = 50 ), with β
=being the optimal.
The trend is consistent across different batch sizes.
[Figure] [Figure] We also notice that the improvement in the token reordering stage is most signiﬁcant for small batch sizes (see Fig.
6 (right)).
We hypothesize this is due to locality; the beam search result for smaller batch sizes is usually closer to its corresponding ground truth sentence than for larger batch sizes, which makes ﬁne-grained reordering more powerful.
Discussion and Future Work
This paper presents the ﬁrst attack that can recover multiple sentences from gradients in the federated training of the language modeling task.
Our method FILM works well with modern neural language models based on Transformers and large batch sizes (up to 128), raising serious privacy-leakage concerns for federated learning in real-world scenarios.
The key insight of our study is that an adversary can gain substantial power to reconstruct sentences by taking advantage of the knowledge memorized by trained language models. We also propose a new defense which freezes the word embeddings during the federated training of language models.
Our evaluation shows that it can effectively defend the FILM attack for the setting of starting from a pretrained languages model, with little utility loss.
Limitations.
The results presented in this paper have several limitations.
First, we have not evaluated whether our attack can generalize to other natural language processing tasks, such as text classiﬁcation.
Second, we have not tried our attack for batch sizes beyond 128 and our ablation study does not report results for batch sizes beyond 16.
We additionally leave open the question of selecting good sentences in the multi-sentence recovery scenario (Sec. 6.2). Furthermore, the proposed defense of freezing embeddings during the pre-training of the language model can be detrimental to the model utility. Therefore, how to effectively defend the FILM attack for this setting remains open.
Ethical considerations.
While our intention is to raise the awareness of the privacy risks in the language modeling task in federated learning, from the attacker’s prospective, FILM is also likely a step towards even stronger attacks.
Therefore, we hope that this work can also motivate a necessary redesign of defenses to provide meaningful privacy guarantees to clients for training language models in federated learning. A Beam Search Algorithm Algorithm 1 demonstrates the step-by-step operations of our beam search algorithm (see Sec. 4.3).
Input: Bag of words B = { w 1 , . . . , w m } , A maximum sentence length
L , beam size k ,
A set of z possible prompts P = { p 1 , . .
, p z } , Distribution deﬁned by the trained language model IP θ t , n -gram count function R n with associated penalty ρ , Output: g ∗ k sentences consisting of words in B 1 π 1 ← P for i ∈
[ L ] do π i +1 ← ∅ foreach ( beam, word ) in π i ×
{+B do π i +
+}
← π i +1 ∪ { CONCAT ( beam, word ) } end
π i +1 ← T OP (
π i +1 , k, f θ t ) 7 end 8 return π L ; 1 Procedure Top( π , k , IP θ t ) :
SCORES ← ∅
SCORES ← SCORES ∪ log IP θ t ( beam ) − ρR n ( beam ) return Top k elements of π ordered by S CORES
B Experimental Details and More Results B.1 Preprocessing details for WikiText-103 and Enron Email
We consider recovering sentences in the current work.
As WikiText-103 and Enron Email have longer context (i.e. paragraph), in our evaluation, we split them into sentences and only keep the sentences with 1) fewer than 40 tokens, 2) fewer than 5 unknown tokens, where unknown tokens are tokens not in the vocabulary of the GPT-2 model, 3) a perplexity smaller than 50 on the pretrained GPT-2 model, 4) at least 1 named entities 10 of type: "PERSON", "ORG", "GPE", "LOC", "PRODUCT", "EVENT".
We leave recovering longer paragraphs as future work.
We keep 2000 examples of each dataset as the evaluation set, and use the left for training.
B.2 Comparison with Previous Attacks
Our FILM attack is unique as it does not rely on end-to-end optimization, is demonstrated on large batch sizes, and is focused on the autoregressive language modelling task
[Table]
[Table]
[Table]
B.3 More Results of DLG (Zhu et al. 2019) and TAG (Deng et al. 2021)
Table 7 shows the attack results of Zhu et al.
with different batch sizes.
Their gradientmatching optimization fails to recover sentences with different batch sizes.
We also ﬁnd that Zhu et al.
(2019) seem sensitive to the initialization.
Table 8 shows the attacking results with different initialization.
We ﬁnd that only when the initialization is very close to the original sentence (i
[Table]
e., oneor two-word different), the attack can recover sentences successfully.
Table 9 compares metrics for reconstructions by previous methods and by ours
[Table]
[Table]
[Table]
B.4 Quantitative Results for Different Batch Sizes
Table 10 shows all four metrics for the attack results with different batch sizes; Fig.
7 shows the performance of the best reconstructions for each metric and batchsize.
[Table]
[Figure]
B.5 More Results For Attacking the Pre-trained Model
Table 11 presents attack results at different training iterations of a pretrained model.
Similar to our ﬁnidngs in Sec. 6, FILM recovers better sentences when t increases.
Table 12 provides more examples for running our attack on a pre-trained language model (i
[Figure]
e. without any training with private data) with different batch sizes.
The attack is relatively weak in this scenario, showing that leveraging the memorization in the model give a much more powerful attack than only using the language prior encoded in the pre-trained model
[Figure]
[Table]
[Table]
B.6 Qualitative Results with Randomly Initialized Models
Table 13 presents attack results at different training iterations of a randomly initialized model.
The overall attack quality is lower than the results with a pre-trained model.
[Table]
the results with a pre-trained model (see Table 11)
[Figure]
B.7 Ablation Study for Training Conﬁgurations
We also perform an ablation study on WikiText-103 to investigate how the attack scales with different training parameters, including the size of the training dataset and the initial learning rate.
Larger training set sizes are not harder to attack.
It was believed that training with large training sets helps avoid over-ﬁtting and generalizes better as it captures the inherent data distribution more effectively, while training with small datasets may result in the model memorizing the training set (Arpit et al., 2017)—which in our case, may yield a better attack performance.
However, our experiments suggest that larger training set sizes are not any more difﬁcult to attack than smaller training sets.
As shown in Fig.
8.a, the attack performance is quite similar across different evaluated training set sizes, from 5 ,examples to 200 , 000 .
[Figure]
Larger initial learning rate is more vulnerable to attack.
We also show how the attack performs with with different initial learning rates by varying the initial learning in { 1 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 , 5 × 10 − 4 } while ﬁxing the batch size b = 16 and dataset size N = 200 , 000 .
As we apply early stopping during training, none of the models we test were overﬁt.
As shown in Fig.
8.b, as the initial learning rate increases, our attack is able to achieve slightly higher ROUGE scores.
Table 14 presents the attack results recovered under different initial learning rates.
We ﬁnd that as the initial learning rate increases, our attack is able to successfully reconstruct more components of the training sentence despite similar ﬁnal training and testing perplexity
[Table]
B.8 Justiﬁcation for the Scoring Function in the Reordering Stage
As discussed in Sec. 4.
[Table]
our scoring function consists of a gradient norm term and a perplexity term (see Eq.4).
This design is mainly motivated by the observation that sentences in the training corpus usually have lower perplexity scores and lower gradient norms on the trained language model, compared with their slightly altered versions.
To demonstrate this, we randomly pick the 5 following sentences from the training corpus:
• “Signiﬁcant increases in sales worldwide were reported by Billboard in the month of his death." ; • “The ofﬁce has subsequently been held by one of the knights, though not necessarily the most senior." ; • “In 1999, several Russian sources reported that Laika had died when the cabin overheated on the fourth orbit." ; • “Ofﬁcials advised 95@,@000 citizens along the New Jersey coastline, an area that rarely experiences hurricanes, to evacuate." ; • “It later won a 2014 Apple Design Award and was named Apple’s best iPhone game of 2014." ;
For each original sentence, we generate its slightly altered versions which performs one of the following three operations (we generate 10 sentences for each operation):
• Swap two randomly selected tokens in the original sentence; • Delete a randomly selected token from the original sentence; • Insert a token (randomly selected from the bag of tokens of the original sentence) into a random position of the original sentence.
As shown in Fig.
9, original sentences usually have lower gradient norm and perplexity score than their corresponding slightly altered versions.
B.9 How much does the bag of words help?
We brieﬂy compare the performance of the attack in a scenario where beam search is performed with the bag of words (as in FILM), and without the bag of words (i.e. if an attacker only had the model, but not bag of words).
Both settings were ran with the same prompts for all samples.
[Figure]
[Figure]