Recovering Private Text in Federated Learning of Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Federated learning allows distributed users to collaboratively train a model whilekeeping each user’s data private.
Recently, a growing body of work has demonstrated that an eavesdropping attacker can effectively recover image data fromgradients transmitted during federated learning.
However, little progress has beenmade in recovering text data.
In this paper, we present a novel attack method
FILM 1 for federated learning of language models —for the ﬁrst time, we show thefeasibility of recovering text from large batch sizes of up to 128 sentences.
Different from image-recovery methods which are optimized to match gradients, wetake a distinct approach that ﬁrst identiﬁes a set of words from gradients and thendirectly reconstructs sentences based on beam search and a prior-based reorderingstrategy.
The key insight of our attack is to leverage either prior knowledge in pretrained language models or memorization during training.
Despite its simplicity,we demonstrate that FILM can work well with several large-scale datasets—it canextract single sentences with high ﬁdelity even for large batch sizes and recovermultiple sentences from the batch successfully if the attack is applied iteratively.
We hope our results can motivate future work in developing stronger attacks aswell as new defense methods for training language models in federated learning.
Introduction
Federated learning [37] is a method to allow multiple participants to train a global model without exchanging their private data collaboratively.
At each step of training, a central server transmits model parameters to every participating client.
Each client then computes a model update (i e., gradients) using its local data and sends it to the server.
Finally, the server aggregates all updates—typically by averaging them—and updates the model.
As each client does not need to share its private data with the server during training, federated learning is actively being considered for privacy-sensitive applications such as virtual mobile keyboards and analysis of electronic health records in hospitals [31]
[Figure]
However, recent studies [58, 16, 52, 57, 25, 42, 17] show that an honest-but-curious eavesdroppingattacker in federated learning—who observes the communication of model parameters and gradientsbetween a client and the server—can recover private data of the client (Fig. 1).
By reformulating theattack as an optimization problem, an attacker can iteratively build a copy of a client’s private images.
However, the success in recovering text data remains limited—for instance, [58, 11] showed leakageof text data using similar optimization-based methods for only a batch size of 1 2 .
In this work, we study federated learning of neural language models (LMs) and present a novel attackcalled FILM ( F ederated I nversion Attack for L anguage M odels), which can recover private text datafrom information transmitted during training.
For the ﬁrst time, we demonstrate that an attackercan successfully recover sentences from large training batches of up to 128 sentences—makingit practical and alarming in real-world scenarios.
3 We focus on language models for two reasons:(1)
In the past few years, Transformer-based language models [50] have achieved unprecedentedsuccess [12, 43, 44, 7] and have been quickly adapted to a number of privacy-sensitive domains [2,22, 33, 35, 30].
They are not only able to generate coherent and ﬂuent text but serve as basic buildingblocks for language understanding tasks; (2) We identify that language-model training is at a higherrisk of information leakage by nature because the attacker can possibly leverage the memorizationcapability of LMs during federated learning.
We ﬁnd that previous gradient-matching approaches, which directly optimize in high-dimensional,continuous space, don’t work well in recovering data for LMs as they are highly sensitive to initializations and language inputs are essentially discrete (Sec. 5).
Especially with large batch sizes,the averaged gradient information (of  128 sentences) does not sufﬁce to recover text information.
Instead, our FILM attack takes a radically different approach: We ﬁrst recover a set of discrete words 4from the embeddings’ gradients based on the insights from Melis et al.
This step can give us aset of words that appear in any sentences of the training batch without the frequency information.
Wethen develop a simple and effective strategy using beam search [45, 46], which tries to reconstruct onesentence from the set of identiﬁed words.
This step is crucial as it takes advantage of prior knowledgeencoded in pre-trained language models and memorization of training data during federated training.
We also further design a token-reordering approach that leverages both language prior and gradientsinformation as the signal to ﬁne-tune the recovered sentence.
Finally, we show that we may recovermultiple sentences from the same training batch by applying the same procedure iteratively.
We evaluate FILM on the WikiText-103 dataset [39] and the Enron Email dataset [29], based on a
GPT-2 model [44], with either pre-trained or randomly-initialized weights.
We analyze the attackperformance with different batch sizes, number of training data points, and number of training epochs.
Our experiments demonstrate high-ﬁdelity recovery of a single sentence feasible, and recovery ofsigniﬁcant portions of sentences for training batches of up to 128 sentences.
We also show thepossibility to recover multiple sentences from the same batch ( 1 / 3 for a batch size of 16), by applying
FILM iteratively.
Together, we believe that our results raise serious privacy-leakage for federatedlearning in realistic scenarios and they should motivate a necessary redesign of defenses to providemeaningful privacy guarantees to clients for training language models in federated learning.
Related work
Data recovery from gradients.
Gradients in federated learning have been shown to leak a severeamount of information of the underlying private data.
Several attacks [58, 56, 16, 14, 52, 57, 27, 26 ?, 42, 17] successfully reconstruct individual users’ private images from gradients.
These methodsusually formulate such a reconstruction as an optimization problem, which encourages reconstructedimages to yield similar gradients as observed gradients computed on the private images, usingregularization terms based on image priors such as total variation [16] and batch normalizationstatistics [52].
Existing research has primarily focused on recovering images, and the earliest attempt to recovertext from gradients is Zhu et al.
[58], which brieﬂy presented leakage results in masked languagemodeling [12].
Their attack aims to match target gradients with continuous representations and mapsthem back to words which are closest in the embedding matrix.
Deng et al.
[11] improve [58] byadding a regularization term which prioritizes the gradient matching in layers that are closer to theinput data.
However, both works only present results with batch size of 1 and do not work well withlarger batch sizes, as we will show in Sec. 5.
Another earlier work [38] shows the information leakagefrom the embedding layer by observing which entries of words are non-zero in the gradients—a keyinsight that we also build on in this work.
However, different from their attack which only infers a person or location from a shallow GRU model [10], we are proposing a much stronger attack thataims at recovering full sentences from larger language models.
Finally, very recent works [5, 15]tries to recover text data in federated learning under a stronger threat model: the server is maliciousand can manipulate the training model’s weights to enable easier text reconstruction.
The closest concurrent work is Dimitrov et al.
[13], where the attacker tries to reconstruct privatesentences via discrete text transformation that minimizes both the gradient matching loss, and theprior text probability computed by an auxiliary language model.
They are able to recover a mini-batchof 4 sentences with binary classiﬁcation tasks using the BERT [12] model.
In this work, we focus onlanguage models and we show that our attack is able to recover sentences from a larger training batchof up to 128 sentences.
Memorization in language models.
Our method is also inspired by the work related to the memorization of training data in language models.
Recent studies [8, 49, 48, 54, 9, 4] notice that largelanguage models are able to memorize their training dataset and hence can be prompted to outputspeciﬁc sensitive information.
For example, Carlini et al.
[8] propose a way to quantitatively measurethe risk that a training example is unintentionally memorized by a language model, and Thakkar et al.[49] extend this measurement to federated learning.
Zanella-Béguelin et al.
[54] demonstrate that anattacker who has access to the pre-trained and ﬁne-tuned LSTM [19] is able to extract rare or uniquesequences from the ﬁne-tuning dataset.
More recently, Carlini et al.
[9] present an attack to extractpersonally identiﬁable information (e.g., name, email address, phone number, and physical address)from the publicly available GPT-2 model [44].
They also ﬁnd that larger language models are moresusceptible to this style of attack than smaller ones.
Note that while these previously demonstrated memorization attacks are trying to recover a subsetof sensitive text from a large training corpora, our attack aims at recovering sentences from everymini-batch in federated learning.
We stress that our attack is signiﬁcantly more powerful than theprevious works studying memorization.
We discuss further related work about defenses against data recovery from gradients in Appendix A.
Preliminaries
In this section, we describe relevant background of language models, federated learning, as well asthe threat model of our proposed attack method.
3.1 Language Modeling
Language modeling is a core task in natural language processing and has been used as a buildingblock of state-of-the-art pipelines [44, 12].
Given a sequence of n tokens x = { x 1 , x 2 , ..., x n } , thelanguage modeling task is to estimate the probability distribution of IP ( x ) , which is factorized as:
[Equation]
where ✓ represent the neural network parameters.
State-of-the-art neural language models areparameterized by RNNs [40] or Transformers [50], which consist of millions or billions of parameters.
A language model ﬁrst maps discrete words to a sequence of vectors via a word embedding matrix
W 2 R |V| ⇥ d , where V is the vocabulary and d is the hidden dimension.
After computing the hidden 112 representation h i conditioned on x 1 , . . .
, x i  1 , the model predicts the probability of the next token:
[Equation]
where W x i and W j represent the word embeddings corresponding to x i and j .
3.2 Federated Learning
Federated learning (FL) [37] is a communication protocol for training a shared machine learningmodel on decentralized data.
An FL system usually involves N clients: c 1 , c 2 , · · · , c N and a central 117 server s , where the clients wish to collaboratively train a neural network f ✓ (a language model inour case) with their private data D 1 , D 2 , · · · , D N , under the coordination of the server.
The training 119 process optimizes ✓ (model parameters) using a loss function
L and runs for T iterations.
Each iteration t 2 [ T ] of federated learning consists of the following steps.
First, an individual clientc i , i 2 [ N ] computes r ✓ t L ✓ t ( B i ) , the gradients of the current model parameters ✓ t on a randomly 122 sampled private data batch
B i ⇢ D i , |B
i | = b , and shares the gradients with the server.
The server 123 then aggregates gradient updates from all clients and updates the model:
① Let ② The
Let the
The mouse
Let dogs …
Let the dogs
The mouse chases
Let dogs lie
Let the dogs lie.
The mouse chases the cat.
Let dogs lie sleeping.
“Let”, “The”, “cat”, “chases”, “dogs”,“lie”, “Mouse”, “sleeping”, “the”, “.” … … … … …
[Figure]
[Figure]
(3) where ⌘ is the learning rate.
Finally, the updated parameters ✓ t +1 are broadcast to individual clients.
Note that Eq.
3 uses SGD for an easier demonstration for an update step in federated learning.
Inreal-world training of neural language models, people usually use Adam [28].
In this work, we focus on federated learning of language models, where clients can start from arandomly-initialized neural network, or a pre-trained public model [18]—we consider a GPT-2model [44] in our experiments.
3.3 Threat Model
Adversary’s capabilities.
We consider an honest-but-curious adversary who is eavesdropping onthe communication between the central server s and
an arbitrary client c
i , i 2 [ N ] in the federatedtraining of a language model parameterized by ✓ , as described above.
This allows the adversary awhite-box access to 1) the gradients r ✓ t L ✓ t ( B i ) sent by the client c i , and 2) the model parameters 135 ✓ t (Fig. 1), including the entire vocabulary and the embedding matrix.
Note that the adversary caninspect these information at any training iterations t 2 [ T ] .
Adversary’s objective.
The adversary aims at recovering at least one sentence from the privatetraining data batch B i based on the information they observe, as recovering a single sentence isalready sufﬁcient to break the privacy guarantee of federated learning—since each training batchconsists of randomly selected sentences from the private training dataset, if the adversary couldrecover at least one sentence from each training batch during the whole training procedure, he/shewould be able to recover a considerable fraction of the original training dataset.
Additionally, theattacker may also apply the attack on a single batch multiple times to recover more sentences from thebatch (Sec. 4.5).
The strength of an attack will be measured by the similarity between the recoveredsentence and its corresponding private sentence in the original batch.
Our Attack Method
4.1 Overview
In this section, we present a novel text-recovery attack method, which is able to construct sentencesfrom any training batches ( |B i |  128 ).
Our method consists of three steps:(1)
Inspired by Melis et al.
[38], our ﬁrst step is to recover a set of words that may appear in a privatebatch ( B i ) from gradients (Sec. 4.2).
While their goal is to identify sensitive single words fromgradients, our approach is able to recover the order of words in a sentence.(2) With the set of words identiﬁed, we then attempt searching for full sentences using beam search,based on the probability distribution provided by f ✓ ( x ) .
Our method is able to leverage either thelanguage prior from a pre-trained language model ( t = 0 ), or the memorization capability of thelanguage model during training ( t > 0 )
(Sec. 4.3).
Finally, we design a prior-based scoring function and demonstrate that we can further improve thequality of recovered sentences by performing a simple and lightweight reordering step (Sec. 4.4).
We show that the above pipeline is able to recover single sentences from large training batchessuccessfully.
In Sec. 4.5, we discuss that we can iterate the above procedure to recover multiplesentences from the same training batch, which makes our attack even stronger.
In the following, we describe our attack method in detail.
To better illustrate our attack’s functionality,we provide a simpliﬁed step-by-step example that recovers a batch of 2 simple sentences in Fig.2. 4.2 Bag- of -Words Extraction
Input: r ✓ t L ✓ t ( B i ) , gradients of the language model on the private batch of sentences.
Output: bag of words B : {“Let", “The", “cat", “chases", “dogs", “lie", “mouse", “sleeping", “the", “."} .
Our ﬁrst step is to extract a bag of words from r ✓ t L ✓ t ( B i ) , the gradients observed by an eavesdropperat the t -th iteration of training.
We follow the method outlined by Melis et al.
[38]—more speciﬁcally,we recover a set of words B = { w 1 , w 2 , . . .
, w m } by considering the non-zero rows of the embeddinggradients r W t L ✓ t ( B i ) ( W t is the word embedding matrix at training step t ).
Additionally, we are 172 also able to recover the maximum length of sentences in the batch by analyzing the non-zero rows ofthe gradients corresponding to the positional embedding matrix.
Note that this step only determines the set of words but not their frequency .
Although the frequencyinformation can be inferred from the norm of gradients [15, 51] due to the fact that more frequentwords tend to have seen more SGD updates and thus larger norms 7 , as we will see in Section 4.3, ourattack can achieve similar performance as the case when oracle frequency is given.
4.3 Beam Search for Sentence Reconstruction
Input: B , the bag of private words recovered by Sec.
Output: a recovered sentence from the batch, “The mouse chases the cat." .
The second stage of our attack is to recover one sentence based on the extracted bag of words B and
IP ✓ t ( x ) (note that this probability distribution is deﬁned based on the parameters at the time of attackt ).
Since it is intractable to generate text directly using IP ✓ t ( x ) , we use the beam search [45, 46],which is a greedy auto-regressive algorithm to generate sequences in a step-by-step manner with atrained language model.
We provide a detailed algorithm in Appendix B.
To initiate beam search, we ﬁrst ﬁnd a set of words that can be chosen as starting tokens of sentences.
A straightforward method is to select words which begin with a capital letter from B .
For example,we would select “Let" and “My" as possible prompts for the illustrated batch.
Then at each beamsearch step, we keep the top k possible beam states, ﬁnds the k most likely next words to each beamby using the language model (see procedure T OP ( · ) in Appendix B, Algorithm 1), and keeps the topk new beam states.
The beam tree continues until the generated sentence reaches a speciﬁed length(we use 40 words), and outputs the text that has the overall highest probability (see Fig.2).
As the bag of words B does not contain information on the frequency of each word in the sentence,and beam search is prone to generating repetitive content [20], we also include a penalty for repeatedn -grams in a sentence (see line 3 in procedure T OP ( · )).
We compare the performance of the attackunder different n -gram penalties relative to a hypothetical optimal scenario where the frequencyof each word is known by the attacker (implemented by sampling without replacement from thebag-of-words with word frequency information) in Sec. 6.
4.4 Prior-Guided Token Reordering
[Table]
“The cat." [Table] “The cat
By construction, beam search is a greedy method which generates sentences from left to right.
Despitethat it generally outputs ﬂuent sentences by maximizing IP ✓ t ( x ) , it does not leverage the gradientsinformation which is the main signal in previous image attack methods.
Thus, the ﬁnal stage ofour attack is to reorder tokens in the extracted sentence from the previous step, with an objective toimprove its quality measured by a prior-based scoring function.
The scoring function leverages boththe perplexity and the gradient norm.
Formally, given a sentence x = { x 1 , x 2 , ..., x n } of n tokens,and the trained language model parameterized by ✓ , x ’s prior score S ( x ) is deﬁned as:
[Equation]
where log IP ✓ ( x ) is the log-likelihood of the sentence (Eq. 1) according to the model parameters ✓ , 210 and  is a hyper-parameter that controls the importance of gradient norm.
Appendix C.6 providesempirical analysis that supports the design of this scoring function.
We also notice that sometimes the beam search preserves the structure of the sentence while theorder of original phrases or words is wrong.
Thus, we propose to reorder the sentences with twoconsecutive steps, from coarse to ﬁne granularity.
Phrase-wise reordering.
We ﬁrst pre-process the sentence x by removing its redundant segments 8 .
We then adjust the preprocessed sentence by reordering phrases inside it.
At each reordering iteration,we generate multiple candidates by ﬁrstly cutting the sentence at p positions using the strategy in
Malkin et al.
[36], and then permuting the phrases between the cuts to form a new sentence.
We thenselect the best candidate measured by S ( · ) .
This stage terminates after 200 steps.
Token-wise reordering.
We then reorder individual tokens.
At each reordering step, we generatecandidates using the following token-wise operations: 1) randomly swaps two tokens in the sentence,or 2) randomly deletes a token from the sentence, or 3) randomly inserts a token from the bag ofwords (recovered by Sec. 4.2) into the sentence.
We then select the best candidate measured by S ( · )and move to the next iteration.
This stage terminates after 200 steps.
4.5 Recovering Multiple Sentences
The previous steps only consider the recovery of a single sentence from a training batch.
In thisstep, we describe an extension of our attack which is able to support recovery of multiple sentencesfrom the same batch.
We ﬁrst perform beam search as described in Sec. 4.3.
We store the results,and repeat beam search again—except this time applying an additional n -gram penalty based on theresults of the previous search.
Over many repetitions of this procedure, we are able to build a set ofcandidate recoveries for the original batch.
Experiments
5.1 Setup
Model and datasets.
We evaluate the proposed attack with the GPT-2 base model [44] (around117M parameters) on two language modeling datasets, including WikiText-103 [39] (license: CC
BY-SA 3.0) and the Enron Email dataset [29] (license: unknown).
Both datasets are publicly availablefor research uses.
We choose WikiText-103 because it is commonly used in the language modelingtask, and the Enron Email dataset because it consists of private email messages that contain abundantprivate information such as individuals’ names, addresses, and even passwords.
Note that the GPT-2model is not trained on any Wikipedia data, and very unlikely on the Enron Email data 9 .
We evaluate our attack on a subset of the WikiText-103 and Enron datasets.
After pre-processing,we have 203,456 sentences from the WikiText-103 dataset and 31,797 sentences from the Enrondataset 10 .
Training and attack settings.
Unless otherwise noted, we train the model on these sentences for90 , 000 iterations using an initial learning rate of 1 ⇥ 10  5 , with a linearly decayed learning ratescheduler.
All models were trained using early stopping—i.e., models were trained until the loss ofthe model on the evaluation set increased.
We note that the running time of our algorithm is also quitefast.
Our approach can recover a single sentence in under a minute using an Nvidia 2080TI GPU.
Evaluation metrics.
We propose to use the following metrics to evaluate the attack performance:
(a) Recall-Oriented Understudy for Gisting Evaluation (ROUGE)
[34] is a set of metrics for evaluatingautomatic summarization of texts as well as machine translation.
We use ROUGE-1, ROUGE-2,and ROUGE-L F-Scores to evaluate the similarity between the recovered and the original sentences,following [11, 13].
More speciﬁcally, ROUGE-1 and ROUGE-2 refer to the overlap of unigrams andbigrams between the recovered and the original text, respectively.
ROUGE-L measures the longestmatching subsequence of tokens.
For ablation studies we only show ROUGE-L as a high score ismore representative of signiﬁcant leakage than ROUGE-1 or ROUGE-2.(b)
Named entity recovery ratio (NERR) computes the percentage of original name entities that canbe perfectly retrieved in the
recovered text.
Since named entities usually contain sensitive information(e.g., names, addresses, dates, or events), NERR measures how well the attacker recovers suchinformation, ranging from a complete mismatch (NERR = 0) to a perfect recovery (NERR = 1).
Weidentify named entities using the API provided by SpaCy [21].
[Table]
5.2 Performance of Our Attack
Scaling to large training batches.
We ﬁrst compare the performance with respect to differentbatch sizes.
In this setting, we start from a pre-trained GPT-2 model and ﬁne-tune it on both datasetsrespectively (WikiText-103 and Enron emails).
All models were ﬁnetuned for 90,000 iterations, atwhich the model loss over the evaluation set starts to increase.
Fig.
3 shows the performance of ourattack using different batch sizes with the WikiText-103 dataset and the Enron Email dataset.
Weobserve that for a batch size of 1, all four scores (ROUGE-1, ROUGE-2, ROUGE-L, NERR) areclose to 1, indicating a near-perfect recovery.
We observe that as the batch size increases, the qualityof the recovery by ROUGE-1, ROUGE-2, and NERR start to decrease.
We attribute this drop inperformance to the bigger set of words in the same training batch, as well as the growing search spaceof possible word orderings.
We also observe that the Enron dataset is more susceptible to attack.
Table 1 shows the approach by Zhu et al.
[58] fails to recover the original sentence for batch sizeb = 1 .
We ﬁnd that the gradient matching method in Zhu et al.
[58] is sensitive to initialization and itcan perform better recovery when using a good initialization (see Appendix C.1).
Fig.
3 shows theirapproach has poor NERR and ROUGE scores.
On the contrary, our approach is able to recover a largeportion of the original sentences on both datasets even when the batch size is 128.
This demonstratesthe superior effectiveness of leveraging the knowledge encoded in pre-trained language models and [Figure] [Figure] [Figure]
[Figure]
[Figure]
its memorization ability to recover sentences, compared to directly performing optimization in ahigh-dimensional continuous space.
Attack performance at different training iterations.
Previous attack on the image data suggestthat a well-trained model is harder to attack [16], because its loss may converge to around zero, resulting in very small-magnitude gradients.
However, we ﬁnd well-trained models are more vulnerable toour approach because the model is able to memorize data during training.
We show the attack performance of our approach to the models at different numbers of trainingiterations t .
Speciﬁcally, we vary the number of training iterations t 2 { 0 , 10000 , . . .
, 90000 } and ﬁxthe batch size b = 16 and dataset size N = 200 , 000 .
As shown in Fig.
4 and Table 2, a pre-trainedmodel which has not been ﬁne-tuned on client data can recover sentences with an average ROUGE-Lscore of 0.41, indicating that prior knowledge encoded in pre-trained models may help recoversentences.
When the model is ﬁne-tuned on Wikitext-103, the average score increases to 0.53 - analmost 30% increase.
Our results suggest that leveraging the memorization in the model yields a morepowerful attack than only using the language prior encoded in the pre-trained model.
We note thatwe would expect stronger attack performance (especially for smaller training epochs) when attackinglarger-capacity models, as stronger models can memorize training instances easily.
We report moreresults for t = 0 with different batch sizes in Table 9 in Appendix C.3.
Pre-trained vs randomly-initialized models.
We additionally compare the attack performancefor training that starts from a pre-trained model (GPT-2 in our case) and a randomly-initialized model(we use the same GPT-2 architecture but re-initialize all the weights randomly).
As shown in Fig.
4,we ﬁnd that starting from a pre-trained model is more susceptible to attack than training from scratch.
However, the gap in performance gradually decreases as the number of iterations increases.
Recovering multiple sentences by iterative attack.
We further investigate a variant of our attackwhere we apply the beam search for multiple repetitions;
For each repetition applying an n -grampenalty based on the results of previous generations.
We consider a sentence to have strong leakage ifthe longest recovered subsequence is > 25% of the original sentence (i.e., ROUGE-L score > 0.25).
Table 3 shows that for a batch size of 16, an attacker who performs 100 repetitions of beam search is able to recover roughly 34% of the original batch (roughly 5 sentences) with ROUGE-L score over 0.25 (recall).
However, since 100 repetitions results in 100 different generated sentences, an attacker would still need a method that picks good recoveries out of them 11 .
We estimate the difﬁculty of choosing good sentences by using the ratio of good sentences to total generated sentences (precision).
Table 4 shows a qualitative example of this iterative process.
[Table] [Table]
Analysis
[Figure]
We ﬁnally discuss the impact of different parameters of our FILM attack in Fig.
5 and Fig.
We alsostudy the effect of different training parameters and present the results in Appendix C.5.
Beam size.
As discussed in Sec. 4.3, beam size (denoted by k ) is a hyper-parameter of beamsearch which controls the number of beams active at any time.
As shown in Fig.
5 (left), the attackperformance grows as the beam size is increased.
This result aligns with our intuition using a largerbeam size corresponds with consideration of more sentences during search.
Considering that thecomputational resources required for the attack also increases with a larger beam size, we ﬁnd that abeam size of around 32 achieves the best trade-off in terms of attack performance and efﬁciency.
N-gram penalty.
The n -gram penalty controls for what size of n -gram the penalty is applied.
In
Fig.
5 (right), we observe that having no penalty performs signiﬁcantly worse than applying a penaltyon repeated 2-grams.
We note that performance for n = 2 (i.e., penalizing repeated 2-grams) is theclosest to the “oracle” case, where the frequency of each word is known from the bag of words.
Gradient norm’s coefﬁcient  in prior-based scoring.
As discussed in Sec. 4.4, the priorbased token recording for a sentence x uses a scoring function S ( x )
=
Perplexity ( x ) +  ·
Gradient Norm ( x ) , where  is a hyper-parameter that controls the importance of the gradient norm
[Figure]
term.
Our results (see Fig.
(left)) show that the ﬁne-tuning stage achieves a trade-off between fullyperplexity-guided (i.e.,  = 0 ) and almost fully gradient norm-guided (i.e.,  = 50 ), with
= 1being the optimal.
The trend is consistent across different batch sizes.
We also notice that the improvement in the token reordering stage is most signiﬁcant for small batchsizes (see Fig.
6 (right)).
This is probably because that the beam search result for smaller batchsizes is usually closer to its corresponding ground truth sentence than for larger batch sizes, whichincreases the chance of being further improved by ﬁne-tuning.
Discussion and Future Work
This paper presents the ﬁrst attack that can recover multiple sentences from gradients in the federatedtraining of the language modeling task.
Our method FILM works well with modern neural languagemodels based on Transformers and large batch sizes (up to 128)—raising serious privacy-leakageconcerns for federated learning in real-world scenarios.
The key insight of our study is that anadversary can gain substantial power to reconstruct sentences by taking advantage of the knowledgememorized by trained language models.
Limitations.
The results presented in this paper have several limitations.
First, we haven’t evaluatedwhether our attack can generalize to other natural language processing tasks, such as sentenceclassiﬁcation.
Second, we have not tried our attack for batch sizes beyond 128 and our ablation studydoes not report results for batch sizes beyond 16.
Third, we have not evaluated how existing defenseswork in terms of preventing our attack.
We additionally leave open the question of selecting goodsentences in the multi-sentence recovery scenario (Sec. 5.2).
Ethical considerations.
While our intention is to raise the awareness of the privacy risks in thelanguage modeling task in federated learning, from the attacker’s prospective, FILM is also likely astep towards even stronger attacks.
Therefore, we hope that this work can also motivate a necessaryredesign of defenses to provide meaningful privacy guarantees to clients for training language modelsin federated learning.