Original: SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Revised: SeqPATE: Differentially Private Text Generation via Knowledge Distillation

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Protecting the privacy of user data is crucial for text generation models, whichmay leak sensitive information during generation.
Revised: Protecting the privacy of user data is crucial for text generation models, which can leak sensitive information during generation.

Original: Differentially private (DP)learning methods provide guarantees against identifying the existence of a trainingsample from model outputs.
Revised: Differentially private (DP) learning methods provide guarantees against identifying the existence of a training sample from model outputs.

Original: PATE is a recent DP learning algorithm that achieveshigh utility with strong privacy protections on training samples.
Revised: PATE is a recent DP learning algorithm that achieves high utility with strong privacy protection on training samples.

Original: However, textgenerations conduct sequential generations on a large output space, and PATE  is notcustomized for that paradigm.
Revised: However, text generation models output tokens sequentially in a large output space; the classic PATE algorithm is not customized for this setting.

Original: Besides, PATE does well in protecting sample-levelprivacy but requires a high privacy cost on protecting phrases in samples.
Revised: Furthermore, PATE works well to protect sample-level privacy, but is not designed to protect phrases in samples.

Original: In thispaper, we propose SeqPATE, an extension of PATE on text generation, which aimsto protect the privacy of both training samples and sensitive phrases in samples.
Revised: In this paper, we propose SeqPATE, an extension of PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data.

Original: To adapt  to text generations, we generate pseudo inputs and reduce the sequencegeneration problem to next word predictions.
Revised: To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem to a next-word prediction problem.

Original: To handle the large output space, wepropose a candidate filtering strategy to dynamically reduce the  space, and refinethe teacher aggregation of PATE to avoid low agreement due to voting for a largenumber of candidates.
Revised: To handle the large output space, we propose a candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates.

Original: To  reduce privacy losses, we design an efficient knowledgedistillation to decrease the frequency of querying teachers.
Revised: To further reduce privacy losses, we use knowledge distillation to reduce the number of teacher queries.

Original: The experiments verifythe effectiveness of SeqPATE in protecting both  samples and sensitive phrases.
Revised: The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.

Original:  Introduction
Revised:  Introduction

Original: Recent work showed that sensitive user information inthe training corpora, such as addresses andnames, can be extracted from text generation models [5].
Revised: Recent work has shown that sensitive user information in training corpora, such as addresses and names, can be extracted from text generation models [6].

Original: Providing privacy guarantees to thetraining corpora of text generation models becomes a critical problem.
Revised: Providing privacy guarantees to the training corpora of text generation models has become a critical problem.

Original: Differential privacy (DP)provides provable guarantees against detecting individuals in datasets.
Revised: Differential privacy (DP) provides provable guarantees against detecting individuals in datasets.

Original: Deep learning models with
Revised: Deep learning models with

Original: DP guarantees ensure that the existence of a specific training sample cannot be detected.
Revised: DP guarantees ensure that the existence of a specific training sample cannot be detected.

Original: NoisySGD [39, 3, 1] is a popular DP algorithm for deep learning, which adds noise to the gradients.
Revised: NoisySGD [42, 3, 1] is a popular DP algorithm for deep learning that adds noise to the gradients.

Original: PATE [29] is another type of DP learning algorithm that transfers knowledge from teachers trained onprivate data to a student model, where noises are added tothe teacher predictions to satisfy DP.
Revised: PATE [31] is another type of DP learning algorithm that transfers knowledge from teachers trained on private data to a student model, where noises are added to teacher predictions to satisfy DP.

Original: PATEis model-agnostic and its privacy cost derives from the knowledge distillation  instead of thegradientsof the whole model  in NoisySGD [39, 22].
Revised: PATE is model-agnostic, and its privacy cost derives from the knowledge distillation process instead of the model gradients in NoisySGD [42, 24].

Original: Hence, noises required by PATE do not scale with modelsize.
Revised: Therefore, the noises required by PATE do not scale with model size.

Original: With this benefit, PATE hasa great potential forbeing adapted to text generations, since largemodels (e.g., GPT-2 [33]) become the mainstream in text generation.
Revised: Given this benefit, PATE has great potential for text generation, since large language models (e.g., GPT-2 [35]) have become the backbone of most text generation models.

Original: However, NoisySGD and PATEare used to protect sample-level privacy [46, 22] but require a high privacy cost on protecting sensitivephrases in samples [20, 36].
Revised: However, NoisySGD and PATE are used to protect sample-level privacy [50, 24] and not customized to protect sensitive phrases in the data with a low privacy cost [22, 39].

Original:  PATE,  designed for classification tasks, is not customized for sequentialgeneration on a large output space (i.e.,  vocabulary), which is very common in text generation.
Revised: Additionally, PATE, originally designed for classification tasks, is not customized for sequential generation on a large output space (i.e., the natural language vocabulary), which is very common in text generation.

Original: In this paper, we propose SeqPATE, a DP learning algorithm for text generations to protect privacyin training corpora.
Revised: In this paper, we propose SeqPATE, a DP learning algorithm for text generation to protect the privacy of training corpora.

Original: By satisfying DP, SeqPATE have a quantifiable guarantee on preventing theexistence of training samples and sensitive phrases in  training corpora from being detected
Revised: By satisfying DP, SeqPATE has the guarantee of preventing the existence of training samples and sensitive phrases in the training corpora from being detected. Similarly to PATE,

Original: SeqPATE employs a teacher-student framework: (i) a student model learns to generate texts on non-sensitivesamples; and (ii) a number of teacher models, trained on sensitive texts, supervise the student throughthe noised outputs of aggregated teachers.
Revised: SeqPATE employs a teacher-student framework: (i) a student model learns to generate text from nonsensitive samples; and (ii) a number of teacher models, trained on sensitive text, supervise the student through noised outputs of aggregated teachers.

Original: The calibrated noises on the outputs ensure SeqPATE tosatisfy DP requirements.
Revised: The calibrated noise added to the output ensures that SeqPATE satisfies the DP requirements.

Original: This framework still faces some challenges in text generations.
Revised: This framework still faces some challenges in text generation.

Original: Firstly,it suffers from  high costs of GPU memory and time.
Revised: First, it suffers from the high costs of GPU memory and time.

Original: To obtain sentence-level supervision for textgeneration, the model needs to roll out all teachers to produce a sentence (i.e. all teachers vote togenerate a word, which is  used as the input for the next word prediction).
Revised: To obtain sentence-level supervision for text generation, the model needs to roll out all teachers to produce a sentence (i.e. all teachers vote to generate a word, which is then used as the input for the next word prediction).

Original: It results in a high inferencecost with a large number of teachers (e.g. 2 k teachers which are common in PATE).
Revised: It results in a high inference cost with a large number of teachers (e.g. 2 k teachers which are common in PATE).

Original: Secondly, thelarge output space (i.e.,  vocabulary) in text generation leads to (i) low agreement rates among teachersand (ii) large noises required by DP, both of which significantly hurt the task performance.
Revised: Second, the large output space (i.e., the vocabulary) in text generation leads to (i) low agreement rates among teachers and (ii) large noises required by DP, both of which significantly hurt the task performance.

Original: To address the challenges, we generate pseudo data using a pre-trained language model so thatteachers only need to provide token-level supervision given the pseudo inputs.
Revised: To address the challenges, we generate pseudo-data using a pre-trained language model so that teachers only need to provide token-level supervision given the pseudo inputs.

Original: To handle the largeoutput space and reduce the noise, we propose to dynamically filter the candidate words and selectonly words with large probabilities.
Revised: To handle the large output space and reduce the noise, we propose to dynamically filter the candidate words and select only words with high probabilities.

Original: Also, we aggregate teachers’ outputs by interpolating their outputdistributions instead of voting with argmax predictions.
Revised: Also, we aggregate teachers’ outputs by interpolating their output distributions instead of voting with argmax predictions.

Original: DP learning methods provide protectionsby adding noises and correspondingly reduce the model utility.
Revised: DP learning methods provide privacy protection by adding noise, which also reduces the utility of the model.

Original: To reducethe utility loss, we avoidunnecessary knowledge distillation: selectively applyingthe knowledge distillation to the generationstep only if the student has trouble with the generation.
Revised: To reduce utility loss, we avoid unnecessary knowledge distillation by selectively applying knowledge distillation to generation steps where the student struggles.

Original: Most DP learning methods, including SeqPATE, prevent samples from being extracted.
Revised: Most DP learning methods, including SeqPATE, prevent samples from being extracted.

Original: SeqPATE has further advantages in protecting users’secret phrases that occur multiple times in  corpora.
Revised: SeqPATE has further advantages in protecting users’ secret phrases that occur multiple times in the corpora.

Original: We evaluate SeqPATE on a sentence completiontask, which shows its advantages on protecting samples and phrases compared to the baselines.
Revised: We evaluate SeqPATE on a sentence completion task, which demonstrates its advantage in protecting samples and phrases compared to the baselines.

Original: Our contribution is twofold: (i)
Revised: Our contribution is twofold: (i)

Original: We propose SeqPATE to achieve the protections on both samplelevel and  phrase level and also feed theoretical analyses on the two levels.
Revised: We propose SeqPATE that provides privacy at both the sample level and the phrase level with theoretical analyses.

Original: (ii) We propose severalstrategies for SeqPATE to handle  text generations with asequence of classifications over large spaces.
Revised: (ii) We propose several strategies for SeqPATE to handle autoregressive text generation models with a large vocabulary.

Original:  Problem Setup
Revised:  Problem Setup

Original: Our goal is to achieve the privacy protection quantified by DP inthe text generation: preventingattackers from inferring whether a sample or an n-gram appears in the training set.
Revised: Our goal is to achieve the privacy protection quantified by DP in text generation to prevent attackers from inferring whether a sample or an n-gram appears in the training set.

Original: Our settingcontains two types of textual datasets: (1) a private set D pri from a corpus with sensitive information,(2) a public set D pub that contains no sensitive information or comes from data contributors (e.g.,volunteers) who have no objection to publishing their data.
Revised: Our setting contains two types of textual datasets: (1) a private set D pri from a corpus with sensitive information, (2) a public set D pub that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data.

Original: We aim to protect the privacy on theprivate set and can ignore the privacy protection on the public set.
Revised: We aim to protect the privacy on the private set and can ignore the privacy protection on the public set.

Original: Our application, sentence completion, aims to complete the whole sentence given the prefix.
Revised: Our application, sentence completion, aims to complete the whole sentence given the prefix.

Original: We traina language model to achieve the task.
Revised: We train a language model to accomplish the task.

Original: The public set D pub consists ofthe prefixes, which can hardlycontain sensitive information.
Revised: The public set D pub consists of prefixes, which can hardly contain sensitive information.

Original: The private set D pri consists ofthe whole sentences.
Revised: The private set D pri consists of whole sentences.

Original: Such a setting fitssome real-world text generation applications: in dialog systems, the training samples from onlineservices consist of questions and responses.
Revised: Such a setting fits some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses.

Original: The questions from customer service staff or servicerobots can be public and the response from users carrying individual information should be private.
Revised: The questions from customer service staff or service robots can be public, and the response from users carrying individual information should be private.

Original:  Background on DP and PATE
Revised:  Background on DP and PATE

Original: Definition 3.1.
Revised: Definition 3.1.

Original: [Differential privacy (DP) [11, 12]] For any two neighboring datasets D , D ′ (differ inonly one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,
Revised: [Differential privacy (DP) [13, 14]] For any two neighboring datasets D , D ′ (differ in only one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,

Original: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, where S ⊆ Y , ε > 0 , δ ≥ 0 .
Revised: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, where S ⊆ Y , ε > 0 , δ ≥ 0 .

Original: As the definition, DP is a quantifiable definition of privacy that provides guarantees on identificationsof individual data (prevent an adversary from inferring whether the input is D or D ′ ).

Original: ML modelswith DP ensure that each training sample has a degree of plausible deniability , i.e., the trained modelis just as likely as to be trained on an alternative dataset without that sample.

Original: In SeqPATE, M is thewhole training and inference process, S is the vocabulary, and Pr [ · ] denotes the output distributionof generating a word.

Original: Attackers cannot tell whether a sample is in the training set or not, since theoutput distributions of the datasets with or without that sample are very similar (bounded by Eq. 1).

Original: PATE [29], designed for classification tasks, takes advantage of an unlabeled public dataset D puband also trains on a labeled private set D pri in a semi-supervised scenario.

Original: PATE achieves DP via a

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: Private Data +Noise
Revised: Private Data +Noise

Original: Teacher
Revised: Teacher

Original: Teacher Teacher
Revised: Teacher Teacher

Original: Student
Revised: Student

Original: NLL loss
Revised: NLL loss

Original: Data Data
Revised: Data Data

Original: Data
Revised: Data

Original: Cats sit on the Cats sit on the
Revised: Cats sit on the Cats sit on the

Original: Cats sit on the
Revised: Cats sit on the

Original: Candidate Filtering |Vocabulary|
Revised: Candidate Filtering |Vocabulary|

Original: Sit 0.42 Sat 0.33 Lay 0.
Revised: Sit 0.42 Sat 0.33 Lay 0.

Original: Are 0.08 ...
Revised: Are 0.08 ...

Original: Alaia 1eRuth 1eTop-K
Revised: Alaia 1eRuth 1eTop-K

Original: Aggregate
Revised: Aggregate

Original: Teacher Inference sit on the mats Teacher

Revised: Teacher Inference sit on the mats Teacher

Original: Teacher

Revised: Teacher

Original: Teacher
Revised: Teacher

Original: Cats sit
Revised: Cats sit

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: [Figure]
Revised: [Figure]

Original: supervisions
Revised: supervisions

Original: [Figure]
Revised: [Figure]

Original: supervisions
Revised: teacher supervisions)

Original: [Figure]
Revised: [Figure]

Original: Student Training
Revised: Student Training

Original: [Figure]
Revised: [Figure]

Original: Distillation teacher-student framework with  M teacher models and a student model, where the student learnsfrom the private set via knowledge distillation through teachers.
Revised: Distillation By definition, DP is a quantifiable definition of privacy that provides guarantees on identifications of individual data (preventing an adversary from inferring whether the input is D or D ′ ). ML models with DP ensure that each training sample has a degree of plausible deniability , i.e., the trained model is just as likely as to be trained on an alternative dataset without that sample. In SeqPATE, M is the entire training and inference process, S is the vocabulary, and Pr [ · ] denotes the output distribution of generating a word.

Revised: Attackers cannot tell whether a sample is in the training set or not, since the output distributions of the datasets with or without that sample are very similar (bounded by Eq. 1).

Revised: PATE [31], designed for classification tasks, takes advantage of an unlabeled public dataset D pub and also trains on a labeled private set D pri in a semi-supervised scenario.

Revised: PATE achieves DP through a teacher-student framework with M teacher models and a student model, where the student learns from the private set via knowledge distillation through teachers.

Original: PATE has three parts: (i)
Revised: PATE has three parts: (i)

Original: Theteacher models are trained on the private set D pri , which is shuffled and divided into M disjointsubsets.
Revised: The teacher models are trained on the private set D pri , which is shuffled and divided into M disjoint subsets.

Original: Each teacher is trained on one subset.
Revised: Each teacher is trained on one subset.

Original: (ii) Teacher aggregation merges  teachers’ outputs.
Revised: (ii) Teacher aggregation merges the teachers’ outputs.

Original: Given the trained teachers, we employ each teacher to provide supervision on the student’s unlabeledpublic set D pub .
Revised: Each of the trained teachers then provides supervision to the student’s unlabeled public set D pub .

Original: We use noised majority votes from teachers as labels to supervise the student.
Revised: We use noised majority votes from teachers as labels to supervise the student.

Original: (iii) Astudent model is trained on the public set D pub with the supervision of the aggregated teachers.
Revised: (iii) A student model is trained on the public set D pub with the supervision of the aggregated teachers.

Original:  Approach
Revised:  Approach

Original: Fig.
Revised: Fig.

Original: 1 shows an overview of SeqPATE.
Revised: 1 shows an overview of SeqPATE.

Original: Given the public prefix (e.g., “Cats sit”), we first obtain thepseudo inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained languagemodel (Sec. 4.1).
Revised: Given the public prefix (e.g., “Cats sit”), we first obtain the pseudo-inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained language model (Sec. 4.1).

Original: At each word, we then aggregate the teachers’ prediction of the next word assupervision for training the student model (Sec. 4.2).
Revised: At each word, we then aggregate the teachers’ prediction of the next word as supervision for training the student model (Sec. 4.2).

Original: To reduce the noise required by DP for alarge output space of the size of the vocabulary, we reduce the output space by dynamically filteringunimportant words.
Revised: To reduce the noise required by DP for a large output space of the size of the vocabulary, we reduce the output space by dynamically filtering unimportant words.

Original: To reduce the number of teacher queries that incur privacy losses, we proposean efficient knowledge distillation strategy that only queries teacher labels on uncertain examples(Sec.
Revised: To reduce the number of teacher queries that incur privacy losses, we propose an efficient knowledge distillation strategy that only queries teacher labels on uncertain examples (Sec. 4.3).

Original: We show the training algorithm in App.
Revised: We show the training algorithm in App.

Original: B and a running example in App.
Revised: B and a running example in App.

Original: L.
Revised: K.

Original: 4.1 Pseudo Input Generation
Revised: 4.1 Pseudo Input Generation

Original: Conventional text generation models generate words sequentially from left to right.
Revised: Conventional text generation models generate words sequentially from left to right.

Original: Thus naivelyapplying PATE to text generation requires rolling out all teachers word by word, i.e. iterativelysampling the next word from the aggregated teacher prediction.
Revised: Thus, naively applying PATE to text generation requires rolling out all teachers word by word, i.e., iteratively sampling the next word from the aggregated teacher prediction.

Original: This is costly in both computation(running inference for hundreds of teacher models) and privacy costs (querying teachers at everystep).
Revised: This is costly in both computation (running inference for hundreds of teacher models) and privacy costs (querying teachers at every step).

Original: To tackle this challenge, we use a pre-trained language model to complete the public prefixesinto pseudo sentences, thus we only need to query teachers on the next word given a (pseudo) context.
Revised: To tackle this challenge, we use a pre-trained language model to complete the public prefixes into pseudo sentences; thus, we only need to query teachers on the next word given a (pseudo) context.

Original: 4.2 Teacher Aggregation
Revised: 4.2 Teacher Aggregation

Original: PATE aggregates teachers’ predictions by majority vote.
Revised: PATE aggregates teacher predictions by majority vote.

Original: While it works for classification problemswith a relatively small number of classes, the output space of text generation  contains all words in thevocabulary.
Revised: While it works for classification problems with a relatively small number of classes, the output space of text generation models contains all words in the vocabulary.

Original: As a result, the number of votes for each candidate word may be very low without a clearwinner.
Revised: As a result, the number of votes for each candidate word may be very low without a clear winner.

Original: For example, multiple candidates may tie for the top-1 prediction.
Revised: For example, multiple candidates may tie for the top-1 prediction.

Original: Inspired by Chen et al.
Revised: Inspired by Chen et al.

Original: [8, 15], we aggregate teachers’ results by averaging their output distributions.
Revised: [9, 17], we aggregate teacher results by averaging their output distributions.

Original: We first train M teacher models on disjoint subsets of the private data.
Revised: We first train M teacher models on disjoint subsets of the private data.

Original: To produce the aggregatednext word distribution given a context c , we average the teachers’ output distributions, add calibratednoises, and then renormalize the results into a proper distribution.
Revised: To produce the aggregated next word distribution given a context c , we average the teachers’ output distributions, add calibrated noises, and then renormalize the results into a proper distribution.

Original: Following Papernot et al.
Revised: Following Papernot et al.

Original: [30],we apply the Gaussian mechanism.
Revised: [32], we apply the Gaussian mechanism.

Original: Formally, let p mϕ ( · | c ) be the  m -th teacher’s prediction.
Revised: Formally, let p mϕ ( · | c ) be the prediction of the m -th teacher.

Original: Theaggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)
Revised: The aggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)

Original: Mm =

Revised: Mm =

Original: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 1 where the Gaussian noiseis added to the aggregated output distribution.
Revised: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 2 where the Gaussian noise is added to the aggregated output distribution.

Original: The way of SeqPATE satisfies DP guarantee (Eq. 1) isto add that calibrated noise to the teachers’ output as mentioned above (detailed analyses in Sec. 5).
Revised: The way of SeqPATE satisfies DP guarantee (Eq. 1) is to add that calibrated noise to the teachers’ output as mentioned above (detailed analyses in Sec. 5).

Original: 4.3 Training of the Student Model
Revised: 4.3 Training of the Student Model

Original: The student model is trained onthe public pseudo data and also supervised by the aggregated teachers.
Revised: The student model is trained on public pseudo-data and also supervised by the aggregated teachers.

Original: Training objectives.
Revised: Training objectives.

Original: The student model is a language model that predicts the next word givencontexts.
Revised: The student model is a language model that predicts the next word given prior contexts.

Original: Given contexts from  (public) pseudo data autocompleted by a pre-trained language model(GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in thepseudo data (i.e. pseudo label).
Revised: Given contexts from the (public) pseudo-data autocompleted by a pre-trained language model (GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in the pseudo-data (i.e. pseudo label).

Original: The pseudo data act as a prior for the student given that the numberof teacher queries is limited due to privacy concerns.
Revised: The pseudo-data acts as a prior for the student given that the number of teacher queries is limited due to privacy concerns.

Original: The student’s loss function has two parts:
Revised: The student’s loss function has two parts:

Original: • L teacher denotes the loss with respect to the teachers’ supervision.
Revised: • L teacher denotes the loss with respect to teacher supervision.

Original: Note that the teachers’ aggregatedoutput is a distribution over words.
Revised: Note that the aggregated teacher output is a distribution over words.

Original: Therefore, we minimize the

Original:  forward KL divergence between the aggregated teacher distribution p agg and 
Revised: Therefore, we minimize the forward KL divergence between the aggregated teacher distribution p agg and the student output distribution p θ :

Original: the

Original: student output distribution p θ

Original: [Equation]
Revised: [Equation]

Original: • L pseudo denotes the loss with respect to the pseudo labels w from ˜ D pub (i.e. next words generated by a
Revised: L pseudo denotes the loss with respect to the pseudo-labels w from ˜ D pub (i e. next words generated by a

Original: generic language model).
Revised: generic language model

Original: Similar to standard language modeling, weuse the negative log-likelihood:
Revised: Similar to standard language modeling, we

Revised: use

Revised: the negative

Revised: log-

Revised: likelihood:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 4 shows the complete loss.
Revised: 4 shows the complete loss.

Original: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)
Revised: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)

Original: [Equation]
Revised: [Equation]

Original: Reducing the output space via candidate filtering.
Revised: Reducing the output space via candidate filtering.

Original: The high-dimensionality of the output of textgeneration models results in large noise (which is added to each coordinate).
Revised: The high-dimensionality of the output of text generation models results in large noise (which is added to each coordinate).

Original: To reduce the outputdimension (hence the amount of noise), we filter words on the tail of the distribution of the studentmodel (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and thestudent output distribution over the rest words.
Revised: To reduce the output dimension (hence the amount of noise), we filter words on the tail of the distribution of the student model (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and the student output distribution over the rest words.

Original: Note that the candidate filtering is based on the student’s outputs on public or already released inputs,thus it does not affect the privacy guarantee.
Revised: Note that the candidate filtering is based on the student’s outputs on public or already released inputs, thus it does not affect the privacy guarantee.

Original: This choice improves the privacy-utility tradeoff byadaptively allocating theavailable privacy budgets to release the information more helpful to the task.
Revised: This choice improves the privacy-utility tradeoff by adaptively allocating the privacy budget to release the information most helpful to the task.

Original: We experiment with two filtering strategies: top- k and top- p .
Revised: We experiment with two filtering strategies: top- k and top- p .

Original: In top- k filtering, we retain only thetop- k most likely candidates and filter the rest according to the student model.
Revised: In top- k filtering, we retain only the top- k most likely candidates and filter the rest according to the student model.

Original: In top- p filtering [16],k is chosen dynamically such that the top- k words are the minimum set whose cumulative probabilityis at least p .
Revised: In top- p filtering [18], k is chosen dynamically such that the top- k words are the minimum set whose cumulative probability is at least p .

Original: The strategy seldom loses good candidates, because the student usually does well ontop- k predictions since the beginning of the training.
Revised: The strategy seldom loses good candidates because the student usually does well on top- k predictions since the beginning of the training.

Original: Reducing the number of teacher queries via efficient knowledge distillation.
Revised: Reducing the number of teacher queries via efficient knowledge distillation.

Original: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.
Revised: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.

Original: Therefore,we obtainthe teacher supervision only on “hard” examples in student’s training.
Revised: Therefore, we obtain teacher supervision only on “hard” examples when training the student.

Original: Note that the studentis trained on both the pseudo data and local supervision from the teachers.
Revised: Note that the student is trained on both the pseudo-data and local supervision from the teachers.

Original: We consider an exampleso to be hard if the student cannot imitate the pseudo label, in which case distilling knowledge from theteachers that are trained on large private data is helpful.
Revised: We consider an example to be hard if the student cannot imitate the pseudo-label, in which case distilling knowledge from the teachers that are trained on large private data is helpful.

Original: Concretely, we querythe teachers only when the rank of the pseudo label is below a certain thresholdamong words ordered by descending probabilities under the student model.
Revised: Concretely, we query teachers only when the rank of the pseudo-label is below a certain threshold among words ordered by descending probabilities under the student model.

Original: If we query the teachers,the student is trained via complete loss L ( p agg , ˜ D pub )
Revised: If we query the teachers, the student is trained via complete loss L ( p agg , ˜ D pub )

Original: (Eq. 4); otherwise, the student is trained via the
Revised: (Eq. 4); otherwise, the student is trained via the

Original: L pseudo (Eq. 3).
Revised: L pseudo (Eq. 3).

Original: We note that the selection of tokens relies only on the student and is independent ofthe teachers; thus the selection does not cause any additional privacy loss.
Revised: We note that the selection of tokens relies only on the student and is independent of the teachers; thus, the selection does not cause any additional privacy loss.

Original:  Privacy Analyses 
Revised:  Privacy Analyses 5.1 Preliminary of Differential Privacy

Original: 5.1 Preliminary of Differential Privacy

Original: Lemma 5.1 (Analytical Gaussian mechanism [2]) .
Revised: Lemma 5.1 (Analytical Gaussian mechanism [2]) .

Original: For a numeric query f : X n → R d over adataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies( ε, δ ( ε ))
Revised: For a numeric query f : X n → R d over a dataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) =
Revised: -DP for all ε ≥ 0 and δ ( ε ) =

Original: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .
Revised: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .

Original: where ∆ := ∆ ( f )=max D∼D ′ ∥
Revised: where ∆ := ∆ ( f )= max D∼D ′ ∥

Original: f ( D ) − f ( D ′ )
Revised: f ( D ) − f ( D ′ )

Original: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .
Revised: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .

Revised: We can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.

Original: Lemma 5.2 (Composition of Gaussian mechanisms [10]) .
Revised: Lemma 5.2 (Composition of Gaussian mechanisms [11]) .

Original: The adaptive composition of a sequenceof Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .
Revised: The adaptive composition of a sequence of Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .

Original: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .
Revised: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .

Original: satisfies( ε, δ ( ε ))
Revised: satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noisemultiplier σ/ ∆ =
Revised: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noise multiplier σ/ ∆ =

Original: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .
Revised: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .

Original: Specifically, the adaptive composition of a k identical Gaussian mechanism with  noise multiplier σsatisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .
Revised: Specifically, the adaptive composition of a k identical Gaussian mechanism with a noise multiplier σ satisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .

Original: By fixing k and ε , we can calibrate the noises by choosing an appropriate σ in Sec.
Revised: By fixing k and ε , we can calibrate the noise by choosing an appropriate σ in Sec.

Original: 5.2 Differential Privacy for Language Models at the Sample Level
Revised: 5.2 Differential Privacy for Language Models at the Sample Level

Original: Recall that we partition the private dataset to M disjoint subsets, and train each teacher model onone of the subsets.
Revised: Recall that we partition the private dataset into M disjoint subsets, and train each teacher model on one of the subsets.

Original: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -thteacher model given some context, where |V| is the vocabulary size.
Revised: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -th teacher model given some context, where |V| is the vocabulary size.

Original: The aggregation functionf ( D ) :
Revised: The aggregation function f ( D ) :=

Original: = (cid:80) Mi
Revised: (cid:80) Mi

Original: =1 x i is the sum of the probability distributions predicted over allthe teachers.
Revised: =1 x i is the sum of the probability distributions predicted by all teachers.

Original: Since thedatasets are disjoint and have no duplicate samples, changing one sampleonly affects  one teachermodel.
Revised: Since the datasets are disjoint, changing one sample affects only one teacher model.

Original: For neighboring datasets D , D ′ , let j denote the index of each teacher model, then theprobability distributions x j and x ′ j (deriving from D and D ′ respectively) are different.
Revised: For neighboring datasets D , D ′ , let j denote the index of each teacher model; the probability distributions x j and x ′ j (derived from D and D ′ respectively) are different.

Original: Then, thesensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),
Revised: Then, the sensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),

Original: [Equation]
Revised: [Equation]

Original: Adding the noises given by Lemma 5.2 to each coordinate (each candidate at each generation stepof SeqPATE) preserves ( ε , δ ( ε ) )
Revised: Adding the noises given by Lemma 5.2 to each coordinate (each candidate at each generation step of SeqPATE) preserves ( ε , δ ( ε ) )

Original: -DP for f ( D ) .
Revised: -DP for f ( D ) .

Original: Finally, when we extract top- k coordinates by top- kcandidate filtering (Sec. 4.3), the privacy guarantee also holds owing to  post-processing property [12].
Revised: Finally, when we extract top- k coordinates by top- k candidate filtering (Sec. 4.3), the privacy guarantee also holds due to the post-processing property [14].

Original: So, the fact about whether a sample is in SeqPATE’s private sets is protected (satisfying ( ε , δ ( ε ) )-DP).
Revised: Therefore, the fact about whether a sample is in SeqPATE’s private sets is protected (satisfying ( ε , δ ( ε ) )-DP).

Original: 5.3 Differential Privacy of Users’ Secret Phrases
Revised: 5.3 Differential Privacy of Users’ Secret Phrases

Original: The above analyses show that we can protect the privacy of each sample (i.e., one occurrence of asentence).
Revised: The above analyses show that we can protect the privacy of each sample (i.e., one occurrence of a sentence).

Original: However, in practical text generations, we may want to protect all occurrences of somesecret phrases specific to a user (e.g., names and addresses).
Revised: However, in practice, we may want to protect all occurrences of some secret phrases specific to a user (e.g., names and addresses).

Original: 3 Consider a secret phrase s occurring n stimes ( n s ≥ 1 ) in the private set.
Revised: 4 Consider a secret phrase s that occurs n s times ( n s ≥ 1 ) in the private set.

Original: According to group privacy [12], the protection on phrase s satisfies( nε, e nε − 1 e ε − 1 δ ) -DP
Revised: According to group privacy [14], the protection on phrase s satisfies ( nε, e nε − 1 e ε − 1 δ ) -DP

Original: [20, 36], where the privacy loss scales linearly with the number of occurrences of s(We discuss and analyze a better strategy to reduce the privacy loss of baselines in App. N).
Revised: [22, 39], where the privacy loss scales linearly with the number of occurrences of s (We discuss and analyze a better strategy to reduce the privacy loss of baselines in App. M).

Original: Naively applying a DP algorithm requires larger noise to protect phrases that may occur multipletimes, SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or a fewteachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.
Revised: Naively applying a DP algorithm requires larger noise to protect phrases that may occur multiple times. SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or a few teachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.

Original: We denote ˜ n s as the number of teachers whose data contain the phrase s .
Revised: We denote ˜ n s as the number of teachers whose data contain the phrase s .

Original: Since adding or removingthe phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s(See App.
Revised: Since adding or removing the phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s (See App.

Original: D for details).
Revised: D for details).

Original: In this way, the strength of protection on secret phrases is roughly equal tothat we have derived for sample-level DP.
Revised: In this way, the strength of protection on secret phrases is roughly equal to that we have derived for sample-level DP.

Original: The exact ( ε, δ ( ε, ˜ n s ))
Revised: The exact ( ε, δ ( ε, ˜ n s ))

Original: -DP for the phrase s can be obtainedaccording to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =
Revised: -DP for the phrase s can be obtained according to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =

Original: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .
Revised: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .

Original: Unlikeother generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacy loss (i.e.a linear increase in ε ) on user phrases by careful partitioning of the private data.
Revised: Unlike other generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacy loss (i.e., a linear increase in ε ) on user phrases by careful partitioning of the private data.

Original: 5.4 How does DP prevent memorization in SeqPATE?
Revised: 5.4 How does DP prevent memorization in SeqPATE?

Original: In practice, the privacy of the language model is usually interpreted as not generating a secret phrasein the training data as-is during inference.
Revised: In practice, the privacy of the language model is usually interpreted as not generating a secret phrase in the training data as-is during inference.

Original: Thus, one may wonder how DP prevents such unintendedmemorization of the training data.
Revised: Thus, one may wonder how DP prevents such unintended memorization of the training data.

Original: We remark that the protection against memorization follows thedefinition of DP.
Revised: We remark that the protection against memorization follows the definition of DP.

Original: Consider the attack by Carlini et al.
Revised: Consider the attack by Carlini et al.

Original: [5], which uses a language model to predict asecret phrase s given a prefix.
Revised: [6], which uses a language model to predict a secret phrase s given a prefix.

Original: By the closure to post-processing [12], the prediction also satisfies
Revised: By the closure to post-processing [14], the prediction also satisfies

Original: DP.
Revised: DP.

Original: We denote W as the undesirable event where SeqPATE generates the phrase s verbatim.
Revised: We denote W as the undesirable event where SeqPATE generates the phrase s verbatim.

Original: DPdefinition implies that the probability of W to happen when s is in the SeqPATE’s private sets is atmost e ε larger than the probability of an alternative SeqPATE model trained without s in those sets.
Revised: The DP definition implies that the probability of W to happen when s is in the SeqPATE’s private sets is at most e ε larger than the probability of an alternative SeqPATE model trained without s in those sets.

Original: The chances for the latter model to generate text with s are astronomically small.
Revised: The chances for the latter model to generate text with s are astronomically small.

Original: Hence, DP impliesthat the probability of W under the former model (i.e. any SeqPATE model in general) is small.
Revised: Hence, DP implies that the probability of W under the former model (i.e. any SeqPATE model in general) is small.

Original:  Experiments 
Revised:  Experiments 6.1 Experimental Settings

Original: 6.1 Experimental Settings

Original: Datasets.
Revised: Datasets.

Original: We use two datasets: AirDialog [44] has 1M utterances from customer service  on flightbooking; Europarl_v6 has 2M  sentences collected from European Parliament.
Revised: We evaluate our model on two datasets. AirDialog [47] consists of 1M utterances from customer service dialog on flight booking; Europarl_v6 consists of 2M English sentences collected from European Parliament.

Original: (Details in App. E.)
Revised: 5 (See details about datasets in App. E.)

Original: Baselines.
Revised: Baselines.

Original: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on theprivate data with calibrated noise on clipped gradients [1, 20] and further trained on public set D pubwithout protection; (2) based on NoisySGD , NoisySGD+GC [22] applies a ghost clipping whichenables large batch size with memory saving techniques.
Revised: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on the private data with calibrated noise on clipped gradients [1, 22] and further trained on public set D pub without protection; (2) based on NoisySGD , NoisySGD+GC [24] applies a ghost clipping which enables large batch size with memory saving techniques.

Original: Additionally, we use two non-DP methodsas reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the publicpre-trained GPT-2 model Pub-GPT without access to private data.
Revised: Additionally, we use two non-DP methods as reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the public pre-trained GPT-2 model Pub-GPT without access to private data.

Original: For all methods, we can optionallyfine-tune on the generated pseudo data as a warm-up, and the operation is denoted as + ˜ D pub .
Revised: For all methods, we can optionally fine-tune on the generated pseudo-data as a warm-up, and the operation is denoted as + ˜ D pub .

Original: Implementation details.
Revised: Implementation details.

Original: All models are fine-tuned from the (public) pre-trained GPT-2 model [33].
Revised: All models are fine-tuned from the (public) pre-trained GPT-2 model [35].

Original: The batch size is 32 for all comparing methods except the GC [22] (GC [22] requires 2048).
Revised: The batch size is 32 for all comparing methods except the GC [24] (GC [24] requires 2048).

Original: We use
Revised: We use

Original: Adam [21] and adjust the initial learning rate with a range of 10 − 3 to 10 − 6 for all methods.
Revised: Adam [23] and adjust the initial learning rate with a range of 10 − 3 to 10 − 6 for all methods.

Original: The δmentioned in Sec.
Revised: The δ mentioned in Sec.

Original: 5 for all DP methods is 10 − 6 .
Revised: 5 for all DP methods is 10 − 6 .

Original: For SeqPATE, before training the student model withteacher supervision, we first fine-tune it on thepseudo public data ˜ D pub as a warm-up.
Revised: For SeqPATE, before training the student model with teacher supervision, we first fine-tune it on the public pseudo-data ˜ D pub as a warm-up.

Original: The coefficientλ that balances supervision for the teacher and the pseudo data (Eq. 4) is set to 20, where we havetuned it on the validation set of the public pseudo data.
Revised: The coefficient λ that balances supervision for the teacher and the pseudo-data (Eq. 4) is set to 20, where we have tuned it on the validation set of the public pseudo-data.

Original: The default number of teacher models is 2 k ,where our model works well according to the experiments in App.
Revised: The default number of teacher models is 2 k , where our model works well according to the experiments in App.

Original: H. We designed some strategies 4to reducethe use of the memory and disk  (See strategies and the computational cost in App. I).
Revised: H. We designed some strategies 6 to reduce memory and disk usage (See strategies and the computational cost in App. I).

Original: Werun SeqPATE with 2 k teachers on a single GPU within 3 days.
Revised: We run SeqPATE with 2 k teachers on a single GPU in 3 days.

Revised: Our code is publicly accessible.

Original: (See  hyperparameters in App. G.)
Revised: (See details about hyperparameters in App. G.)

Original:  Metrics.
Revised: Evaluation Metrics.

Original: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [31].
Revised: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [33].

Original: 6.2 Overall Performance
Revised: 6.2 Overall Performance

Revised: [Table]

Revised: [Table]

Original: Protection at the sample level.
Revised: Protection at the sample level.

Original: Tab. 1 show the performance on the two datasets (See more metrics(i.e.
Revised: Tab. 1 show the performance on the two datasets.

Original: , Bleu-1 and Bleu-2) in App. K).

Original: Among the non-DP baselines, Pri-GPT acts as an upper boundon the performance since it can fully utilize the private set by discarding privacy protection.
Revised: Among the non-DP baselines, Pri-GPT acts as an upper bound on the performance, since it can fully utilize the private set by discarding privacy protection.

Original: PubGPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additionalablation study on the pseudo data in App.
Revised: Pub-GPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additional ablation study on the pseudo data in App.

Original: J also verifies this).
Revised: J also verifies this).

Original: NoisySGD+GC+
Revised: NoisySGD+GC+

Original: ˜
Revised: ˜

Original: D pub surpasses theabove two methods since it uses a much larger batch size (2048 vs 32) than NoisySGD.
Revised: D pub surpasses the above two methods, since it uses a much larger batch size (2048 vs 32) than NoisySGD.

Original: Our method,[Table] SeqPATE, significantly outperforms NoisySGD+GC+ ˜
Revised: Our method, SeqPATE, significantly outperforms NoisySGD+GC+ ˜

Original: D pub (+59% in Bleu4 on AirDialog and +7.0%in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε .
Revised: D pub (+59% in Bleu4 on AirDialog and +7.0% in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε .

Original: Protection on the user’s secret phrases.
Revised: Protection on the user’s secret phrases.

Original: We evaluate our method for privacy protection of secretphrases mentioned in Sec 5.3.
Revised: We evaluate our method for privacy protection of secret phrases mentioned in Sec 5.3.

Original: The key step is to partition the data such that each phrase only occursin the training data of very few teachers, which is straightforward given the user ID associated withthe private data.
Revised: The key step is to partition the data such that each phrase only occurs in the training data of very few teachers, which is straightforward given the user ID associated with the private data.

Original: In general, SeqPATE works with any set of secret phrases.
Revised: In general, SeqPATE works with any set of secret phrases.

Original: In our experiments, weconsider a user’s full name as their secret phrase since it can be easily recognized from the data.
Revised: In our experiments, we consider a user’s full name as their secret phrase since it can be easily recognized from the data.

Original: Wepartition AirDialog’s private data according to the accompanying user IDs.
Revised: We partition AirDialog’s private data according to the accompanying user IDs.

Original: As a result, there are96.6% users whose data are assigned to a single teacher (details about the data partition in App. F). 
Revised: As a result, there are 96.6% users whose data are assigned to a single teacher (details about the data partition in App. F). PPL ↓ Bleu-3 ↑

Original: [Table]
Revised: [Table]

Original: As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.
Revised: -4 ↑ NoisySGD+ [Equation] +

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: NoisySGD

Revised: [Equation]

Original: In Tab.
Revised: users) [Table] [Table] [Table]

Revised: As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.

Original: 2,we see that NoisySGD+GC+ ˜
Revised: In Tab. 2, we see that NoisySGD+GC+ ˜

Original: D pub needsa large noise to achieve a satisfactory level of protection onphrases, because ε increases linearly with thephrase frequency  (group privacy [12]).
Revised: D pub needs large noise to achieve a satisfactory level of protection on phrases, because ε increases linearly with the frequency of the phrase (group privacy [14]).

Original: “Batching users”indicates partitioning data into batches according to users, which helps NoisySGD protect users’phrases (More analyses in App. N).
Revised: “Batching users” indicates partitioning data into batches according to users, which helps NoisySGD protect users’ phrases (more analyses in App. M).

Original: For SeqPATE, the number of teachers trained on data containingthe phrase ˜ n s is close to 1 on average after our partition.
Revised: For SeqPATE, the number of teachers trained on data containing the phrase ˜ n s is close to 1 on average after our partition.

Original: Thus, SeqPATE provides the same level ofprotection on users’ secret phrases with a smaller noise and thus achievesa better performance (+70%and +36% in Bleu4) (See details about thelevel of protection  on users’ secret phrases in App. F).
Revised: Thus, SeqPATE provides the same level of protection on users’ secret phrases with a smaller noise and thus achieves better performance (+70% and +36% in Bleu4) (see more about the protection level on users’ secret phrases in App. F).

Original: [Figure]
Revised: [Figure]

Original: Privacy-utility tradeoff.
Revised: Privacy-utility tradeoff.

Original: In Fig.
Revised: In Fig.

Original: 2, we show the private-utility tradeoff curve of all DP algorithms.
Revised: 2, we show the private-utility tradeoff curve of all DP algorithms.

Original: Typically, DP with ε ∈ [0 . 1 , 10] is considered to provide a meaningful protection [42].
Revised: Typically, DP with ε ∈ [0 . 1 , 10] is considered to provide a meaningful protection [45].

Original: We observethat SeqPATE outperforms NoisySGD and NoisySGD+GC+ ˜
Revised: We observe that SeqPATE outperforms NoisySGD and NoisySGD+GC+ ˜

Original: D pub in this range.
Revised: D pub in this range.

Original: However, SeqPATEdoes not work better than the two methods when ε > 10 .
Revised: However, SeqPATE does not work better than the two methods when ε > 10 .

Original: The reason is that NoisySGD+GC+ ˜
Revised: The reason is that NoisySGD+GC+ ˜

Original: D pubapproaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).
Revised: D pub approaches Pri-GPT as ε approaches infinity (i [Table] e. the noise approaches 0).

Original: However, SeqPATEwith an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: theteachers cannot completely transfer knowledge from the private data to the student.
Revised: However, SeqPATE with an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: the teachers cannot completely transfer knowledge from the private data to the student.

Original: Therefore, wesuggest using SeqPATE if strong privacy protection is desirable.
Revised: Therefore, we suggest using SeqPATE if strong privacy protection is desirable

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 6.3 Ablation Studies
Revised: 6.3 Ablation Studies

Original: There are several design choices in SeqPATE
Revised: There are several design choices in SeqPATE

Original: and we study the importance of each of them.
Revised: and we study the importance of each of them.

Original: In
Revised: In

Original: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachersby voting instead of averaging their output distributions thecross- similar (although notequivalent) to the original the difference is that PATE needs to roll out all teachers
Revised: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachers by voting instead of averaging their output distributions; (2) − KL: training the student using the cross-entropy loss with respect to teachers’ top- prediction instead of

Revised: KL divergence; (3) −L pseudo : not learning from the pseudo label (Eq. 3); (4) − Effi KD: querying teachers on all samples without

Revised: selection; (5) − Gaussian: using the Laplace mechanism as the original PATE algorithm instead of the

Revised: Gaussian mechanism; and (6) − All: using none of the above strategies, which is similar (although not equivalent) to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1)).

Original: Aggregating the teachers by voting and training with KL loss are the most important strategies for
Revised: Aggregating the teachers by voting and training with KL loss are the most important strategies for

Original: SeqPATE.
Revised: SeqPATE.

Original: The poor performance on − Merge_P shows that voting is not suitable for text generation.
Revised: The poor performance on − Merge_P shows that voting is not suitable for text generation.

Original: The reason is that voting over a large output space leads to low agreement rates.
Revised: The reason is that voting over a large output space leads to low agreement rates.

Original: The results show thatthe L pseudo loss makes little contribution to SeqPATE.
Revised: The results show that the L pseudo loss makes little contribution to SeqPATE.

Original: The reason is that we have pre-trained on thestudent’s training set via L pseudo before the student’s training.
Revised: The reason is that we have pre-trained on the student’s training set via L pseudo before the student’s training.

Original: The promotion caused by the efficientknowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows thatthe “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from thisstrategy.
Revised: The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows that the “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from this strategy.

Original: This is because the “clever” student can dramatically save the privacy cost and transfer itto where it would benefit the student most.
Revised: This is because the “clever” student can dramatically save the privacy cost and transfer it to where it would benefit the student most.

Original: The poor performance of − All verifies that the original
Revised: The poor performance of − All verifies that the original

Original: PATE is not suitable forthe text generation.
Revised: PATE is not suitable for text generation.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 6.4 Analyses on Candidate Filtering and Teacher Numbers
Revised: 6.4 Analyses on Candidate Filtering and Teacher Numbers

Original: To analyze candidate filtering with different filtering strategies, we conductthe experiments on top- pand the top- k filtering.
Revised: To analyze candidate filtering with different filtering strategies, we conduct experiments on top- p and top- k filtering.

Original: As shown in Tab. 4, our full model employs the top- p filtering (the thresholdp is 0 .
Revised: As shown in Tab. 4, our full model employs the top- p filtering (the threshold p is 0 .

Original: 95 ) surpasses most variants with manually chosen k .
Revised: 95 ) surpasses most variants with manually chosen k .

Original: Top-
Revised: Top-

Original: k filtering ( k = 50 or 100) alsoworks well.
Revised: k filtering ( k = 50 or 100) also works well.

Original: The filtering with a too-small k ( k = 1 or k = 10 ) implies discarding too much usefulinformation from the supervision ( k = 1 is different to − KL in Tab. 3, which uses the Top-1 ofteachers’ results).
Revised: Filtering with a too small k ( k = 1 or k = 10 ) implies discarding too much useful information from the supervision ( k = 1 is different from − KL in Tab. 3, which uses the Top-1 of teachers’ results).

Original: Filtering with oversize k results in unnecessarily large noises.
Revised: Filtering with oversize k results in unnecessarily large noises.

Original: The candidates with very small probabilities should be filtered duringthe generation; however, random noises mayincrease their probabilities, so models may generate those words that are misled by the noise.
Revised: Candidates with very small probabilities should be filtered during generation; however, random noises may increase their probabilities, so models may generate those words that are misled by the noise.

Original: The results in App.
Revised: The results in App.

Original: H show that more teachers lead to better results when the teacher numbers are ina range of 1 ∼ 2 k .
Revised: H show that more teachers lead to better results when the number of teachers is in the range of 1 ∼ 2 k .

Original: This is because the noise assigned to each teacher drops linearly as the teachernumbers go up.
Revised: This is because the noise assigned to each teacher drops linearly as the number of teachers increases.

Original: Note that SeqPATE cannot always benefit from increasing the teacher numbers,because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.
Revised: Note that SeqPATE cannot always benefit from increasing the teacher numbers, because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.

Original: Wechoose ε = 3 on the sample level protection for all results in Tabs.
Revised: We choose ε = 3 on the sample level protection for all results in Tabs.

Original: 3, 4, and App. H.
Revised: 3, 4, and App. H.

Revised: Additionally, we conduct empirical comparisons and analyses of SeqPATE versus the original PATE in App.

Revised: N.

Revised: We show the effects of protections on users’ secret phrases in App.

Revised: O. We compare

Original:   Related Work
Revised: SeqPATE with another non-DP based baseline (i.e. blacklist based filtering) in App. P. We also conduct a case study in App. Q.  Related Work

Original: Text generation models may leak user information through the generated texts [17, 6].
Revised: Text generation models may leak user information through the generated texts [19, 7].

Original: One directionof privacy protection is to protect author-level (user-level) information.
Revised: One direction of privacy protection is to protect author-level (user-level) information.

Original: The methods prevent attackersfrom inferring the author attributes (e.g., gender, age) [23] and the relationship between informationand authors [27].
Revised: The methods prevent attackers from inferring the author attributes (e.g., gender, age) [25] and the relationship between information and authors [29].

Original: Some researchers [37, 38] infer the membership (whether samples from a givenauthor are used to train the model) given a black-box model.
Revised: Some researchers [40, 41] infer the membership (whether samples from a given author are used to train the model) given a black-box model.

Original: Some papers protect users’ privacy oftraining data against untrusted servers via federated learning [25, 9].
Revised: Some papers protect user privacy of training data against untrusted servers via federated learning [27, 10].

Original: Another direction is to preventattackers from extracting sensitive information in training sets by analyzing the outputs [28, 20],which is in urgent need [6].
Revised: Another direction is to prevent attackers from extracting sensitive information in training sets by analyzing the outputs [30, 22], which is urgently needed [7].

Original: Our SeqPATE focuses on this direction.
Revised: Our SeqPATE focuses on this direction.

Original: In this direction, regularizationmethods [5, 40, 18] restrict the model capacity and avoid the model  memorizing exact trainingsamples.
Revised: In this direction, regularization methods [6, 43, 20] restrict the model capacity and prevent the model from memorizing exact training samples.

Original: Anonymization methods [24, 41] detect sensitive text and replace it with non-sensitivetexts.
Revised: Anonymization methods [26, 44] detect sensitive text and replace it with non-sensitive text.

Original: Unlike DP [12] methods, the above methods do not provide a quantifiable guarantee for privacyprotection.
Revised: Unlike DP [14] methods, the above methods do not provide a quantifiable guarantee for privacy protection.

Revised: Some researchers focus on protecting user privacy against untrusted servers via federated learning [27, 10].

Original: Some researchers apply DP to text generation.
Revised: Some researchers apply DP to text generation.

Original: For user-level privacy, ER-AE [4] augments thesemantic information in the generated text to hide authors’ writing styles from attackers.
Revised: For user-level privacy, ER-AE [4] augments the semantic information in the generated text to hide authors’ writing styles from attackers.

Original: McMahanet al.
Revised: McMahan et al.

Original: [26] propose a recurrent language model with a DP guarantee against the identification of theusers.
Revised: [28] propose a recurrent language model with a DP guarantee against the identification of users.

Original: Note that the user-level privacy (relationships between users and their information) is differentfrom the privacy of users’ secret phrases in our model: Our model prevents individual users’ phrasesfrom being detected.
Revised: Note that the user-level privacy (relationships between users and their information) is different from the privacy of users’ secret phrases in our model: Our model prevents individual user phrases from being detected.

Original: Some researchers apply NoisySGD to text generation to prevent the sensitivetraining data (not the user-level) from being extracted: some of them [35, 36] employ DP to protect apart of selected tokens; others [20, 45, 22] apply DP to protect both samples and all tokens, but theprivacy cost on tokens is very high (Sec. 5.3).
Revised: Some researchers apply NoisySGD to text generation to prevent sensitive training samples from being extracted: some of them [37, 39] employ DP to protect a part of selected tokens; others [22, 49, 24] apply DP to protect both samples and all tokens, but the privacy cost on tokens is very high (Sec. 5.3).

Original: Our model falls into the latter category and reduces theprivacy cost of tokens.
Revised: Our model falls into the latter category and reduces the privacy cost of tokens.

Original: Kerrigan et al.
Revised: Kerrigan et al.

Original: [20] apply NoisySGD [1] to text generation.
Revised: [22] apply NoisySGD [1] to text generation.

Original: Yu et al.
Revised: Yu et al.

Original: [45]investigate fine-tuning strategies on pre-trained language models with NoisySGD.
Revised: [49] investigate fine-tuning strategies on pre-trained language models with NoisySGD.

Original: Li et al.
Revised: Li et al.

Original: [22] applyghost clipping to pre-trained language models with NoisySGD and reduce memory usage.
Revised: [24] apply ghost clipping to pre-trained language models with NoisySGD and reduce memory usage.

Revised: Shi et al.

Revised: [38] apply DP to particular generation steps instead of training samples or n-grams.

Revised: Brown et al.

Revised: [5] analyze DP based method versus data sanitization of text generation models.

Original:  Differential privacy (DP) [11, 12] formally defines and quantifies privacy.
Revised: Brown et al. [12] propose a efficient NoisySGD to speed up model training. Differential privacy (DP) [13, 14] formally defines and quantifies privacy.

Original: ML models with DPguarantee [43, 13, 47] prevents the existence of individual training examples from being detected [5].
Revised: ML models with DP guarantee [46, 15, 51] prevent the existence of individual training examples from being detected [6].

Original: Some researchers protect the privacy of empirical risk minimization classifiers [7] and SVM [34]with DP.
Revised: Some researchers protect the privacy of empirical risk minimization classifiers [8] and SVM [36] with DP.

Original: Following Song et al.
Revised: Following Song et al.

Original: [39], NoisySGD [1] achieves DP on deep learning models by addingnoises to gradients.
Revised: [42], NoisySGD [1] achieves DP on deep learning models by the gradient

Original: Pichapati et al.

Original: [32] adaptively clip the gradient on NoisySGD. PATE [29, 30]transfers the knowledge from teacher models trained on private sets with noises to a student model.
Revised: [31, 32] transfers the knowledge from teacher models trained on private sets with noises to a student model.

Original: KNN-PATE [46] refines PATE by accessing only the k-nearest neighbors from the private set.
Revised: KNN-PATE [50] refines PATE by accessing only the k-nearest neighbors from the private set.

Original: Jordonet al.
Revised: Jordon et al.

Original: [19] adversarially learn to generate synthetic data with discriminators trained by PATE.
Revised: [21] adversarially learn to generate synthetic data with discriminators trained by PATE.

Original: Thosemethods are not customized for text generation models.
Revised: These methods are not customized for text generation models.

Revised: Xie et al.

Revised: [48] propose DPGAN to adversarially learn with a generator and a discriminator.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training datafor text generation models with DP guarantees.
Revised: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training data for text generation models with DP guarantees.

Original: SeqPATE achieves a good privacy-utility tradeoff byleveraging both private and public data.
Revised: SeqPATE achieves a good privacy-utility trade-off by leveraging both private and public data.

Original: As an extension of PATE, SeqPATE can handle the sequentialgeneration paradigm with large output space at each step andtherefore is  adaptive to text generationmodels.
Revised: As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and is therefore adaptive to text generation models.

Original: We avoid rolling out teachers by providing pseudo inputs for the teacher’s inference and thestudent’s training.
Revised: We avoid rolling out teachers by providing pseudo-inputs for the teacher’s inference and the student’s training.

Original: We further reduce the output space by candidate filtering and limit privacy lossesvia efficient knowledge distillation.
Revised: We further reduce the output space by candidate filtering and limit privacy losses via efficient knowledge distillation.

Original: SeqPATE achieves a better performance with the sample levelprotection and further provides much stronger protection on users’ secret phrases.
Revised: SeqPATE achieves a better performance with the sample-level protection and further provides much stronger protection on users’ secret phrases.

Original: The limitation,ethical considerations, and social impacts of this paper are in Appendix A and M.
Revised: The limitations, ethical considerations, and social impacts of this paper are in App. A and L.

Original:  Checklist
Revised:  Acknowledgement

Original: For all authors...
Revised: Research in this paper was supported by Hong Kong Research Grants Council under grand No. HH is partly supported by the Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI). The authors thank Mr. Wei Dong and Dr. Yiping Song for their help and insights on this paper.

Original:  (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] It can be found in the body paper or the supplemental materials.
Revised: [Yes] It can be found in the body paper or the supplemental materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] It can be found in the body paper or the supplemental materials.
Revised: [Yes] It can be found in the body paper or the supplemental materials.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?[Yes]
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?[Yes]
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable[Table] information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] It can be found in the supplemental materials.[Table]
Revised: [Yes] It can be found in the supplemental materials.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable?

Original: [N/A]

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]