Original: SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Revised: SeqPATE: Differentially Private Text Generation via Knowledge Distillation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Protecting the privacy of user data is crucial for text generation models, whichmay leak sensitive information during generation.
Revised: Protecting the privacy of user data is crucial for text generation models, whichmay leak sensitive information during generation.

Original: Differentially private (DP)learning methods provide guarantees against identifying the existence of a trainingsample from model outputs.
Revised: Differentially private (DP)learning methods provide guarantees against identifying the existence of a trainingsample from model outputs.

Original: PATE is a recent DP learning algorithm that achieveshigh utility with strong privacy protections on training samples.
Revised: PATE is a recent DP learning algorithm that achieveshigh utility with strong privacy protections on training samples.

Original: However, textgenerations conduct sequential generations on a large output space, and PATE is notcustomized for that paradigm.
Revised: However, textgenerations conduct sequential generations on a large output space, and PATE is notcustomized for that paradigm.

Original: Besides, PATE does well in protecting sample-levelprivacy but requires a high privacy cost on protecting phrases in samples.
Revised: Besides, PATE does well in protecting sample-levelprivacy but requires a high privacy cost on protecting phrases in samples.

Original: In thispaper, we propose SeqPATE, an extension of PATE on text generation, which aimsto protect the privacy of both training samples and sensitive phrases in samples.
Revised: In thispaper, we propose SeqPATE, an extension of PATE on text generation, which aimsto protect the privacy of both training samples and sensitive phrases in samples.

Original: To adapt to text generations, we generate pseudo inputs and reduce the sequencegeneration problem to next word predictions.
Revised: To adapt to text generations, we generate pseudo inputs and reduce the sequencegeneration problem to next word predictions.

Original: To handle the large output space, wepropose a candidate filtering strategy to dynamically reduce the space, and refinethe teacher aggregation of PATE to avoid low agreement due to voting for a largenumber of candidates.
Revised: To handle the large output space, wepropose a candidate filtering strategy to dynamically reduce the space, and refinethe teacher aggregation of PATE to avoid low agreement due to voting for a largenumber of candidates.

Original: To reduce privacy losses, we design an efficient knowledgedistillation to decrease the frequency of querying teachers.
Revised: To reduce privacy losses, we design an efficient knowledgedistillation to decrease the frequency of querying teachers.

Original: The experiments verifythe effectiveness of SeqPATE in protecting both samples and sensitive phrases.
Revised: The experiments verifythe effectiveness of SeqPATE in protecting both samples and sensitive phrases.

Original:  Introduction
Revised:  Introduction

Original: Recent work showed that sensitive user information in the training corpora, such as addresses andnames, can be extracted from text generation models [5].
Revised: Recent work showed that sensitive user information in the training corpora, such as addresses andnames, can be extracted from text generation models [5].

Original: Providing privacy guarantees to thetraining corpora of text generation models becomes a critical problem.
Revised: Providing privacy guarantees to thetraining corpora of text generation models becomes a critical problem.

Original: Differential privacy (DP)provides provable guarantees against detecting individuals in datasets.
Revised: Differential privacy (DP)provides provable guarantees against detecting individuals in datasets.

Original: Deep learning models with
Revised: Deep learning models with

Original: DP guarantees ensure that the existence of a specific training sample cannot be detected.
Revised: DP guarantees ensure that the existence of a specific training sample cannot be detected.

Original: NoisySGD [39, 3, 1] is a popular DP algorithm for deep learning, which adds noise to the gradients.
Revised: NoisySGD [39, 3, 1] is a popular DP algorithm for deep learning, which adds noise to the gradients.

Original: PATE [29] is another type of DP learning algorithm that transfers knowledge from teachers trained onprivate data to a student model, where noises are added to the teacher predictions to satisfy DP.
Revised: PATE [29] is another type of DP learning algorithm that transfers knowledge from teachers trained onprivate data to a student model, where noises are added to the teacher predictions to satisfy DP.

Original: PATEis model-agnostic and its privacy cost derives from the knowledge distillation instead of the gradientsof the whole model in NoisySGD [39, 22].
Revised: PATEis model-agnostic and its privacy cost derives from the knowledge distillation instead of the gradientsof the whole model in NoisySGD [39, 22].

Original: Hence, noises required by PATE do not scale with modelsize.
Revised: Hence, noises required by PATE do not scale with modelsize.

Original: With this benefit, PATE has a great potential for being adapted to text generations, since largemodels (e.g., GPT-2 [33]) become the mainstream in text generation.
Revised: With this benefit, PATE has a great potential for being adapted to text generations, since largemodels (e.g., GPT-2 [33]) become the mainstream in text generation.

Original: However, NoisySGD and PATEare used to protect sample-level privacy [46, 22] but require a high privacy cost on protecting sensitivephrases in samples [20, 36].
Revised: However, NoisySGD and PATEare used to protect sample-level privacy [46, 22] but require a high privacy cost on protecting sensitivephrases in samples [20, 36].

Original: PATE, designed for classification tasks, is not customized for sequentialgeneration on a large output space (i.e., vocabulary), which is very common in text generation.
Revised: PATE, designed for classification tasks, is not customized for sequentialgeneration on a large output space (i.e., vocabulary), which is very common in text generation.

Original: In this paper, we propose SeqPATE, a DP learning algorithm for text generations to protect theprivacy of both samples and sensitive phrases in training corpora with a moderate privacy cost.
Revised: In this paper, we propose SeqPATE, a DP learning algorithm for text generations to protect theprivacy of both samples and sensitive phrases in training corpora with a moderate privacy cost.

Original: As anextension of PATE, SeqPATE employs a teacher-student framework: (i) a student model learns to generate texts on non-sensitive samples; and (ii) a number of teacher models, trained on sensitivetexts, supervise the student through the outputs of aggregated teachers.
Revised: As anextension of PATE, SeqPATE employs a teacher-student framework: (i) a student model learns to generate texts on non-sensitive samples; and (ii) a number of teacher models, trained on sensitivetexts, supervise the student through the outputs of aggregated teachers.

Original: This framework still facessome challenges in text generations.
Revised: This framework still facessome challenges in text generations.

Original: Firstly, it suffers from high costs of GPU memory and time.
Revised: Firstly, it suffers from high costs of GPU memory and time.

Original: To obtain sentence-level supervision for text generation, the model needs to roll out all teachers toproduce a sentence (i.e. all teachers must vote to generate a word, which is then used as the inputfor the next word prediction).
Revised: To obtain sentence-level supervision for text generation, the model needs to roll out all teachers toproduce a sentence (i.e. all teachers must vote to generate a word, which is then used as the inputfor the next word prediction).

Original: It results in a high inference cost with a large number of teachers (e.g.2 k teachers which are common in PATE).
Revised: It results in a high inference cost with a large number of teachers (e.g.2 k teachers which are common in PATE).

Original: Secondly, the large output space (i.e., vocabulary) in textgeneration leads to (i) low agreement rates among teachers and (ii) large noises required by DP, bothof which would significantly hurt the task performance.
Revised: Secondly, the large output space (i.e., vocabulary) in textgeneration leads to (i) low agreement rates among teachers and (ii) large noises required by DP, bothof which would significantly hurt the task performance.

Original: To address the challenges, we generate pseudo data using a pre-trained language model so thatteachers only need to provide token-level supervision given the pseudo inputs.
Revised: To address the challenges, we generate pseudo data using a pre-trained language model so thatteachers only need to provide token-level supervision given the pseudo inputs.

Original: To handle the largeoutput space and reduce the noise, we propose to dynamically filter the candidate words and selectonly words with large probabilities.
Revised: To handle the largeoutput space and reduce the noise, we propose to dynamically filter the candidate words and selectonly words with large probabilities.

Original: Also, we aggregate teachers’ outputs by interpolating their outputdistributions instead of voting with argmax predictions.
Revised: Also, we aggregate teachers’ outputs by interpolating their outputdistributions instead of voting with argmax predictions.

Original: To reduce the privacy losses, selectively applythe knowledge distillation to the generation step only if the student has trouble with the generation.
Revised: To reduce the privacy losses, selectively applythe knowledge distillation to the generation step only if the student has trouble with the generation.

Original: Most DP learning methods, including SeqPATE, prevent the existence of individual training samplesfrom being detected.
Revised: Most DP learning methods, including SeqPATE, prevent the existence of individual training samplesfrom being detected.

Original: SeqPATE has further advantages in protecting users’ secret phrases (e.g.,addresses) that may occur multiple times in the corpus.
Revised: SeqPATE has further advantages in protecting users’ secret phrases (e.g.,addresses) that may occur multiple times in the corpus.

Original: We evaluate our method on a sentencecompletion task.
Revised: We evaluate our method on a sentencecompletion task.

Original: The comparison with a recent NoisySGD baseline [22] on the AirDialog datasetshows that SeqPATE improves Bleu4 by 59% on protecting samples and by 391% on protecting users’secret phrases with a meaningful strength of privacy protection.
Revised: The comparison with a recent NoisySGD baseline [22] on the AirDialog datasetshows that SeqPATE improves Bleu4 by 59% on protecting samples and by 70% on protecting users’secret phrases with a meaningful strength of privacy protection.

Original: Our contribution is twofold: (i)
Revised: Our contribution is twofold: (i)

Original: We propose SeqPATE to achieve the protections on both samplelevel and phrase level and also provide theoretical analyses on the two levels.
Revised: We propose SeqPATE to achieve the protections on both samplelevel and phrase level and also provide theoretical analyses on the two levels.

Original: The experiments show
Revised: The experiments show

Original: SeqPATE’s advantages on the two levels compared to the baselines.
Revised: SeqPATE’s advantages on the two levels compared to the baselines.

Original: (ii)
Revised: (ii)

Original: We propose several strategiesfor SeqPATE to handle text generations with a sequence of classifications over large spaces.
Revised: We propose several strategiesfor SeqPATE to handle text generations with a sequence of classifications over large spaces.

Original:  Problem Setup
Revised:  Problem Setup

Original: Our goal is to achieve the privacy protection quantified by DP in the text generation: preventingattackers from inferring whether a sample or an n-gram appears in the training set.
Revised: Our goal is to achieve the privacy protection quantified by DP in the text generation: preventingattackers from inferring whether a sample or an n-gram appears in the training set.

Original: Our setting contains two types of textual datasets: (1) a private set D pri from a corpus with sensitiveinformation, (2) a public set D pub that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data.
Revised: Our setting contains two types of textual datasets: (1) a private set D pri from a corpus with sensitiveinformation, (2) a public set D pub that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data.

Original: We aim to protect the privacyon the private set and can ignore the privacy protection on the public set.
Revised: We aim to protect the privacyon the private set and can ignore the privacy protection on the public set.

Original: Our application, sentence completion, aims to complete the whole sentence given the prefix.
Revised: Our application, sentence completion, aims to complete the whole sentence given the prefix.

Original: Wetrain an autoregressive language model to achieve the completion.
Revised: Wetrain an autoregressive language model to achieve the completion.

Original: The public set D pub consists ofthe prefixes, which can hardly contain sensitive information.
Revised: The public set D pub consists ofthe prefixes, which can hardly contain sensitive information.

Original: The private set D pri consists of thewhole sentences.
Revised: The private set D pri consists of thewhole sentences.

Original: Such a setting fits some real-world text generation applications: in dialog systems,the training samples from online services consist of questions and responses.
Revised: Such a setting fits some real-world text generation applications: in dialog systems,the training samples from online services consist of questions and responses.

Original: The questions fromcustomer service staff or service robots can be public and the response from users carrying individualinformation should be private.
Revised: The questions fromcustomer service staff or service robots can be public and the response from users carrying individualinformation should be private.

Original: In writing assistants where a sample has an input prompt and an essaybody, the prompt can be public but the body written by authors is private.
Revised: In writing assistants where a sample has an input prompt and an essaybody, the prompt can be public but the body written by authors is private.

Original:  Background on DP and PATE
Revised:  Background on DP and PATE

Original: Definition 3.1.
Revised: Definition 3.1.

Original: [Differential privacy (DP) [11, 12]]
Revised: [Differential privacy (DP) [11, 12]]

Original: For any two neighboring datasets D , D ′ (differ inonly one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,
Revised: For any two neighboring datasets D , D ′ (differ inonly one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,

Original: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, where S ⊆ Y , ε > 0 , δ ≥ 0 .
Revised: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, where S ⊆ Y , ε > 0 , δ ≥ 0 .

Original: As the definition, DP is a quantifiable definition of privacy that provides guarantees on identificationsof individual data (prevent an adversary from inferring whether the input is D or D ′ ).
Revised: As the definition, DP is a quantifiable definition of privacy that provides guarantees on identificationsof individual data (prevent an adversary from inferring whether the input is D or D ′ ).

Original: ML modelswith DP guarantee ensure that each individual training sample has a degree of plausible deniability ,i.e., the trained model is just as likely as to be trained on an alternative dataset without that sample.
Revised: ML modelswith DP guarantee ensure that each individual training sample has a degree of plausible deniability ,i.e., the trained model is just as likely as to be trained on an alternative dataset without that sample.

Original: PATE [29], designed for classification tasks, takes advantage of an unlabeled public dataset D puband also trains on a labeled private set D pri in a semi-supervised scenario.
Revised: PATE [29], designed for classification tasks, takes advantage of an unlabeled public dataset D puband also trains on a labeled private set D pri in a semi-supervised scenario.

Original: PATE achieves DP via a
Revised: PATE achieves DP via a

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: Private Data +Noise
Revised: Private Data +Noise

Original: Teacher
Revised: Teacher

Original: Teacher Teacher
Revised: Teacher Teacher

Original: Student
Revised: Student

Original: NLL loss
Revised: NLL loss

Original: Data Data
Revised: Data Data

Original: Data
Revised: Data

Original: Cats sit on the Cats sit on the
Revised: Cats sit on the Cats sit on the

Original: Cats sit on the
Revised: Cats sit on the

Original: Candidate Filtering |Vocabulary|
Revised: Candidate Filtering |Vocabulary|

Original: Sit 0.42 Sat 0.33 Lay 0.
Revised: Sit 0.42 Sat 0.33 Lay 0.

Original: Are 0.08 ...
Revised: Are 0.08 ...

Original: Alaia 1eRuth 1eTop-K
Revised: Alaia 1eRuth 1eTop-K

Original: Aggregate
Revised: Aggregate

Original: Teacher Inference sit on the mats Teacher

Revised: Teacher Inference sit on the mats Teacher

Original: Teacher

Revised: Teacher

Original: Teacher
Revised: Teacher

Original: Cats sit
Revised: Cats sit

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: [Figure]
Revised: [Figure]

Original: supervisions
Revised: supervisions

Original: [Figure]
Revised: [Figure]

Original: supervisions
Revised: supervisions

Original: [Figure]
Revised: [Figure]

Original: Student Training
Revised: Student Training

Original: [Figure]
Revised: [Figure]

Original: Distillation teacher-student framework with M teacher models and a student model, where the student learnsfrom the private set via knowledge distillation through teachers.
Revised: Distillation teacher-student framework with M teacher models and a student model, where the student learnsfrom the private set via knowledge distillation through teachers.

Original: PATE has three parts: (i)
Revised: PATE has three parts: (i)

Original: Theteacher models are trained on the private set D pri , which is shuffled and divided into M disjointsubsets.
Revised: Theteacher models are trained on the private set D pri , which is shuffled and divided into M disjointsubsets.

Original: Each teacher is trained on one subset.
Revised: Each teacher is trained on one subset.

Original: (ii) Teacher aggregation merges teachers’ outputs.
Revised: (ii) Teacher aggregation merges teachers’ outputs.

Original: Given the trained teachers, we employ each teacher to provide supervision on the student’s unlabeledpublic set D pub .
Revised: Given the trained teachers, we employ each teacher to provide supervision on the student’s unlabeledpublic set D pub .

Original: We use noised majority votes from teachers as labels to supervise the student.
Revised: We use noised majority votes from teachers as labels to supervise the student.

Original: (iii) Astudent model is trained on the public set D pub with the supervision of the aggregated teachers.
Revised: (iii) Astudent model is trained on the public set D pub with the supervision of the aggregated teachers.

Original:  Approach
Revised:  Approach

Original: Fig.
Revised: Fig.

Original: 1 shows an overview of SeqPATE.
Revised: 1 shows an overview of SeqPATE.

Original: Given the public prefix (e.g., “Cats sit”), we first obtain thepseudo inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained languagemodel (Sec. 4.1).
Revised: Given the public prefix (e.g., “Cats sit”), we first obtain thepseudo inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained languagemodel (Sec. 4.1).

Original: At each word, we then aggregate the teachers’ prediction of the next word assupervision for training the student model (Sec. 4.2).
Revised: At each word, we then aggregate the teachers’ prediction of the next word assupervision for training the student model (Sec. 4.2).

Original: To reduce the noise required by DP for alarge output space of the size of the vocabulary, we reduce the output space by dynamically filteringunimportant words.
Revised: To reduce the noise required by DP for alarge output space of the size of the vocabulary, we reduce the output space by dynamically filteringunimportant words.

Original: To reduce the number of teacher queries that incur privacy losses, we proposean efficient knowledge distillation strategy that only queries teacher labels on uncertain examples(Sec.
Revised: To reduce the number of teacher queries that incur privacy losses, we proposean efficient knowledge distillation strategy that only queries teacher labels on uncertain examples(Sec.

Original: We show the training algorithm in App.
Revised: We show the training algorithm in App.

Original: B and a running example in App.
Revised: B and a running example in App.

Original: L.
Revised: L.

Original: 4.1 Pseudo Input Generation
Revised: 4.1 Pseudo Input Generation

Original: Conventional text generation models generate words sequentially from left to right.
Revised: Conventional text generation models generate words sequentially from left to right.

Original: Thus naivelyapplying PATE to text generation requires rolling out all teachers word by word, i.e. iterativelysampling the next word from the aggregated teacher prediction.
Revised: Thus naivelyapplying PATE to text generation requires rolling out all teachers word by word, i.e. iterativelysampling the next word from the aggregated teacher prediction.

Original: This is costly in both computation(running inference for hundreds of teacher models) and privacy costs (querying teachers at everystep).
Revised: This is costly in both computation(running inference for hundreds of teacher models) and privacy costs (querying teachers at everystep).

Original: To tackle this challenge, we use a pre-trained language model to complete the public prefixesinto pseudo sentences, thus we only need to query teachers on the next word given a (pseudo) context.
Revised: To tackle this challenge, we use a pre-trained language model to complete the public prefixesinto pseudo sentences, thus we only need to query teachers on the next word given a (pseudo) context.

Original: 4.2 Teacher Aggregation
Revised: 4.2 Teacher Aggregation

Original: PATE aggregates teachers’ predictions by majority vote.
Revised: PATE aggregates teachers’ predictions by majority vote.

Original: While it works for classification problemswith a relatively small number of classes, the output space of text generation contains all words in thevocabulary.
Revised: While it works for classification problemswith a relatively small number of classes, the output space of text generation contains all words in thevocabulary.

Original: As a result, the number of votes for each candidate word may be very low without a clearwinner.
Revised: As a result, the number of votes for each candidate word may be very low without a clearwinner.

Original: For example, multiple candidates may tie for the top-1 prediction.
Revised: For example, multiple candidates may tie for the top-1 prediction.

Original: Inspired by Chen et al.
Revised: Inspired by Chen et al.

Original: [8, 15], we aggregate teachers’ results by averaging their output distributions.
Revised: [8, 15], we aggregate teachers’ results by averaging their output distributions.

Original: We first train M teacher models on disjoint subsets of the private data.
Revised: We first train M teacher models on disjoint subsets of the private data.

Original: To produce the aggregatednext word distribution given a context c , we average the teachers’ output distributions, add calibratednoises, and then renormalize the results into a proper distribution.
Revised: To produce the aggregatednext word distribution given a context c , we average the teachers’ output distributions, add calibratednoises, and then renormalize the results into a proper distribution.

Original: Following Papernot et al.
Revised: Following Papernot et al.

Original: [30],we apply the Gaussian mechanism.
Revised: [30],we apply the Gaussian mechanism.

Original: Formally, let p mϕ ( · | c ) be the m -th teacher’s prediction.
Revised: Formally, let p mϕ ( · | c ) be the m -th teacher’s prediction.

Original: Theaggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)
Revised: Theaggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)

Original: Mm =

Revised: Mm =

Original: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 1 where the Gaussian noiseis added to the aggregated output distribution.
Revised: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 1 where the Gaussian noiseis added to the aggregated output distribution.

Original: 4.3 Training of the Student Model
Revised: 4.3 Training of the Student Model

Original: The student model is trained on the public pseudo data and also supervised by the aggregated teachers.
Revised: The student model is trained on the public pseudo data and also supervised by the aggregated teachers.

Original: Training objectives.
Revised: Training objectives.

Original: The student model is a language model that predicts the next word givencontexts.
Revised: The student model is a language model that predicts the next word givencontexts.

Original: Given contexts from (public) pseudo data autocompleted by a pre-trained language model(GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in thepseudo data (i.e. pseudo label).
Revised: Given contexts from (public) pseudo data autocompleted by a pre-trained language model(GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in thepseudo data (i.e. pseudo label).

Original: The pseudo data act as a prior for the student given that the numberof teacher queries is limited due to privacy concerns.
Revised: The pseudo data act as a prior for the student given that the numberof teacher queries is limited due to privacy concerns.

Original: The student’s loss function has two parts:
Revised: The student’s loss function has two parts:

Original: • L teacher denotes the loss with respect to the teachers’ supervision.
Revised: • L teacher denotes the loss with respect to the teachers’ supervision.

Original: Note that
Revised: Note that

Original: the
Revised: the

Original: teachers’ aggregatedoutput
Revised: teachers’ aggregatedoutput

Original: is a distribution over words.
Revised: is a distribution over words.

Original: Therefore, we minimize the forward KL divergence between theaggregated teacher distribution p
Revised: Therefore, we minimize the forward KL divergence between theaggregated teacher distribution p

Original: agg
Revised: agg

Original: and the student output distribution p θ :
Revised: and the student output distribution p θ :

Original: [Equation]
Revised: [Equation]

Original: • L pseudo denotes the loss with respect to the pseudo labels w from ˜ D pub (i.e. next words generated by a
Revised: • L pseudo denotes the loss with respect to the pseudo labels w from ˜ D pub (i.e. next words generated by a

Original: generic language model).
Revised: generic language model).

Original: Similar to standard language modeling, we use the negative log-likelihood:
Revised: Similar to standard language modeling, we use the negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 4 shows the complete loss.
Revised: 4 shows the complete loss.

Original: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)
Revised: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)

Original: [Equation]
Revised: [Equation]

Original: Reducing the output space via candidate filtering.
Revised: Reducing the output space via candidate filtering.

Original: The high-dimensionality of the output of textgeneration models results in large noise (which is added to each coordinate).
Revised: The high-dimensionality of the output of textgeneration models results in large noise (which is added to each coordinate).

Original: To reduce the outputdimension (hence the amount of noise), we filter words on the tail of the distribution of the studentmodel (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and thestudent output distribution over the rest words.
Revised: To reduce the outputdimension (hence the amount of noise), we filter words on the tail of the distribution of the studentmodel (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and thestudent output distribution over the rest words.

Original: Note that the candidate filtering is based on the student’s outputs on public or already released inputs,thus it does not affect the privacy guarantee.
Revised: Note that the candidate filtering is based on the student’s outputs on public or already released inputs,thus it does not affect the privacy guarantee.

Original: This choice improves the privacy-utility tradeoff byadaptively allocating the available privacy budgets to release the information more helpful to the task.
Revised: This choice improves the privacy-utility tradeoff byadaptively allocating the available privacy budgets to release the information more helpful to the task.

Original: We experiment with two filtering strategies: top- k and top- p .
Revised: We experiment with two filtering strategies: top- k and top- p .

Original: In top- k filtering, we retain only thetop- k most likely candidates and filter the rest according to the student model.
Revised: In top- k filtering, we retain only thetop- k most likely candidates and filter the rest according to the student model.

Original: In top- p filtering [16],k is chosen dynamically such that the top- k words are the minimum set whose cumulative probabilityis at least p .
Revised: In top- p filtering [16],k is chosen dynamically such that the top- k words are the minimum set whose cumulative probabilityis at least p .

Original: The strategy seldom loses good candidates, because the student usually does well ontop- k predictions since the beginning of the training.
Revised: The strategy seldom loses good candidates, because the student usually does well ontop- k predictions since the beginning of the training.

Original: Reducing the number of teacher queries via efficient knowledge distillation.
Revised: Reducing the number of teacher queries via efficient knowledge distillation.

Original: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.
Revised: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.

Original: Therefore,we obtain the teacher supervision only on “hard” examples in student’s training.
Revised: Therefore,we obtain the teacher supervision only on “hard” examples in student’s training.

Original: Note that the studentis trained on both the pseudo data and local supervision from the teachers.
Revised: Note that the studentis trained on both the pseudo data and local supervision from the teachers.

Original: We consider an exampleto be hard if the student cannot imitate the pseudo label, in which case distilling knowledge from theteachers that are trained on large private data is helpful.
Revised: We consider an exampleto be hard if the student cannot imitate the pseudo label, in which case distilling knowledge from theteachers that are trained on large private data is helpful.

Original: Concretely, we query the teachers only when the rank of the pseudo label is below a certain thresholdamong words ordered by descending probabilities under the student model.
Revised: Concretely, we query the teachers only when the rank of the pseudo label is below a certain thresholdamong words ordered by descending probabilities under the student model.

Original: If we query the teachers,the student is trained via complete loss L ( p agg , ˜ D pub )
Revised: If we query the teachers,the student is trained via complete loss L ( p agg , ˜ D pub )

Original: (Eq. 4); otherwise, the student is trained via the
Revised: (Eq. 4); otherwise, the student is trained via the

Original: L pseudo (Eq. 3).
Revised: L pseudo (Eq. 3).

Original: We note that the selection of tokens relies only on the student and is independent ofthe teachers; thus the selection does not cause any additional privacy loss.
Revised: We note that the selection of tokens relies only on the student and is independent ofthe teachers; thus the selection does not cause any additional privacy loss.

Original:  Privacy Analyses
Revised:  Privacy Analyses

Original: 5.1 Preliminary of Differential Privacy
Revised: 5.1 Preliminary of Differential Privacy

Original: Lemma 5.1 (Analytical Gaussian mechanism [2]) .
Revised: Lemma 5.1 (Analytical Gaussian mechanism [2]) .

Original: For a numeric query f : X n → R d over adataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies( ε, δ ( ε ))
Revised: For a numeric query f : X n → R d over adataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) =
Revised: -DP for all ε ≥ 0 and δ ( ε ) =

Original: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .
Revised: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .

Original: where ∆ := ∆ ( f )=max D∼D ′ ∥
Revised: where ∆ := ∆ ( f )=max D∼D ′ ∥

Original: f ( D ) − f ( D ′ )
Revised: f ( D ) − f ( D ′ )

Original: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .
Revised: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .

Original: We can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.

Original: Lemma 5.2 (Composition of Gaussian mechanisms [10]) .
Revised: Lemma 5.2 (Composition of Gaussian mechanisms [10]) .

Original: The adaptive composition of a sequenceof Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .
Revised: The adaptive composition of a sequenceof Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .

Original: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .
Revised: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .

Original: satisfies( ε, δ ( ε ))
Revised: satisfies( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noisemultiplier σ/ ∆ =
Revised: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noisemultiplier σ/ ∆ =

Original: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .
Revised: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .

Original: Specifically, the adaptive composition of a k identical Gaussian mechanism with noise multiplier σsatisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .
Revised: Specifically, the adaptive composition of a k identical Gaussian mechanism with noise multiplier σsatisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .

Original: By fixing k and ε , we can calibrate the noises by choosing an appropriate σ in our model.
Revised: By fixing k and ε , we can calibrate the noises by choosing an appropriate σ in our model.

Original: 5.2 Differential Privacy for Language Models at the Sample Level
Revised: 5.2 Differential Privacy for Language Models at the Sample Level

Original: Recall that we partition the private dataset to M disjoint subsets, and train each teacher model onone of the subsets.
Revised: Recall that we partition the private dataset to M disjoint subsets, and train each teacher model onone of the subsets.

Original: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -thteacher model given some context, where |V| is the vocabulary size.
Revised: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -thteacher model given some context, where |V| is the vocabulary size.

Original: The aggregation functionf ( D ) :
Revised: The aggregation functionf ( D ) :

Original: = (cid:80) Mi
Revised: = (cid:80) Mi

Original: =1 x i is the sum of the probability distributions predicted over all the teachers.
Revised: =1 x i is the sum of the probability distributions predicted over all the teachers.

Original: Since thedatasets are disjoint and have no duplicate samples, changing one sample only affects one teachermodel.
Revised: Since thedatasets are disjoint and have no duplicate samples, changing one sample only affects one teachermodel.

Original: For neighboring datasets D , D ′ , let j denote the index of each teacher model, then theprobability distributions x j and x ′ j (deriving from D and D ′ respectively) are different.
Revised: For neighboring datasets D , D ′ , let j denote the index of each teacher model, then theprobability distributions x j and x ′ j (deriving from D and D ′ respectively) are different.

Original: Then, thesensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),
Revised: Then, thesensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),

Original: [Equation]
Revised: [Equation]

Original: Then, adding the noises calibrated in Lemma 5.2 to each coordinate is sufficient to preserve ( ε ,δ ( ε ) )
Revised: Then, adding the noises calibrated in Lemma 5.2 to each coordinate is sufficient to preserve ( ε ,δ ( ε ) )

Original: -DP for f ( D ) .
Revised: -DP for f ( D ) .

Original: Finally, when we extract top- k coordinates of f ( D ) (Sec. 4.3), the privacyguarantee also holds according to the post-processing property [12].
Revised: Finally, when we extract top- k coordinates of f ( D ) (Sec. 4.3), the privacyguarantee also holds according to the post-processing property [12].

Original: Intuitively, the information aboutwhether a sample occurs in the private set is protected, and the protection satisfies ( ε , δ ( ε ) )
Revised: Intuitively, the information aboutwhether a sample occurs in the private set is protected, and the protection satisfies ( ε , δ ( ε ) )

Original: -DP.
Revised: -DP.

Revised: Theconnection between DP and SeqPATE is that: (1) by satisfying DP theory, SeqPATE quantifies thestrength of protection; (2) DP requires SeqPATE to add noise to the knowledge distillation (Sec. 4.2).

Original: 5.3 Differential Privacy of Users’ Secret Phrases
Revised: 5.3 Differential Privacy of Users’ Secret Phrases

Original: The above analysis shows that we can protect the privacy of each sample (i.e., one occurrence of asentence).
Revised: The above analysis shows that we can protect the privacy of each sample (i.e., one occurrence of asentence).

Original: However, in practical text generations, we may want to protect all occurrences of somesecret phrases specific to a user (e.g., names and addresses).
Revised: However, in practical text generations, we may want to protect all occurrences of somesecret phrases specific to a user (e.g., names and addresses).

Original: 3 Consider a secret phrase s occurring n stimes ( n s ≥ 1 ) in the private set.
Revised: 3 Consider a secret phrase s occurring n stimes ( n s ≥ 1 ) in the private set.

Original: According to group privacy [12], the protection on phrase s satisfies( nε, e nε − 1 e ε − 1 δ ) -DP
Revised: According to group privacy [12], the protection on phrase s satisfies( nε, e nε − 1 e ε − 1 δ ) -DP

Original: [20, 36], where the privacy loss scales linearly with the number of occurrences of s .
Revised: [20, 36], where the privacy loss scales linearly with the number of occurrences of s(We discuss and analyze a better strategy to reduce the privacy loss of baselines in App. N).

Original: While naively applying a DP algorithm requires larger noise to protect user phrases that may occurmultiple times, SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or afew teachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.
Revised: While naively applying a DP algorithm requires larger noise to protect user phrases that may occurmultiple times, SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or afew teachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.

Original: We denote ˜ n s as the number of teachers whose data contain the phrase s .
Revised: We denote ˜ n s as the number of teachers whose data contain the phrase s .

Original: Since adding or removingthe phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s (See App.
Revised: Since adding or removing the phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s(See App.

Original: D for detailed deductions).
Revised: D for detailed deductions).

Original: In this way, the strength of privacy guarantee on secret phrasesis similar to that we have derived for sample-level DP.
Revised: In this way, the strength of privacy guarantee on secret phrasesis similar to that we have derived for sample-level DP.

Original: The exact ( ε, δ ( ε, ˜ n s ))
Revised: The exact ( ε, δ ( ε, ˜ n s ))

Original: -DP for the phrase s canbe obtained according to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =
Revised: -DP for the phrase s canbe obtained according to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =

Original: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .
Revised: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .

Original: Unlike other generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacyloss on user phrases by careful partitioning of the private data.
Revised: Unlike other generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacyloss on user phrases by careful partitioning of the private data.

Original: 5.4 How does DP prevent memorization?
Revised: 5.4 How does DP prevent memorization?

Original: In practice, the privacy of the language model is sometimes interpreted as not generating a secretphrase in the training data as-is during inference.
Revised: In practice, the privacy of the language model is sometimes interpreted as not generating a secretphrase in the training data as-is during inference.

Original: Thus, one may wonder how DP prevents suchunintended memorization of the training data.
Revised: Thus, one may wonder how DP prevents suchunintended memorization of the training data.

Original: We remark that the protection against memorizationfollows the definition of DP.
Revised: We remark that the protection against memorizationfollows the definition of DP.

Original: Consider the attack by Carlini et al.
Revised: Consider the attack by Carlini et al.

Original: [5], which uses a language model topredict a secret phrase s given a prefix.
Revised: [5], which uses a language model topredict a secret phrase s given a prefix.

Original: By the closure to post-processing [12], the prediction alsosatisfies DP.
Revised: By the closure to post-processing [12], the prediction alsosatisfies DP.

Original: We denote S as the undesirable event where the model generates the phrase s verbatim.
Revised: We denote S as the undesirable event where the model generates the phrase s verbatim.

Original: The definition of DP implies that the probability of S to happen when s is in the training data is atmost e ε larger than the probability of an alternative model trained without s in the data.
Revised: The definition of DP implies that the probability of S to happen when s is in the training data is atmost e ε larger than the probability of an alternative model trained without s in the data.

Original: The chancesfor the latter model to generate a sequence containing s are astronomically small; thus DP impliesthat the probability of S under the former model needs to be just as small.
Revised: The chancesfor the latter model to generate a sequence containing s are astronomically small; thus DP impliesthat the probability of S under the former model needs to be just as small.

Original:  Experiments
Revised:  Experiments

Original: 6.1 Experimental Settings
Revised: 6.1 Experimental Settings

Original: Datasets.
Revised: Datasets.

Original: We evaluate our model on two datasets. AirDialog [44] consists of 1M utterances fromcustomer servicedialog on flight booking; Europarl_v6 consists of 2MEnglish sentences collectedfrom European Parliament.
Revised: We use two datasets: AirDialog [44] has 1M utterances from customer service on flightbooking; Europarl_v6 has 2M sentences collected from European Parliament.

Original: 4 (See details about datasets in App. E.)
Revised: (Details in App. E.)

Original: Baselines.
Revised: Baselines.

Original: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on theprivate data with calibrated noise on clipped gradients [1, 20] and further trained on public set D pubwithout protection; (2) based on NoisySGD , NoisySGD+GC [22] applies a ghost clipping whichenables large batch size with memory saving techniques.
Revised: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on theprivate data with calibrated noise on clipped gradients [1, 20] and further trained on public set D pubwithout protection; (2) based on NoisySGD , NoisySGD+GC [22] applies a ghost clipping whichenables large batch size with memory saving techniques.

Original: Additionally, we use two non-DP methodsas reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the publicpre-trained GPT-2 model Pub-GPT without access to private data.
Revised: Additionally, we use two non-DP methodsas reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the publicpre-trained GPT-2 model Pub-GPT without access to private data.

Original: For all methods, we can optionallyfine-tune on the generated pseudo data as a warm-up, and the operation is denoted as + ˜ D pub .
Revised: For all methods, we can optionallyfine-tune on the generated pseudo data as a warm-up, and the operation is denoted as + ˜ D pub .

Original: Hyper-parameters and implementation details.
Revised: Implementation details.

Original: All models are fine-tuned from the (public) pretrained GPT-2 model [33].
Revised: All models are fine-tuned from the (public) pre-trained GPT-2 model [33].

Original: The batch size is 32 for all comparing methods except the GC [22] (GC[22] requires 2048).
Revised: The batch size is 32 for all comparing methods except the GC [22] (GC [22] requires 2048).

Original: We use Adam [21] and adjust the initial learning rate with a range of 10 − 3 to10 − 6 for all methods.
Revised: We use Adam [21] and adjust the initial learning rate with a range of 10 − 3 to 10 − 6 for all methods.

Original: The δ mentioned in Sec.
Revised: The δmentioned in Sec.

Original: 5 for all DP methods is 10 − 6 .
Revised: 5 for all DP methods is 10 − 6 .

Original: For SeqPATE, before training the student model with teacher supervision, we first fine-tune it on thepseudo public data ˜ D pub as a warm-up.
Revised: For SeqPATE, before training the student model withteacher supervision, we first fine-tune it on the pseudo public data ˜ D pub as a warm-up.

Original: The coefficient λ that balances supervision for the teacherand the pseudo data (Eq. 4) is set to 20, where we have tuned it on the validation set of the publicpseudo data.
Revised: The coefficientλ that balances supervision for the teacher and the pseudo data (Eq. 4) is set to 20, where we havetuned it on the validation set of the public pseudo data.

Original: The default number of teacher models is 2 k , where our model works well according tothe experiments in App.
Revised: The default number of teacher models is 2 k ,where our model works well according to the experiments in App.

Original: H. We designed some strategies 5 to reduce the usage of the memory andhard-disk (Seemore details about the strategies and the computational cost in App. I).We can run
Revised: H. We designed some strategies 4to reduce the use of the memory and disk (See strategies and the computational cost in App. I).

Original:  SeqPATE with 2 k teachers on a single GPU within 3 days.
Revised: Werun SeqPATE with 2 k teachers on a single GPU within 3 days.

Original: (Seemore hyperparameters in App. G.)
Revised: (See hyperparameters in App. G.)

Original: Metrics.
Revised: Metrics.

Original: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [31].
Revised: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [31].

Original: 6.2 Overall Performance
Revised: 6.2 Overall Performance

Original: Protection at the sample level.
Revised: Protection at the sample level.

Original: Tab. 1 show the performance on the two datasets (See more metrics(i.e.
Revised: Tab. 1 show the performance on the two datasets (See more metrics(i.e.

Original: , Bleu-1 and Bleu-2) in App. K).
Revised: , Bleu-1 and Bleu-2) in App. K).

Original: Among the non-DP baselines, Pri-GPT acts as an upper boundon the performance since it can fully utilize the private set by discarding privacy protection.
Revised: Among the non-DP baselines, Pri-GPT acts as an upper boundon the performance since it can fully utilize the private set by discarding privacy protection.

Original: PubGPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additionalablation study on the pseudo data in App.
Revised: PubGPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additionalablation study on the pseudo data in App.

Original: J also verifies this).
Revised: J also verifies this).

Original: NoisySGD+GC+
Revised: NoisySGD+GC+

Original: ˜
Revised: ˜

Original: D pub surpasses the [Table] above two methods since it uses a much larger batch size (2048 vs 32) than NoisySGD.
Revised: D pub surpasses theabove two methods since it uses a much larger batch size (2048 vs 32) than NoisySGD.

Original: Our method,
Revised: Our method,

Original: SeqPATE, significantly outperforms NoisySGD+GC+ ˜
Revised: SeqPATE, significantly outperforms NoisySGD+GC+ ˜

Original: D pub (+59% in Bleu4 on AirDialog and +7.0%in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε . 
Revised: D pub (+59% in Bleu4 on AirDialog and +7.0%in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε . [Table]

Original: Protection on the user’s secret phrases.
Revised: Protection on the user’s secret phrases.

Original: We evaluate our method for privacy protection of secretphrases mentioned in Sec 5.3.
Revised: We evaluate our method for privacy protection of secretphrases mentioned in Sec 5.3.

Original: The key step is to partition the data such that each phrase only occursin the training data of very few teachers, which is straightforward given the user ID associated withthe private data.
Revised: The key step is to partition the data such that each phrase only occursin the training data of very few teachers, which is straightforward given the user ID associated withthe private data.

Original: In general, SeqPATE works with any set of secret phrases.
Revised: In general, SeqPATE works with any set of secret phrases.

Original: In our experiments, weconsider a user’s full name as their secret phrase since it can be easily recognized from the data.
Revised: In our experiments, weconsider a user’s full name as their secret phrase since it can be easily recognized from the data.

Original: Wepartition AirDialog’s private data according to the accompanying user IDs.
Revised: Wepartition AirDialog’s private data according to the accompanying user IDs.

Original: As a result, there are96.6% users whose data are assigned to a single teacher (details about the data partition in App. F). 
Revised: As a result, there are96.6% users whose data are assigned to a single teacher (details about the data partition in App. F). PPL ↓ Bleu [Table]

Revised: ↑

Revised: Bleu

Original: [Table]
Revised: [Table]

Revised: 4 ↑

Revised: NoisySGD+GC+ ˜

Original:  As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.
Revised: D pub 16. [Table] As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.

Original: In Tab.
Revised: In Tab.

Original: 2,we see that NoisySGD+GC+ ˜
Revised: 2,we see that NoisySGD+GC+ ˜

Original: D pub needs a large noise to achieve a satisfactory level of protection onphrases, because ε increases linearly with the phrase frequency (group privacy [12]).
Revised: D pub needs a large noise to achieve a satisfactory level of protection onphrases, because ε increases linearly with the phrase frequency (group privacy [12]).

Revised: “Batching users” indicates partitioning data into batches to

Revised: , which helps NoisySGD protect in

Original: For SeqPATE,the number of teachers trained ondata containing the phrase ˜ n s is close to 1 on average after ourpartition.
Revised: For SeqPATE, the number of teachers trained on the phrase ˜ n s is close to 1 on average after our partition.

Original: Thus, SeqPATE provides the same level of protection on users’ secret phrases with a smallernoise and thus achieves a better performance (+431% and +565% in Bleu4)
Revised: Thus, SeqPATE provides the same level ofprotection on users’ secret phrases with a smaller noise and thus achieves a better performance (+70%

Original: (See details aboutthelevel of protection onusers’ secret phrasesin App. F).
Revised: Bleu4) ( about on secret phrases

Original: [Figure]
Revised: [Figure]

Original: Privacy-utility tradeoff.
Revised: Privacy-utility tradeoff.

Original: In Fig.
Revised: In Fig.

Original: 2, we show the private-utility tradeoff curve of all DP algorithms.
Revised: 2, we show the private-utility tradeoff curve of all DP algorithms.

Original: Typically, DP with ε ∈ [0 . 1 , 5] is considered to provide a meaningful protection [42].
Revised: Typically, DP with ε ∈ [0 . 1 , 10] is considered to provide a meaningful protection [42].

Original: We observethat SeqPATE outperforms NoisySGD and NoisySGD+GC+
Revised: We observethat SeqPATE outperforms NoisySGD and NoisySGD+GC+

Original: ˜
Revised: ˜

Original: D pub in this range.
Revised: D pub in this range.

Original: However, SeqPATEdoes not work better than the two methods when ε > 10 .
Revised: However, SeqPATEdoes not work better than the two methods when ε > 10 .

Original: The reason is that NoisySGD+GC+ ˜
Revised: The reason is that NoisySGD+GC+ ˜

Original: D pubapproaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).
Revised: D pubapproaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).

Original: However, SeqPATEwith an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: theteachers cannot completely transfer knowledge from the private data to the student.
Revised: However, SeqPATEwith an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: theteachers cannot completely transfer knowledge from the private data to the student.

Original: Therefore, wesuggest using SeqPATE if strong privacy protection is desirable.
Revised: Therefore, wesuggest using SeqPATE if strong privacy protection is desirable.

Original: [Table]
Revised: [Table]

Original: 6.3 Ablation Studies
Revised: 6.3 Ablation Studies

Original: There are several design choices in SeqPATE
Revised: There are several design choices in SeqPATE

Original: and we study the importance of each of them.
Revised: and we study the importance of each of them.

Original: In
Revised: In

Original: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachersby voting instead of averaging their output distributions thecross- similar (although notequivalent) to the original the difference is that PATE needs to roll out all teachers 
Revised: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachersby voting instead of averaging their output distributions; (2) − KL: training the student using thecross-entropy loss with respect to teachers’ top-1 prediction instead of KL divergence; (3) −L pseudo :not learning from the pseudo label (Eq. 3); (4) − Effi KD: querying teachers on all samples withoutselection; (5) − Gaussian: using the Laplace mechanism as the original PATE algorithm instead of the Gaussian mechanism; and (6) − All: using none of the above strategies, which is similar (although notequivalent) to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1)).

Original: Aggregating the teachers by voting and training with KL loss are the most important strategies for
Revised: Aggregating the teachers by voting and training with KL loss are the most important strategies for

Original: SeqPATE.
Revised: SeqPATE.

Original: The poor performance on − Merge_P shows that voting is not suitable for text generation.
Revised: The poor performance on − Merge_P shows that voting is not suitable for text generation.

Original: The reason is that voting over a large output space leads to low agreement rates.
Revised: The reason is that voting over a large output space leads to low agreement rates.

Original: The results show thatthe L pseudo loss makes little contribution to SeqPATE.
Revised: The results show thatthe L pseudo loss makes little contribution to SeqPATE.

Original: The reason is that we have pre-trained on thestudent’s training set via L pseudo before the student’s training.
Revised: The reason is that we have pre-trained on thestudent’s training set via L pseudo before the student’s training.

Original: The promotion caused by the efficientknowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows thatthe “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from thisstrategy.
Revised: The promotion caused by the efficientknowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows thatthe “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from thisstrategy.

Original: This is because the “clever” student can dramatically save the privacy cost and transfer itto where it would benefit the student most.
Revised: This is because the “clever” student can dramatically save the privacy cost and transfer itto where it would benefit the student most.

Original: The poor performance of − All verifies that the original
Revised: The poor performance of − All verifies that the original

Original: PATE is not suitable for the text generation.
Revised: PATE is not suitable for the text generation.

Original: [Table]
Revised: [Table]

Original: 6.4 Analyses on Candidate Filtering and Teacher Numbers
Revised: 6.4 Analyses on Candidate Filtering and Teacher Numbers

Original: To analyze candidate filtering with different filtering strategies, we conduct the experiments on top- pand the top- k filtering.
Revised: To analyze candidate filtering with different filtering strategies, we conduct the experiments on top- pand the top- k filtering.

Original: As shown in Tab. 4, our full model employs the top- p filtering (the thresholdp is 0 .
Revised: As shown in Tab. 4, our full model employs the top- p filtering (the thresholdp is 0 .

Original: 95 ) surpasses most variants with manually chosen k .
Revised: 95 ) surpasses most variants with manually chosen k .

Original: Top-
Revised: Top-

Original: k filtering ( k = 50 or 100) alsoworks well.
Revised: k filtering ( k = 50 or 100) alsoworks well.

Original: The filtering with a too-small k ( k = 1 or k = 10 ) implies discarding too much usefulinformation from the supervision ( k = 1 is different to − KL in Tab. 3, which uses the Top-1 ofteachers’ results).
Revised: The filtering with a too-small k ( k = 1 or k = 10 ) implies discarding too much usefulinformation from the supervision ( k = 1 is different to − KL in Tab. 3, which uses the Top-1 ofteachers’ results).

Original: Filtering with oversize k results in unnecessarily large noises.
Revised: Filtering with oversize k results in unnecessarily large noises.

Original: The candidates with very small probabilities should be filtered during the generation; however, random noises mayincrease their probabilities, so models may generate those words that are misled by the noise.
Revised: The candidates with very small probabilities should be filtered during the generation; however, random noises mayincrease their probabilities, so models may generate those words that are misled by the noise.

Original: The results in App.
Revised: The results in App.

Original: H show that more teachers lead to better results when the teacher numbers are ina range of 1 ∼ 2 k .
Revised: H show that more teachers lead to better results when the teacher numbers are ina range of 1 ∼ 2 k .

Original: This is because the noise assigned to each teacher drops linearly as the teachernumbers go up.
Revised: This is because the noise assigned to each teacher drops linearly as the teachernumbers go up.

Original: Note that SeqPATE cannot always benefit from increasing the teacher numbers,because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.
Revised: Note that SeqPATE cannot always benefit from increasing the teacher numbers,because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.

Original: Wechoose ε = 3 on the sample level protection for all results in Tabs.
Revised: Wechoose ε = 3 on the sample level protection for all results in Tabs.

Original: 3, 4, and App. H.
Revised: 3, 4, and App. H.

Original:  Related Work
Revised:  Related Work

Original: Text generation models may leak user information through the generated texts [17, 6].
Revised: Text generation models may leak user information through the generated texts [17, 6].

Original: One directionof privacy protection is to protect author-level (user-level) information.
Revised: One directionof privacy protection is to protect author-level (user-level) information.

Original: The methods prevent attackersfrom inferring the author attributes (e.g., gender, age) [23] and the relationship between informationand authors [27].
Revised: The methods prevent attackersfrom inferring the author attributes (e.g., gender, age) [23] and the relationship between informationand authors [27].

Original: Some researchers [37, 38] infer the membership (whether samples from a givenauthor are used to train the model) given a black-box model.
Revised: Some researchers [37, 38] infer the membership (whether samples from a givenauthor are used to train the model) given a black-box model.

Original: Some papers protect users’ privacy oftraining data against untrusted servers via federated learning [25, 9].
Revised: Some papers protect users’ privacy oftraining data against untrusted servers via federated learning [25, 9].

Original: Another direction is to preventattackers from extracting sensitive information in training sets by analyzing the outputs [28, 20],which is in urgent need since Carlini et al.
Revised: Another direction is to preventattackers from extracting sensitive information in training sets by analyzing the outputs [28, 20],which is in urgent need since Carlini et al.

Original: [6] have achieved the extraction.
Revised: [6] have achieved the extraction.

Original: Our SeqPATE focuseson this direction.
Revised: Our SeqPATE focuseson this direction.

Original: In this direction, regularization methods [5, 40, 18] restrict the model capacityand avoid the model memorizing exact training samples.
Revised: In this direction, regularization methods [5, 40, 18] restrict the model capacityand avoid the model memorizing exact training samples.

Original: Anonymization methods [24, 41] detectsensitive text and replace it with non-sensitive texts.
Revised: Anonymization methods [24, 41] detectsensitive text and replace it with non-sensitive texts.

Original: Unlike DP [12] methods, the above methods donot provide a quantifiable guarantee for privacy protection.
Revised: Unlike DP [12] methods, the above methods donot provide a quantifiable guarantee for privacy protection.

Original: Some researchers apply DP to text generation.
Revised: Some researchers apply DP to text generation.

Original: For user-level privacy, ER-AE [4] augments thesemantic information in the generated text to hide authors’ writing styles from attackers.
Revised: For user-level privacy, ER-AE [4] augments thesemantic information in the generated text to hide authors’ writing styles from attackers.

Original: McMahanet al.
Revised: McMahanet al.

Original: [26] propose a recurrent language model with a DP guarantee against the identification of theusers.
Revised: [26] propose a recurrent language model with a DP guarantee against the identification of theusers.

Original: Note that the user-level privacy (relationships between users and their information) is differentfrom the privacy of users’ secret phrases in our model: Our model prevents individual users’ phrasesfrom being detected.
Revised: Note that the user-level privacy (relationships between users and their information) is differentfrom the privacy of users’ secret phrases in our model: Our model prevents individual users’ phrasesfrom being detected.

Original: Some researchers apply NoisySGD to text generation to prevent the sensitivetraining data (not the user-level) from being extracted: some of them [35, 36] employ DP to protect apart of selected tokens; others [20, 45, 22] apply DP to protect both samples and all tokens, but theprivacy cost on tokens is very high (Sec. 5.3).
Revised: Some researchers apply NoisySGD to text generation to prevent the sensitivetraining data (not the user-level) from being extracted: some of them [35, 36] employ DP to protect apart of selected tokens; others [20, 45, 22] apply DP to protect both samples and all tokens, but theprivacy cost on tokens is very high (Sec. 5.3).

Original: Our model falls into the latter category and reduces theprivacy cost of tokens.
Revised: Our model falls into the latter category and reduces theprivacy cost of tokens.

Original: Kerrigan et al.
Revised: Kerrigan et al.

Original: [20] apply NoisySGD [1] to text generation.
Revised: [20] apply NoisySGD [1] to text generation.

Original: Yu et al.
Revised: Yu et al.

Original: [45]investigate fine-tuning strategies on pre-trained language models with NoisySGD.
Revised: [45]investigate fine-tuning strategies on pre-trained language models with NoisySGD.

Original: Li et al.
Revised: Li et al.

Original: [22] applyghost clipping to pre-trained language models with NoisySGD and reduce memory usage.
Revised: [22] applyghost clipping to pre-trained language models with NoisySGD and reduce memory usage.

Original: Differential privacy (DP) [11, 12] formally defines and quantifies privacy.
Revised: Differential privacy (DP) [11, 12] formally defines and quantifies privacy.

Original: ML models with DPguarantee [43, 13, 47] prevents the existence of individual training examples from being detected [5].
Revised: ML models with DPguarantee [43, 13, 47] prevents the existence of individual training examples from being detected [5].

Original: Some researchers protect the privacy of empirical risk minimization classifiers [7] and SVM [34]with DP.
Revised: Some researchers protect the privacy of empirical risk minimization classifiers [7] and SVM [34]with DP.

Original: Following Song et al.
Revised: Following Song et al.

Original: [39], NoisySGD [1] achieves DP on deep learning models by addingnoises to gradients.
Revised: [39], NoisySGD [1] achieves DP on deep learning models by addingnoises to gradients.

Original: Pichapati et al.
Revised: Pichapati et al.

Original: [32] adaptively clip the gradient on NoisySGD.
Revised: [32] adaptively clip the gradient on NoisySGD.

Original: PATE [29, 30]transfers the knowledge from teacher models trained on private sets with noises to a student model.
Revised: PATE [29, 30]transfers the knowledge from teacher models trained on private sets with noises to a student model.

Original: KNN-PATE [46] refines PATE by accessing only the k-nearest neighbors from the private set.
Revised: KNN-PATE [46] refines PATE by accessing only the k-nearest neighbors from the private set.

Original: Jordonet al.
Revised: Jordonet al.

Original: [19] adversarially learn to generate synthetic data with discriminators trained by PATE.
Revised: [19] adversarially learn to generate synthetic data with discriminators trained by PATE.

Original: Thosemethods are not customized for text generation models.
Revised: Thosemethods are not customized for text generation models.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training datafor text generation models with DP guarantees.
Revised: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training datafor text generation models with DP guarantees.

Original: SeqPATE achieves a good privacy-utility tradeoff byleveraging both private and public data.
Revised: SeqPATE achieves a good privacy-utility tradeoff byleveraging both private and public data.

Original: As an extension of PATE, SeqPATE can handle the sequentialgeneration paradigm with large output space at each step and therefore is adaptive to text generationmodels.
Revised: As an extension of PATE, SeqPATE can handle the sequentialgeneration paradigm with large output space at each step and therefore is adaptive to text generationmodels.

Original: We avoid rolling out teachers by providing pseudo inputs for the teacher’s inference and thestudent’s training.
Revised: We avoid rolling out teachers by providing pseudo inputs for the teacher’s inference and thestudent’s training.

Original: We further reduce the output space by candidate filtering and limit privacy lossesvia efficient knowledge distillation.
Revised: We further reduce the output space by candidate filtering and limit privacy lossesvia efficient knowledge distillation.

Original: SeqPATE achieves a better performance with the sample levelprotection and further provides much stronger protection on users’ secret phrases.
Revised: SeqPATE achieves a better performance with the sample levelprotection and further provides much stronger protection on users’ secret phrases.

Revised: The limitation,ethical considerations, and social impacts of this paper are in Appendix A and M.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] It can be found in the supplemental
Revised: [Yes] It can be found in the supplemental

Original: materials.
Revised: materials.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] It can befound in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] It can be found in the
Revised: [Yes] It can be found in the

Original: supplemental materials.
Revised: supplemental materials.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] It can be found
Revised: [Yes] It can be found

Original: in the supplemental materials.
Revised: in the supplemental materials.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] It can be found in the body paper or the supplemental materials.
Revised: [Yes] It can be found in the body paper or the supplemental materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] It can be found in the body paper
Revised: [Yes] It can be found in the body paper

Original: or the supplemental materials.
Revised: or the supplemental materials.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] It can be found in the supplemental materials.[Table]
Revised: [Yes] It can be found in the supplemental materials.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]