Original: SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Revised: SeqPATE: Differentially Private Text Generation via Knowledge Distillation

Original: Abstract
Revised: Abstract

Original: Protecting the privacy of user data is crucial for text generation models, which can leak sensitive information during generation.
Revised: Protecting the privacy of user data is crucial for text generation models, which can leak sensitive information during generation.

Original: Differentially private (DP) learning methods provide guarantees against identifying the existence of a training sample from model outputs.
Revised: Differentially private (DP) learning methods provide guarantees against identifying the existence of a training sample from model outputs.

Original: PATE is a recent DP learning algorithm that achieves high utility with strong privacy protection on training samples.
Revised: PATE is a recent DP learning algorithm that achieves high utility with strong privacy protection on training samples.

Original: However, text generation models output tokens sequentially in a large output space; the classic PATE algorithm is not customized for this setting.
Revised: However, text generation models output tokens sequentially in a large output space; the classic PATE algorithm is not customized for this setting.

Original: Furthermore, PATE works well to protect sample-level privacy, but is not designed to protect phrases in samples.
Revised: Furthermore, PATE works well to protect sample-level privacy, but is not designed to protect phrases in samples.

Original: In this paper, we propose SeqPATE, an extension of PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data.
Revised: In this paper, we propose SeqPATE, an extension of PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data.

Original: To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem to a next-word prediction problem.
Revised: To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem to a next-word prediction problem.

Original: To handle the large output space, we propose a candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates.
Revised: To handle the large output space, we propose a candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates.

Original: To further reduce privacy losses, we use knowledge distillation to reduce the number of teacher queries.
Revised: To further reduce privacy losses, we use knowledge distillation to reduce the number of teacher queries.

Original: The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.
Revised: The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.

Original:  Introduction
Revised:  Introduction

Original: Recent work has shown that sensitive user information in training corpora, such as addresses and names, can be extracted from text generation models [6].
Revised: Recent work has shown that sensitive user information in training corpora, such as addresses and names, can be extracted from text generation models [6].

Original: Providing privacy guarantees to the training corpora of text generation models has become a critical problem.
Revised: Providing privacy guarantees to the training corpora of text generation models has become a critical problem.

Original: Differential privacy (DP) provides provable guarantees against detecting individuals in datasets.
Revised: Differential privacy (DP) provides provable guarantees against detecting individuals in datasets.

Original: Deep learning models with
Revised: Deep learning models with

Original: DP guarantees ensure that the existence of a specific training sample cannot be detected.
Revised: DP guarantees ensure that the existence of a specific training sample cannot be detected.

Original: NoisySGD [42, 3, 1] is a popular DP algorithm for deep learning that adds noise to the gradients.
Revised: NoisySGD [42, 3, 1] is a popular DP algorithm for deep learning that adds noise to the gradients.

Original: PATE [31] is another type of DP learning algorithm that transfers knowledge from teachers trained on private data to a student model, where noises are added to teacher predictions to satisfy DP.
Revised: PATE [31] is another type of DP learning algorithm that transfers knowledge from teachers trained on private data to a student model, where noises are added to teacher predictions to satisfy DP.

Original: PATE is model-agnostic, and its privacy cost derives from the knowledge distillation process instead of the model gradients in NoisySGD [42, 24].
Revised: PATE is model-agnostic, and its privacy cost derives from the knowledge distillation process instead of the model gradients in NoisySGD [42, 24].

Original: Therefore, the noises required by PATE do not scale with model size.
Revised: Therefore, the noises required by PATE do not scale with model size.

Original: Given this benefit, PATE has great potential for text generation, since large language models (e.g., GPT-2 [35]) have become the backbone of most text generation models.
Revised: Given this benefit, PATE has great potential for text generation, since large language models (e.g., GPT-2 [35]) have become the backbone of most text generation models.

Original: However,
Revised: However,

Original: NoisySGD and PATE are used to protect sample-level privacy [50, 24] and not customized to protect sensitive phrases in the data with a low privacy cost [22, 39].
Revised: NoisySGD and PATE are used to protect sample-level privacy [51, 24] and not customized to protect sensitive phrases in the data with a low privacy cost [22, 39, 50].

Original: Additionally, PATE, originally designed for classification tasks, is not customized for sequential generation on a large output space (i.e., the natural language vocabulary), which is very common in text generation.
Revised: Additionally, PATE, originally designed for classification tasks, is not customized for sequential generation on a large output space (i.e., the natural language vocabulary), which is very common in text generation.

Original: In this paper, we propose SeqPATE, a DP learning algorithm for text generation to protect the privacy of training corpora.
Revised: In this paper, we propose SeqPATE, a DP learning algorithm for text generation to protect the privacy of training corpora.

Original: By satisfying DP, SeqPATE has the guarantee of preventing the existence of training samples and sensitive phrases in the training corpora from being detected.
Revised: By satisfying DP, SeqPATE has the guarantee of preventing the existence of training samples and sensitive phrases in the training corpora from being detected.

Original: Similarly to PATE,
Revised: Similarly to PATE,

Original: SeqPATE employs a teacher-student framework: (i) a student model learns to generate text from nonsensitive samples; and (ii) a number of teacher models, trained on sensitive text, supervise the student through noised outputs of aggregated teachers.
Revised: SeqPATE employs a teacher-student framework: (i) a student model learns to generate text from nonsensitive samples; and (ii) a number of teacher models, trained on sensitive text, supervise the student through noised outputs of aggregated teachers.

Original: The calibrated noise added to the output ensures that
Revised: The calibrated noise added to the output ensures that

Original: SeqPATE satisfies the DP requirements.
Revised: SeqPATE satisfies the DP requirements.

Original: This framework still faces some challenges in text generation.
Revised: This framework still faces some challenges in text generation.

Original: First, it suffers from the high costs of GPU memory and time.
Revised: First, it suffers from the high costs of GPU memory and time.

Original: To obtain sentence-level supervision for text generation, the model needs to roll out all teachers to produce a sentence (i.e. all teachers vote to generate a word, which is then used as the input for the next word prediction).
Revised: To obtain sentence-level supervision for text generation, the model needs to roll out all teachers to produce a sentence (i.e. all teachers vote to generate a word, which is then used as the input for the next word prediction).

Original: It results in a high inference cost with a large number of teachers (e.g. 2 k teachers which are common in PATE).
Revised: It results in a high inference cost with a large number of teachers (e.g. 2 k teachers which are common in PATE).

Original: Second, the large output space (i.e., the vocabulary) in text generation leads to (i) low agreement rates among teachers and (ii) large noises required by DP, both of which significantly hurt the task performance.
Revised: Second, the large output space (i.e., the vocabulary) in text generation leads to (i) low agreement rates among teachers and (ii) large noises required by DP, both of which significantly hurt the task performance.

Original: To address the challenges, we generate pseudo-data using a pre-trained language model so that teachers only need to provide token-level supervision given the pseudo inputs.
Revised: To address the challenges, we generate pseudo-data using a pre-trained language model so that teachers only need to provide token-level supervision given the pseudo inputs.

Original: To handle the large output space and reduce the noise, we propose to dynamically filter the candidate words and select only words with high probabilities.
Revised: To handle the large output space and reduce the noise, we propose to dynamically filter the candidate words and select only words with high probabilities.

Original: Also, we aggregate teachers’ outputs by interpolating their output distributions instead of voting with argmax predictions.
Revised: Also, we aggregate teachers’ outputs by interpolating their output distributions instead of voting with argmax predictions.

Original: DP learning methods provide privacy protection by adding noise, which also reduces the utility of the model.
Revised: DP learning methods provide privacy protection by adding noise, which also reduces the utility of the model.

Original: To reduce utility loss, we avoid unnecessary knowledge distillation by selectively applying knowledge distillation to generation steps where the student struggles.
Revised: To reduce utility loss, we avoid unnecessary knowledge distillation by selectively applying knowledge distillation to generation steps where the student struggles.

Original: Most DP learning methods, including SeqPATE, prevent samples from being extracted.
Revised: Most DP learning methods, including SeqPATE, prevent samples from being extracted.

Original: SeqPATE has further advantages in protecting users’ secret phrases that occur multiple times in the corpora.
Revised: SeqPATE has further advantages in protecting users’ secret phrases that occur multiple times in the corpora.

Original: We evaluate SeqPATE on a sentence completion task, which demonstrates its advantage in protecting samples and phrases compared to the baselines.
Revised: We evaluate SeqPATE on a sentence completion task, which demonstrates its advantage in protecting samples and phrases compared to the baselines.

Original: Our contribution is twofold: (i)
Revised: Our contribution is twofold: (i)

Original: We propose SeqPATE that provides privacy at both the sample level and the phrase level with theoretical analyses.
Revised: We propose SeqPATE that provides privacy at both the sample level and the phrase level with theoretical analyses.

Original: (ii)
Revised: (ii)

Original: We propose several strategies for SeqPATE to handle autoregressive text generation models with a large vocabulary.
Revised: We propose several strategies for SeqPATE to handle autoregressive text generation models with a large vocabulary.

Original:  Problem Setup
Revised:  Problem Setup

Original: Our goal is to achieve the privacy protection quantified by DP in text generation to prevent attackers from inferring whether a sample or an n-gram appears in the training set.
Revised: Our goal is to achieve the privacy protection quantified by DP in text generation to prevent attackers from inferring whether a sample or an n-gram appears in the training set.

Original: Our setting contains two types of textual datasets: (1) a private set D pri from a corpus with sensitive information, (2) a public set D pub that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data.
Revised: Our setting contains two types of textual datasets: (1) a private set D pri from a corpus with sensitive information, (2) a public set D pub that contains no sensitive information or comes from data contributors (e.g., volunteers) who have no objection to publishing their data.

Original: We aim to protect the privacy on the private set and can ignore the privacy protection on the public set.
Revised: We aim to protect the privacy on the private set and can ignore the privacy protection on the public set.

Original: Our application, sentence completion, aims to complete the whole sentence given the prefix.
Revised: Our application, sentence completion, aims to complete the whole sentence given the prefix.

Original: We train a language model to accomplish the task.
Revised: We train a language model to accomplish the task.

Original: The public set D pub consists of prefixes, which can hardly contain sensitive information.
Revised: The public set D pub consists of prefixes, which can hardly contain sensitive information.

Original: The private set D pri consists of whole sentences.
Revised: The private set D pri consists of whole sentences.

Original: Such a setting fits some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses.
Revised: Such a setting fits some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses.

Original: The questions from customer service staff or service robots can be public, and the response from users carrying individual information should be private.
Revised: The questions from customer service staff or service robots can be public, and the response from users carrying individual information should be private.

Original:  Background on DP and PATE
Revised:  Background on DP and PATE

Original: Definition 3.1.
Revised: Definition 3.1.

Original: [Differential privacy (DP) [13, 14]] For any two neighboring datasets D , D ′ (differ in only one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,
Revised: [Differential privacy (DP) [13, 14]] For any two neighboring datasets D , D ′ (differ in only one individual), a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if,

Original: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, where S ⊆ Y ,  ε > 0 , δ ≥ 0 .
Revised: Pr [ M ( D ) ∈ S ] ≤ e ε · Pr [ M ( D ′ ) ∈ S ] + δ, ∀ S ⊆ Y , where ε > 0 , δ ≥ 0 .

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: Private Data +Noise
Revised: Private Data +Noise

Original: Teacher
Revised: Teacher

Original: Teacher Teacher
Revised: Teacher Teacher

Original: Student
Revised: Student

Original: NLL loss
Revised: NLL loss

Original: Data Data
Revised: Data Data

Original: Data
Revised: Data

Original: Cats sit on the Cats sit on the
Revised: Cats sit on the Cats sit on the

Original: Cats sit on the
Revised: Cats sit on the

Original: Candidate Filtering |Vocabulary|
Revised: Candidate Filtering |Vocabulary|

Original: Sit 0.42 Sat 0.33 Lay 0.
Revised: Sit 0.42 Sat 0.33 Lay 0.

Original: Are 0.08 ...
Revised: Are 0.08 ...

Original: Alaia 1eRuth 1eTop-K
Revised: Alaia 1eRuth 1eTop-K

Original: Aggregate
Revised: Aggregate

Original: Teacher Inference sit on the mats Teacher

Revised: Teacher Inference sit on the mats Teacher

Original: Teacher

Revised: Teacher

Original: Teacher
Revised: Teacher

Original: Cats sit
Revised: Cats sit

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: [Figure]
Revised: [Figure]

Original: supervisions
Revised: supervisions

Original: [Figure]
Revised: [Figure]

Original: teacher supervisions)
Revised: teacher supervisions)

Original: [Figure]
Revised: [Figure]

Original: Student Training
Revised: Student Training

Original: [Figure]
Revised: [Figure]

Original: Distillation
Revised: Distillation

Original: By definition, DP is a quantifiable definition of privacy that provides guarantees on identifications of individual data (preventing an adversary from inferring whether the input is D or D ′ ).
Revised: By definition, DP is a quantifiable definition of privacy that provides guarantees on identifications of individual data (preventing an adversary from inferring whether the input is D or D ′ ).

Original: ML models with DP ensure that each training sample has a degree of plausible deniability , i.e., the trained model is just as likely as to be trained on an alternative dataset without that sample.
Revised: ML models with DP ensure that each training sample has a degree of plausible deniability , i.e., the trained model is just as likely as to be trained on an alternative dataset without that sample.

Original: In SeqPATE, M is the entire training and inference process, S is the vocabulary, and Pr [ · ] denotes the output distribution of generating a word.
Revised: In SeqPATE, M is the entire training and inference process, S is the vocabulary, and Pr [ · ] denotes the output distribution of generating a word.

Original: Attackers cannot tell whether a sample is in the training set or not, since the output distributions of the datasets with or without that sample are very similar (bounded by Eq. 1).
Revised: Attackers cannot tell whether a sample is in the training set or not, since the output distributions of the datasets with or without that sample are very similar (bounded by Eq. 1).

Original: PATE [31], designed for classification tasks, takes advantage of an unlabeled public dataset D pub and also trains on a labeled private set D pri in a semi-supervised scenario.
Revised: PATE [31], designed for classification tasks, takes advantage of an unlabeled public dataset D pub and also trains on a labeled private set D pri in a semi-supervised scenario.

Original: PATE achieves DP through a teacher-student framework with M teacher models and a student model, where the student learns from the private set via knowledge distillation through teachers.
Revised: PATE achieves DP through a teacher-student framework with M teacher models and a student model, where the student learns from the private set via knowledge distillation through teachers.

Original: PATE has three parts: (i)
Revised: PATE has three parts: (i)

Original: The teacher models are trained on the private set D pri , which is shuffled and divided into M disjoint subsets.
Revised: The teacher models are trained on the private set D pri , which is shuffled and divided into M disjoint subsets.

Original: Each teacher is trained on one subset.
Revised: Each teacher is trained on one subset.

Original: (ii) Teacher aggregation merges the teachers’ outputs.
Revised: (ii) Teacher aggregation merges the teachers’ outputs.

Original: Each of the trained teachers then provides supervision to the student’s unlabeled public set D pub .
Revised: Each of the trained teachers then provides supervision to the student’s unlabeled public set D pub .

Original: We use noised majority votes from teachers as labels to supervise the student.
Revised: We use noised majority votes from teachers as labels to supervise the student.

Original: (iii)
Revised: (iii)

Original: A student model is trained on the public set D pub with the supervision of the aggregated teachers.
Revised: A student model is trained on the public set D pub with the supervision of the aggregated teachers.

Original:  Approach
Revised:  Approach

Original: Fig.
Revised: Fig.

Original: 1 shows an overview of SeqPATE.
Revised: 1 shows an overview of SeqPATE.

Original: Given the public prefix (e.g., “Cats sit”), we first obtain the pseudo-inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained language model (Sec. 4.1).
Revised: Given the public prefix (e.g., “Cats sit”), we first obtain the pseudo-inputs by completing the sentence (e.g., “Cats sit on the mats”) using a pre-trained language model (Sec. 4.1).

Original: At each word, we then aggregate the teachers’ prediction of the next word as supervision for training the student model (Sec. 4.2).
Revised: At each word, we then aggregate the teachers’ prediction of the next word as supervision for training the student model (Sec. 4.2).

Original: To reduce the noise required by DP for a large output space of the size of the vocabulary, we reduce the output space by dynamically filtering unimportant words.
Revised: To reduce the noise required by DP for a large output space of the size of the vocabulary, we reduce the output space by dynamically filtering unimportant words.

Original: To reduce the number of teacher queries that incur privacy losses, we propose an efficient knowledge distillation strategy that only queries teacher labels on uncertain examples (Sec. 4.3).
Revised: To reduce the number of teacher queries that incur privacy losses, we propose an efficient knowledge distillation strategy that only queries teacher labels on uncertain examples (Sec. 4.3).

Original: We show the training algorithm in App.
Revised: We show the training algorithm in App.

Original: B and a running example in App.
Revised: B and a running example in App.

Original: K.
Revised: K.

Original: 4.1 Pseudo Input Generation
Revised: 4.1 Pseudo Input Generation

Original: Conventional text generation models generate words sequentially from left to right.
Revised: Conventional text generation models generate words sequentially from left to right.

Original: Thus, naively applying PATE to text generation requires rolling out all teachers word by word, i.e., iteratively sampling the next word from the aggregated teacher prediction.
Revised: Thus, naively applying PATE to text generation requires rolling out all teachers word by word, i.e., iteratively sampling the next word from the aggregated teacher prediction.

Original: This is costly in both computation (running inference for hundreds of teacher models) and privacy costs (querying teachers at every step).
Revised: This is costly in both computation (running inference for hundreds of teacher models) and privacy costs (querying teachers at every step).

Original: To tackle this challenge, we use a pre-trained language model to complete the public prefixes into pseudo sentences; thus, we only need to query teachers on the next word given a (pseudo) context.
Revised: To tackle this challenge, we use a pre-trained language model to complete the public prefixes into pseudo sentences; thus, we only need to query teachers on the next word given a (pseudo) context.

Original: 4.2 Teacher Aggregation
Revised: 4.2 Teacher Aggregation

Original: PATE aggregates teacher predictions by majority vote.
Revised: PATE aggregates teacher predictions by majority vote.

Original: While it works for classification problems with a relatively small number of classes, the output space of text generation models contains all words in the vocabulary.
Revised: While it works for classification problems with a relatively small number of classes, the output space of text generation models contains all words in the vocabulary.

Original: As a result, the number of votes for each candidate word may be very low without a clear winner.
Revised: As a result, the number of votes for each candidate word may be very low without a clear winner.

Original: For example, multiple candidates may tie for the top-1 prediction.
Revised: For example, multiple candidates may tie for the top-1 prediction.

Original: Inspired by Chen et al.
Revised: Inspired by Chen et al.

Original: [9, 17], we aggregate teacher results by averaging their output distributions.
Revised: [9, 17], we aggregate teacher results by averaging their output distributions.

Original: We first train M teacher models on disjoint subsets of the private data.
Revised: We first train M teacher models on disjoint subsets of the private data.

Original: To produce the aggregated next word distribution given a context c , we average the teachers’ output distributions, add calibrated noises, and then renormalize the results into a proper distribution.
Revised: To produce the aggregated next word distribution given a context c , we average the teachers’ output distributions, add calibrated noises, and then renormalize the results into a proper distribution.

Original: Following Papernot et al.
Revised: Following Papernot et al.

Original: [32], we apply the Gaussian mechanism.
Revised: [32], we apply the Gaussian mechanism.

Original: Formally, let p mϕ ( · | c ) be the prediction of the m -th teacher.
Revised: Formally, let p mϕ ( · | c ) be the prediction of the m -th teacher.

Original: The aggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)
Revised: The aggregated distribution is p agg ( · | c ) ∝ 1 M (cid:80)

Original: Mm =

Revised: Mm =

Original: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 2 where the Gaussian noise is added to the aggregated output distribution.
Revised: ( p mϕ ( · | c ) + N (0 , σ 2 )) , 2 where the Gaussian noise is added to the aggregated output distribution.

Original: The way of SeqPATE satisfies DP guarantee (Eq. 1) is to add that calibrated noise to the teachers’ output as mentioned above (detailed analyses in Sec. 5).
Revised: The way of SeqPATE satisfies DP guarantee (Eq. 1) is to add that calibrated noise to the teachers’ output as mentioned above (detailed analyses in Sec. 5).

Original: 4.3 Training of the Student Model
Revised: 4.3 Training of the Student Model

Original: The student model is trained on public pseudo-data and also supervised by the aggregated teachers.
Revised: The student model is trained on public pseudo-data and also supervised by the aggregated teachers.

Original: Training objectives.
Revised: Training objectives.

Original: The student model is a language model that predicts the next word given prior contexts.
Revised: The student model is a language model that predicts the next word given prior contexts.

Original: Given contexts from the (public) pseudo-data autocompleted by a pre-trained language model (GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in the pseudo-data (i.e. pseudo label).
Revised: Given contexts from the (public) pseudo-data autocompleted by a pre-trained language model (GPT-2), the student is supervised by both the aggregated teacher predictions and the next word in the pseudo-data (i.e. pseudo label).

Original: The pseudo-data acts as a prior for the student given that the number of teacher queries is limited due to privacy concerns.
Revised: The pseudo-data acts as a prior for the student given that the number of teacher queries is limited due to privacy concerns.

Original: The student’s loss function has two parts:
Revised: The student’s loss function has two parts:

Original: • L teacher denotes the loss with respect
Revised: • L teacher denotes the loss with respect

Original: to teacher supervision.
Revised: to teacher supervision.

Original: Note that the aggregated teacher output is a distribution over words.
Revised: Note that the aggregated teacher output is a distribution over words.

Original: Therefore, we minimize the forward KL divergence between the aggregated teacher distribution p agg and the student output distribution p θ :
Revised: Therefore, we minimize the forward KL divergence between the aggregated teacher distribution p agg and the student output distribution p θ :

Original: [Equation]
Revised: [Equation]

Original: L pseudo denotes the loss with respect to the pseudo-labels w from ˜ D pub (i
Revised: L pseudo denotes the loss with respect to the pseudo-labels w from ˜ D pub (i

Original: e. next words generated by a
Revised: e. next words generated by a

Original: generic language
Revised: generic language

Original: model
Revised: model

Original: Similar to
Revised: Similar to

Original: standard language
Revised: standard language

Original: modeling,
Revised: modeling,

Original: we
Revised: we

Original: use
Revised: use

Original: the negative
Revised: the negative

Original: log-
Revised: log-

Original: likelihood:
Revised: likelihood:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 4 shows the complete loss.
Revised: 4 shows the complete loss.

Original: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)
Revised: ( λ balances the two terms and we discuss the noise scale σ in Sec. 5.)

Original: [Equation]
Revised: [Equation]

Original: Reducing the output space via candidate filtering.
Revised: Reducing the output space via candidate filtering.

Original: The high-dimensionality of the output of text generation models results in large noise (which is added to each coordinate).
Revised: The high-dimensionality of the output of text generation models results in large noise (which is added to each coordinate).

Original: To reduce the output dimension (hence the amount of noise), we filter words on the tail of the distribution of the student model (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and the student output distribution over the rest words.
Revised: To reduce the output dimension (hence the amount of noise), we filter words on the tail of the distribution of the student model (i.e. set their probability to zero), and renormalize the teacher’s aggregated distribution and the student output distribution over the rest words.

Original: Note that the candidate filtering is based on the student’s outputs on public or already released inputs, thus it does not affect the privacy guarantee.
Revised: Note that the candidate filtering is based on the student’s outputs on public or already released inputs, thus it does not affect the privacy guarantee.

Original: This choice improves the privacy-utility tradeoff by adaptively allocating the privacy budget to release the information most helpful to the task.
Revised: This choice improves the privacy-utility tradeoff by adaptively allocating the privacy budget to release the information most helpful to the task.

Original: We experiment with two filtering strategies: top- k and top- p .
Revised: We experiment with two filtering strategies: top- k and top- p .

Original: In top- k filtering, we retain only the top- k most likely candidates and filter the rest according to the student model.
Revised: In top- k filtering, we retain only the top- k most likely candidates and filter the rest according to the student model.

Original: In top- p filtering [18], k is chosen dynamically such that the top- k words are the minimum set whose cumulative probability is at least p .
Revised: In top- p filtering [18], k is chosen dynamically such that the top- k words are the minimum set whose cumulative probability is at least p .

Original: The strategy seldom loses good candidates because the student usually does well on top- k predictions since the beginning of the training.
Revised: The strategy seldom loses good candidates because the student usually does well on top- k predictions since the beginning of the training.

Original: Reducing the number of teacher queries via efficient knowledge distillation.
Revised: Reducing the number of teacher queries via efficient knowledge distillation.

Original: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.
Revised: While the aggregated teacher model satisfies DP, each query from the student incurs some privacy loss.

Original: Therefore, we obtain teacher supervision only on “hard” examples when training the student.
Revised: Therefore, we obtain teacher supervision only on “hard” examples when training the student.

Original: Note that the student is trained on both the pseudo-data and local supervision from the teachers.
Revised: Note that the student is trained on both the pseudo-data and local supervision from the teachers.

Original: We consider an example to be hard if the student cannot imitate the pseudo-label, in which case distilling knowledge from the teachers that are trained on large private data is helpful.
Revised: We consider an example to be hard if the student cannot imitate the pseudo-label, in which case distilling knowledge from the teachers that are trained on large private data is helpful.

Original: Concretely, we query teachers only when the rank of the pseudo-label is below a certain threshold among words ordered by descending probabilities under the student model.
Revised: Concretely, we query teachers only when the rank of the pseudo-label is below a certain threshold among words ordered by descending probabilities under the student model.

Original: If we query the teachers, the student is trained via complete loss L ( p agg , ˜ D pub )
Revised: If we query the teachers, the student is trained via complete loss L ( p agg , ˜ D pub )

Original: (Eq. 4); otherwise, the student is trained via the
Revised: (Eq. 4); otherwise, the student is trained via the

Original: L pseudo (Eq. 3).
Revised: L pseudo (Eq. 3).

Original: We note that the selection of tokens relies only on the student and is independent of the teachers; thus, the selection does not cause any additional privacy loss.
Revised: We note that the selection of tokens relies only on the student and is independent of the teachers; thus, the selection does not cause any additional privacy loss.

Original:  Privacy Analyses 5.1 Preliminary of Differential Privacy
Revised:  Privacy Analyses 5.1 Preliminary of Differential Privacy

Original: Lemma 5.1 (Analytical Gaussian mechanism [2]) .
Revised: Lemma 5.1 (Analytical Gaussian mechanism [2]) .

Original: For a numeric query f : X n → R d over a dataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies ( ε, δ ( ε ))
Revised: For a numeric query f : X n → R d over a dataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) =
Revised: -DP for all ε ≥ 0 and δ ( ε ) =

Original: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .
Revised: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .

Original: where ∆ := ∆ ( f )= max D∼D ′ ∥
Revised: where ∆ := ∆ ( f )= max D∼D ′ ∥

Original: f ( D ) − f ( D ′ )
Revised: f ( D ) − f ( D ′ )

Original: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .
Revised: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .

Original: We can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.
Revised: We can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.

Original: Lemma 5.2 (Composition of Gaussian mechanisms [11]) .
Revised: Lemma 5.2 (Composition of Gaussian mechanisms [11]) .

Original: The adaptive composition of a sequence of Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .
Revised: The adaptive composition of a sequence of Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .

Original: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .
Revised: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .

Original: satisfies ( ε, δ ( ε ))
Revised: satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noise multiplier σ/ ∆ =
Revised: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noise multiplier σ/ ∆ =

Original: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .
Revised: (cid:0)(cid:80) i (∆ i /σ i ) 2 (cid:1) − 1 / 2 .

Original: Specifically, the adaptive composition of a k identical Gaussian mechanism with a noise multiplier σ satisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .
Revised: Specifically, the adaptive composition of a k identical Gaussian mechanism with a noise multiplier σ satisfies the same privacy guarantee of a single Gaussian mechanism with a noise multiplier σ/ √ k .

Original: By fixing k and ε , we can calibrate the noise by choosing an appropriate σ in Sec.
Revised: By fixing k and ε , we can calibrate the noise by choosing an appropriate σ in Sec.

Original: 5.2 Differential Privacy for Language Models at the Sample Level
Revised: 5.2 Differential Privacy for Language Models at the Sample Level

Original: Recall that we partition the private dataset into M disjoint subsets, and train each teacher model on one of the subsets.
Revised: Recall that we partition the private dataset into M disjoint subsets, and train each teacher model on one of the subsets.

Original: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -th teacher model given some context, where |V| is the vocabulary size.
Revised: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -th teacher model given some context, where |V| is the vocabulary size.

Original: The aggregation function f ( D ) :=
Revised: The aggregation function f ( D ) :=

Original: (cid:80) Mi
Revised: (cid:80) Mi

Original: =1 x i is the sum of the probability distributions predicted by all teachers.
Revised: =1 x i is the sum of the probability distributions predicted by all teachers.

Original: Since the datasets are disjoint, changing one sample affects only one teacher model.
Revised: Since the datasets are disjoint, changing one sample affects only one teacher model.

Original: For neighboring datasets
Revised: For neighboring datasets

Original: D , D ′ , let j denote the index of each teacher model; the probability distributions x j and x ′ j (derived from D and D ′ respectively) are different.
Revised: D , D ′ , let j denote the index of each teacher model; the probability distributions x j and x ′ j (derived from D and D ′ respectively) are different.

Original: Then, the sensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),
Revised: Then, the sensitivity ∆ in Lemma 5.1 & 5.2 is (See detailed deductions in App. C),

Original: [Equation]
Revised: [Equation]

Original: Adding the noises given by Lemma 5.2 to each coordinate (each candidate at each generation step of SeqPATE) preserves ( ε , δ ( ε ) )
Revised: Adding the noises given by Lemma 5.2 to each coordinate (each candidate at each generation step of SeqPATE) preserves ( ε , δ ( ε ) )

Original: -DP for f ( D ) .
Revised: -DP for f ( D ) .

Original: Finally, when we extract top- k coordinates by top- k candidate filtering (Sec. 4.3), the privacy guarantee also holds due to the post-processing property [14].
Revised: Finally, when we extract top- k coordinates by top- k candidate filtering (Sec. 4.3), the privacy guarantee also holds due to the post-processing property [14].

Original: Therefore, the fact about whether a sample is in SeqPATE’s private sets is protected (satisfying ( ε , δ ( ε ) )-DP).
Revised: Therefore, the fact about whether a sample is in SeqPATE’s private sets is protected (satisfying ( ε , δ ( ε ) )-DP).

Original: 5.3 Differential Privacy of Users’ Secret Phrases
Revised: 5.3 Differential Privacy of Users’ Secret Phrases

Original: The above analyses show that we can protect the privacy of each sample (i.e., one occurrence of a sentence).
Revised: The above analyses show that we can protect the privacy of each sample (i.e., one occurrence of a sentence).

Original: However, in practice, we may want to protect all occurrences of some secret phrases specific to a user (e.g., names and addresses).
Revised: However, in practice, we may want to protect all occurrences of some secret phrases specific to a user (e.g., names and addresses).

Original: 4 Consider a secret phrase s that occurs n s times ( n s ≥ 1 ) in the private set.
Revised: 4 Consider a secret phrase s that occurs n s times ( n s ≥ 1 ) in the private set.

Original: According to group privacy [14], the protection on phrase s satisfies ( nε, e nε − 1 e ε − 1 δ ) -DP
Revised: According to group privacy [14], the protection on phrase s satisfies ( nε, e nε − 1 e ε − 1 δ ) -DP

Original: [22, 39], where the privacy loss scales linearly with the number of occurrences of s (We discuss and analyze a better strategy to reduce the privacy loss of baselines in App. M).
Revised: [22], where the privacy loss scales linearly with the number of occurrences of s (We discuss and analyze a better strategy to reduce the privacy loss of baselines in App. M).

Original: Naively applying a DP algorithm requires larger noise to protect phrases that may occur multiple times.
Revised: Naively applying a DP algorithm requires larger noise to protect phrases that may occur multiple times.

Original: SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or a few teachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.
Revised: SeqPATE enjoys a stronger guarantee by assigning all data of a single user to one or a few teachers, such that any user-specific phrase occurs in the training data of only one or a few teachers.

Original: We denote ˜ n s as the number of teachers whose data contain the phrase s .
Revised: We denote ˜ n s as the number of teachers whose data contain the phrase s .

Original: Since adding or removing the phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s (See App.
Revised: Since adding or removing the phrase s affects only ˜ n s teachers ( ˜ n s is usually 1 or 2 ) and thus results in a sensitivity of √ 2˜ n s (See App.

Original: D for details).
Revised: D for details).

Original: In this way, the strength of protection on secret phrases is roughly equal to that we have derived for sample-level DP.
Revised: In this way, the strength of protection on secret phrases is roughly equal to that we have derived for sample-level DP.

Original: The exact ( ε, δ ( ε, ˜ n s ))
Revised: The exact ( ε, δ ( ε, ˜ n s ))

Original: -DP for the phrase s can be obtained according to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =
Revised: -DP for the phrase s can be obtained according to Lemma 5.1 & 5.2, where δ ( ε, ˜ n s ) =

Original: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .
Revised: Φ( ˜ n s √ 2 σ − εσ √ 2˜ n s ) − e ε Φ( − ˜ n s √ 2 σ − εσ √ 2˜ n s ) .

Original: Unlike other generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacy loss (i.e., a linear increase in ε ) on user phrases by careful partitioning of the private data.
Revised: Unlike other generic DP algorithms such as NoisySGD, SeqPATE avoids a linear increase in privacy loss (i.e., a linear increase in ε ) on user phrases by careful partitioning of the private data.

Revised: This effect is complimentary to other generic, but more intrusive, techniques such as redaction and deduplication [50] for addressing the same issue.

Revised: Finally, a user-specific partitioning with SeqPATE also protects multiple secret phrases of the same user (e.g., a combination of SSN, credit card numbers, address, day of birth) jointly without incurring a larger privacy loss — a benefit that deduplication does not provide.

Original: 5.4 How does DP prevent memorization in SeqPATE?
Revised: 5.4 How does DP prevent memorization in SeqPATE?

Original: In practice, the privacy of the language model is usually interpreted as not generating a secret phrase in the training data as-is during inference.
Revised: In practice, the privacy of the language model is usually interpreted as not generating a secret phrase in the training data as-is during inference.

Original: Thus, one may wonder how DP prevents such unintended memorization of the training data.
Revised: Thus, one may wonder how DP prevents such unintended memorization of the training data.

Original: We remark that the protection against memorization follows the definition of DP.
Revised: We remark that the protection against memorization follows the definition of DP.

Original: Consider the attack by Carlini et al.
Revised: Consider the attack by Carlini et al.

Original: [6], which uses a language model to predict a secret phrase s given a prefix.
Revised: [6], which uses a language model to predict a secret phrase s given a prefix.

Original: By the closure to post-processing [14], the prediction also satisfies
Revised: By the closure to post-processing [14], the prediction also satisfies

Original: DP.
Revised: DP.

Original: We denote W as the undesirable event where SeqPATE generates the phrase s verbatim.
Revised: We denote W as the undesirable event where SeqPATE generates the phrase s verbatim.

Original: The DP definition implies that the probability of W to happen when s is in the SeqPATE’s private sets is at most e ε larger than the probability of an alternative SeqPATE model trained without s in those sets.
Revised: The DP definition implies that the probability of W to happen when s is in the SeqPATE’s private sets is at most e ε larger than the probability of an alternative SeqPATE model trained without s in those sets.

Original: The chances for the latter model to generate text with s are astronomically small.
Revised: The chances for the latter model to generate text with s are astronomically small.

Original: Hence, DP implies that the probability of W under the former model (i.e. any SeqPATE model in general) is small.
Revised: Hence, DP implies that the probability of W under the former model (i.e. any SeqPATE model in general) is small.

Original:  Experiments 6.1 Experimental Settings
Revised:  Experiments 6.1 Experimental Settings

Original: Datasets.
Revised: Datasets.

Original: We evaluate our model on two datasets.
Revised: We evaluate our model on two datasets.

Original: AirDialog [47] consists of 1M utterances from customer service dialog on flight booking; Europarl_v6 consists of 2M English sentences collected from European Parliament.
Revised: AirDialog [47] consists of 1M utterances from customer service dialog on flight booking; Europarl_v6 consists of 2M English sentences collected from European Parliament.

Original: 5 (See details about datasets in App. E.)
Revised: 5 (See details about datasets in App. E.)

Original: Baselines.
Revised: Baselines.

Original: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on the private data with calibrated noise on clipped gradients [1, 22] and further trained on public set D pub without protection; (2) based on NoisySGD , NoisySGD+GC [24] applies a ghost clipping which enables large batch size with memory saving techniques.
Revised: We compare SeqPATE with two DP baselines: (1) standard NoisySGD trained on the private data with calibrated noise on clipped gradients [1, 22] and further trained on public set D pub without protection; (2) based on NoisySGD , NoisySGD+GC [24] applies a ghost clipping which enables large batch size with memory saving techniques.

Original: Additionally, we use two non-DP methods as reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the public pre-trained GPT-2 model Pub-GPT without access to private data.
Revised: Additionally, we use two non-DP methods as reference: (1) Pri-GPT trained on the private set without any privacy protection; (2) the public pre-trained GPT-2 model Pub-GPT without access to private data.

Original: For all methods, we can optionally fine-tune on the generated pseudo-data as a warm-up, and the operation is denoted as + ˜ D pub .
Revised: For all methods, we can optionally fine-tune on the generated pseudo-data as a warm-up, and the operation is denoted as + ˜ D pub .

Original: Implementation details.
Revised: Implementation details.

Original: All models are fine-tuned from the (public) pre-trained GPT-2 model [35].
Revised: All models are fine-tuned from the (public) pre-trained GPT-2 model [35].

Original: The batch size is 32 for all comparing methods except the GC [24] (GC [24] requires 2048).
Revised: The batch size is 32 for all comparing methods except the GC [24] (GC [24] requires 2048).

Original: We use
Revised: We use

Original: Adam [23] and adjust the initial learning rate with a range of 10 − 3 to 10 − 6 for all methods.
Revised: Adam [23] and adjust the initial learning rate with a range of 10 − 3 to 10 − 6 for all methods.

Original: The δ mentioned in Sec.
Revised: The δ mentioned in Sec.

Original: 5 for all DP methods is 10 − 6 .
Revised: 5 for all DP methods is 10 − 6 .

Original: For SeqPATE, before training the student model with teacher supervision, we first fine-tune it on the public pseudo-data ˜ D pub as a warm-up.
Revised: For SeqPATE, before training the student model with teacher supervision, we first fine-tune it on the public pseudo-data ˜ D pub as a warm-up.

Original: The coefficient λ that balances supervision for the teacher and the pseudo-data
Revised: The coefficient λ that balances supervision for the teacher and the pseudo-data

Original: (Eq. 4) is set to 20, where we have tuned it on the validation set of the public pseudo-data.
Revised: (Eq. 4) is set to 20, where we have tuned it on the validation set of the public pseudo-data.

Original: The default number of teacher models is 2 k , where our model works well according to the experiments in App.
Revised: The default number of teacher models is 2 k , where our model works well according to the experiments in App.

Original: H. We designed some strategies 6 to reduce memory and disk usage (See strategies and the computational cost in App. I).
Revised: H. We designed some strategies 6 to reduce memory and disk usage (See strategies and the computational cost in App. I).

Original: We run SeqPATE with 2 k teachers on a single GPU in 3 days.
Revised: We run SeqPATE with 2 k teachers on a single GPU in 3 days.

Original: Our code is publicly accessible.
Revised: Our code is publicly accessible.

Original: (See details about hyperparameters in App. G.)
Revised: (See details about hyperparameters in App. G.)

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [33].
Revised: We evaluate the generated text by perplexity (PPL) and Bleu (Bleu-n) [33].

Original: 6.2 Overall Performance
Revised: 6.2 Overall Performance

Original: [Table]
Revised: are

Original: [Table]
Revised: [Table]

Original: Protection at the sample level.
Revised: Protection at the sample level.

Original: Tab. 1 show the performance on the two datasets.
Revised: Tab. 1 show the performance on the two datasets.

Original: Among the non-DP baselines, Pri-GPT acts as an upper bound on the performance, since it can fully utilize the private set by discarding privacy protection.
Revised: Among the non-DP baselines, Pri-GPT acts as an upper bound on the performance, since it can fully utilize the private set by discarding privacy protection.

Original: Pub-GPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additional ablation study on the pseudo data in App.
Revised: Pub-GPT+ ˜ D pub outperforms Pub-GPT on both datasets, showing that the pseudo data is helpful (additional ablation study on the pseudo data in App.

Original: J also verifies this).
Revised: J also verifies this).

Original: NoisySGD+GC+
Revised: NoisySGD+GC+

Original: ˜
Revised: ˜

Original: D pub surpasses the above two methods, since it uses a much larger batch size (2048 vs 32) than NoisySGD.
Revised: D pub surpasses the above two methods, since it uses a much larger batch size (2048 vs 32) than NoisySGD.

Original: Our method, SeqPATE, significantly outperforms
Revised: Our method, SeqPATE, significantly outperforms

Original: NoisySGD+GC+ ˜
Revised: NoisySGD+GC+ ˜

Original: D pub (+59% in Bleu4 on AirDialog and +7.0% in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε .
Revised: D pub (+59% in Bleu4 on AirDialog and +7.0% in Bleu4 on Europarl_v6) while ensuring the same level of privacy protection in terms of ε .

Original: Protection on the user’s secret phrases.
Revised: Protection on the user’s secret phrases.

Original: We evaluate our method for privacy protection of secret phrases mentioned in Sec 5.3.
Revised: We evaluate our method for privacy protection of secret phrases mentioned in Sec 5.3.

Original: The key step is to partition the data such that each phrase only occurs in the training data of very few teachers, which is straightforward given the user ID associated with the private data.
Revised: The key step is to partition the data such that each phrase only occurs in the training data of very few teachers, which is straightforward given the user ID associated with the private data.

Original: In general, SeqPATE works with any set of secret phrases.
Revised: In general, SeqPATE works with any set of secret phrases.

Original: In our experiments, we consider a user’s full name as their secret phrase since it can be easily recognized from the data.
Revised: In our experiments, we consider a user’s full name as their secret phrase since it can be easily recognized from the data.

Original: We partition AirDialog’s private data according to the accompanying user IDs.
Revised: We partition AirDialog’s private data according to the accompanying user IDs.

Original: As a result, there are 96.6% users whose data are assigned to a single teacher (details about the data partition in App. F).PPL ↓ Bleu-3 ↑ [Table]
Revised: As a result, there are 96.6% users whose data are assigned to a single teacher (details about the data partition in App. F).

Original: -4 ↑

Original: NoisySGD+

Original: [Equation]

Original: +

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: NoisySGD

Original: [Equation]

Original: users)

Original: [Table] [Table] [Table] As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.
Revised: As described in Sec. 5.3, standard DP methods incur larger privacy loss on secret phrases.

Original: In Tab. 2, we see that NoisySGD+GC+ ˜
Revised: In Tab. 2, we see that NoisySGD+GC+ ˜

Original: D pub needs large noise to achieve a satisfactory level of protection on phrases, because ε increases linearly with the frequency of the phrase (group privacy [14]).
Revised: D pub needs large noise to achieve a satisfactory level of protection on phrases, because ε increases linearly with the frequency of the phrase (group privacy [14]).

Original: “Batching users” indicates partitioning data into batches according to users, which helps NoisySGD protect users’ phrases (more analyses in App. M).
Revised: “Batching users” indicates partitioning data into batches according to users, which helps NoisySGD protect users’ phrases (more analyses in App. M).

Original: For SeqPATE, the number of teachers trained on data containing the phrase ˜ n s is close to 1 on average after our partition.
Revised: For SeqPATE, the number of teachers trained on data containing the phrase ˜ n s is close to 1 on average after our partition.

Original: Thus, SeqPATE provides the same level of protection on users’ secret phrases with a smaller noise and thus achieves better performance (+70% and +36% in Bleu4) (see more about the protection level on users’ secret phrases in App. F). 
Revised: Thus, SeqPATE provides the same level of protection on users’ secret phrases with a smaller noise and thus achieves better performance (+70% and +36% in Bleu4) (see more about the protection level on users’ secret phrases in App. F). [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: Privacy-utility tradeoff.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: show the results on AirDialog; c & d show the results on

Revised: Europarl_v6.

Revised: The grey lines show the “lower bound” since the method does not access the private set.

Revised: Privacy-utility tradeoff.

Original: In Fig.
Revised: In Fig.

Original: 2, we show the private-utility tradeoff curve of all DP algorithms.
Revised: 2, we show the private-utility tradeoff curve of all DP algorithms.

Original: Typically, DP with ε ∈ [0 . 1 , 10] is considered to provide a meaningful protection [45].
Revised: Typically, DP with ε ∈ [0 . 1 , 10] is considered to provide a meaningful protection [45].

Original: We observe that SeqPATE outperforms NoisySGD and NoisySGD+GC+ ˜
Revised: We observe that SeqPATE outperforms NoisySGD and NoisySGD+GC+ ˜

Original: D pub in this range.
Revised: D pub in this range.

Original: However, SeqPATE does not work better than the two methods when ε > 10 .
Revised: However, SeqPATE does not work better than the two methods when ε > 10 .

Original: The reason is that NoisySGD+GC+ ˜
Revised: The reason is that NoisySGD+GC+ ˜

Original: D pub approaches Pri-GPT as ε approaches infinity (i [Table] e. the noise approaches 0).
Revised: D pub approaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).

Original: However, SeqPATE with an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: the teachers cannot completely transfer knowledge from the private data to the student.
Revised: However, SeqPATE with an infinite ε is still weaker than Pri-GPT because distillation still incurs performance loss: the teachers cannot completely transfer knowledge from the private data to the student.

Original: Therefore, we suggest using SeqPATE if strong privacy protection is desirable
Revised: Therefore, we suggest using SeqPATE if strong privacy protection is desirable.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 6.3 Ablation Studies
Revised: 6.3 Ablation Studies

Original: There are several design choices in SeqPATE
Revised: There are several design choices in SeqPATE

Original: and we study the importance of each of them.
Revised: and we study the importance of each of them.

Original: In
Revised: In

Original: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachers by voting instead of averaging their output distributions; (2) − KL: training the student using the cross-entropy loss with respect to teachers’ top- prediction instead of 
Revised: Tab. 3, we consider the following variants of SeqPATE: (1) − Merge_P: aggregating the teachers by voting instead of averaging their output distributions; (2) − KL: training the student using the cross-entropy loss with respect to teachers’ top-1 prediction instead of KL divergence; (3) −L pseudo : not learning from the pseudo label (Eq. 3);

Original: KL divergence; (3) −L pseudo : not learning from the pseudo label (Eq. 3); (4) − Effi KD: querying teachers on all samples without selection; (5) − Gaussian: using the Laplace mechanism as the original PATE algorithm instead of the
Revised: (4) − Effi KD: querying teachers on all samples without selection; (5) − Gaussian: using the Laplace mechanism as the original PATE algorithm instead of the

Original: Gaussian mechanism; and (6) − All: using none of the above strategies, which is similar (although not equivalent) to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1)).
Revised: Gaussian mechanism; and (6) − All: using none of the above strategies, which is similar (although not equivalent) to the original PATE (the difference is that PATE needs to roll out all teachers (Sec. 4.1)).

Original: Aggregating the teachers by voting and training with KL loss are the most important strategies for
Revised: Aggregating the teachers by voting and training with KL loss are the most important strategies for

Original: SeqPATE.
Revised: SeqPATE.

Original: The poor performance on − Merge_P shows that voting is not suitable for text generation.
Revised: The poor performance on − Merge_P shows that voting is not suitable for text generation.

Original: The reason is that voting over a large output space leads to low agreement rates.
Revised: The reason is that voting over a large output space leads to low agreement rates.

Original: The results show that the L pseudo loss makes little contribution to SeqPATE.
Revised: The results show that the L pseudo loss makes little contribution to SeqPATE.

Original: The reason is that we have pre-trained on the student’s training set via L pseudo before the student’s training.
Revised: The reason is that we have pre-trained on the student’s training set via L pseudo before the student’s training.

Original: The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows that the “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from this strategy.
Revised: The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl_v6, which shows that the “clever” student (e.g., models on AirDialog with low PPL and high Bleu) benefits more from this strategy.

Original: This is because the “clever” student can dramatically save the privacy cost and transfer it to where it would benefit the student most.
Revised: This is because the “clever” student can dramatically save the privacy cost and transfer it to where it would benefit the student most.

Original: The poor performance of − All verifies that the original
Revised: The poor performance of − All verifies that the original

Original: PATE is not suitable for text generation.
Revised: PATE is not suitable for text generation.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 6.4 Analyses on Candidate Filtering and Teacher Numbers
Revised: 6.4 Analyses on Candidate Filtering and Teacher Numbers

Original: To analyze candidate filtering with different filtering strategies, we conduct experiments on top- p and top- k filtering.
Revised: To analyze candidate filtering with different filtering strategies, we conduct experiments on top- p and top- k filtering.

Original: As shown in Tab. 4, our full model employs the top- p filtering (the threshold p is 0 .
Revised: As shown in Tab. 4, our full model employs the top- p filtering (the threshold p is 0 .

Original: 95 ) surpasses most variants with manually chosen k .
Revised: 95 ) surpasses most variants with manually chosen k .

Original: Top-
Revised: Top-

Original: k filtering ( k = 50 or 100) also works well.
Revised: k filtering ( k = 50 or 100) also works well.

Original: Filtering with a too small k ( k = 1 or k = 10 ) implies discarding too much useful information from the supervision ( k = 1 is different from − KL in Tab. 3, which uses the Top-1 of teachers’ results).
Revised: Filtering with a too small k ( k = 1 or k = 10 ) implies discarding too much useful information from the supervision ( k = 1 is different from − KL in Tab. 3, which uses the Top-1 of teachers’ results).

Original: Filtering with oversize k results in unnecessarily large noises.
Revised: Filtering with oversize k results in unnecessarily large noises.

Original: Candidates with very small probabilities should be filtered during generation; however, random noises may increase their probabilities, so models may generate those words that are misled by the noise.
Revised: Candidates with very small probabilities should be filtered during generation; however, random noises may increase their probabilities, so models may generate those words that are misled by the noise.

Original: The results in App.
Revised: The results in App.

Original: H show that more teachers lead to better results when the number of teachers is in the range of 1 ∼ 2 k .
Revised: H show that more teachers lead to better results when the number of teachers is in the range of 1 ∼ 2 k .

Original: This is because the noise assigned to each teacher drops linearly as the number of teachers increases.
Revised: This is because the noise assigned to each teacher drops linearly as the number of teachers increases.

Original: Note that SeqPATE cannot always benefit from increasing the teacher numbers, because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.
Revised: Note that SeqPATE cannot always benefit from increasing the teacher numbers, because the scale of each teacher’s data is linearly decreased as the teacher numbers go up.

Original: We choose ε = 3 on the sample level protection for all results in Tabs.
Revised: We choose ε = 3 on the sample level protection for all results in Tabs.

Original: 3, 4, and App. H.
Revised: 3, 4, and App. H.

Original: Additionally, we conduct empirical comparisons and analyses of SeqPATE versus the original PATE in App.
Revised: Additionally, we conduct empirical comparisons and analyses of SeqPATE versus the original PATE in App.

Original: N.
Revised: N.

Original: We show the effects of protections on users’ secret phrases in App.
Revised: We show the effects of protections on users’ secret phrases in App.

Original: O. We compare
Revised: O. We compare

Original: SeqPATE with another non-DP based baseline (i.e. blacklist based filtering) in App.
Revised: SeqPATE with another non-DP based baseline (i.e. blacklist based filtering) in App.

Original: P. We also conduct a case study in App.
Revised: P. We also conduct a case study in App.

Original: Q.
Revised: Q.

Original:  Related Work
Revised:  Related Work

Original: Text generation models may leak user information through the generated texts [19, 7].
Revised: Text generation models may leak user information through the generated texts [19, 7].

Original: One direction of privacy protection is to protect author-level (user-level) information.
Revised: One direction of privacy protection is to protect author-level (user-level) information.

Original: The methods prevent attackers from inferring the author attributes (e.g., gender, age) [25] and the relationship between information and authors [29].
Revised: The methods prevent attackers from inferring the author attributes (e.g., gender, age) [25] and the relationship between information and authors [29].

Original: Some researchers [40, 41] infer the membership (whether samples from a given author are used to train the model) given a black-box model.
Revised: Some researchers [40, 41] infer the membership (whether samples from a given author are used to train the model) given a black-box model.

Original: Some papers protect user privacy of training data against untrusted servers via federated learning [27, 10].
Revised: Some papers protect user privacy of training data against untrusted servers via federated learning [27, 10].

Original: Another direction is to prevent attackers from extracting sensitive information in training sets by analyzing the outputs [30, 22], which is urgently needed [7].
Revised: Another direction is to prevent attackers from extracting sensitive information in training sets by analyzing the outputs [30, 22], which is urgently needed [7].

Original: Our SeqPATE focuses on this direction.
Revised: Our SeqPATE focuses on this direction.

Original: In this direction, regularization methods [6, 43, 20] restrict the model capacity and prevent the model from memorizing exact training samples.
Revised: In this direction, regularization methods [6, 43, 20] restrict the model capacity and prevent the model from memorizing exact training samples.

Original: Anonymization methods [26, 44] detect sensitive text and replace it with non-sensitive text.
Revised: Anonymization methods [26, 44] detect sensitive text and replace it with non-sensitive text.

Original: Unlike DP [14] methods, the above methods do not provide a quantifiable guarantee for privacy protection.
Revised: Unlike DP [14] methods, the above methods do not provide a quantifiable guarantee for privacy protection.

Original: Some researchers focus on protecting user privacy against untrusted servers via federated learning [27, 10].
Revised: Some researchers focus on protecting user privacy against untrusted servers via federated learning [27, 10].

Original: Some researchers apply DP to text generation.
Revised: Some researchers apply DP to text generation.

Original: For user-level privacy, ER-AE [4] augments the semantic information in the generated text to hide authors’ writing styles from attackers.
Revised: For user-level privacy, ER-AE [4] augments the semantic information in the generated text to hide authors’ writing styles from attackers.

Original: McMahan et al.
Revised: McMahan et al.

Original: [28] propose a recurrent language model with a DP guarantee against the identification of users.
Revised: [28] propose a recurrent language model with a DP guarantee against the identification of users.

Original: Note that the user-level privacy (relationships between users and their information) is different from the privacy of users’ secret phrases in our model: Our model prevents individual user phrases from being detected.
Revised: Note that the user-level privacy (relationships between users and their information) is different from the privacy of users’ secret phrases in our model: Our model prevents individual user phrases from being detected.

Original: Some researchers apply NoisySGD to text generation to prevent sensitive training samples from being extracted: some of them [37, 39] employ DP to protect a part of selected tokens; others [22, 49, 24] apply DP to protect both samples and all tokens, but the privacy cost on tokens is very high (Sec. 5.3).
Revised: Some researchers apply NoisySGD to text generation to prevent sensitive training samples from being extracted: some of them [37, 39, 50] employ DP to protect a part of selected tokens; others [22, 49, 24] apply DP to protect both samples and all tokens, but the privacy cost on tokens is very high (Sec. 5.3).

Original: Our model falls into the latter category and reduces the privacy cost of tokens.
Revised: Our model falls into the latter category and reduces the privacy cost of tokens.

Original: Kerrigan et al.
Revised: Kerrigan et al.

Original: [22] apply NoisySGD [1] to text generation.
Revised: [22] apply NoisySGD [1] to text generation.

Original: Yu et al.
Revised: Yu et al.

Original: [49] investigate fine-tuning strategies on pre-trained language models with NoisySGD.
Revised: [49] investigate fine-tuning strategies on pre-trained language models with NoisySGD.

Original: Li et al.
Revised: Li et al.

Original: [24] apply ghost clipping to pre-trained language models with NoisySGD and reduce memory usage.
Revised: [24] apply ghost clipping to pre-trained language models with NoisySGD and reduce memory usage.

Original: Shi et al.
Revised: Shi et al.

Original: [38] apply DP to particular generation steps instead of training samples or n-grams.
Revised: [38] apply DP to particular generation steps instead of training samples or n-grams.

Original: Brown et al.
Revised: Brown et al.

Original: [5] analyze DP based method versus data sanitization of text generation models.
Revised: [5] analyze DP based method versus data sanitization of text generation models.

Original: Brown et al.
Revised: Brown et al.

Original: [12] propose a efficient NoisySGD to speed up model training.
Revised: [12] propose a efficient NoisySGD to speed up model training.

Original: Differential privacy (DP) [13, 14] formally defines and quantifies privacy.
Revised: Differential privacy (DP) [13, 14] formally defines and quantifies privacy.

Original: ML models with DP guarantee [46, 15, 51] prevent the existence of individual training examples from being detected [6].
Revised: ML models with DP guarantee [46, 15, 52] prevent the existence of individual training examples from being detected [6].

Original: Some researchers protect the privacy of empirical risk minimization classifiers [8] and SVM [36] with DP.
Revised: Some researchers protect the privacy of empirical risk minimization classifiers [8] and SVM [36] with DP.

Original: Following Song et al.
Revised: Following Song et al.

Original: [42], NoisySGD [1] achieves DP on deep learning models by the gradient
Revised: [42], NoisySGD [1] achieves DP on deep learning models by adding noises to gradients.

Revised: Pichapati et al.

Original:  [31, 32] transfers the knowledge from teacher models trained on private sets with noises to a student model.
Revised: [34] adaptively clip the gradient in NoisySGD. PATE [31, 32] transfers the knowledge from teacher models trained on private sets with noises to a student model.

Original: KNN-PATE [50] refines PATE by accessing only the k-nearest neighbors from the private set.
Revised: KNN-PATE [51] refines PATE by accessing only the k-nearest neighbors from the private set.

Original: Jordon et al.
Revised: Jordon et al.

Original: [21] adversarially learn to generate synthetic data with discriminators trained by PATE.
Revised: [21] adversarially learn to generate synthetic data with discriminators trained by PATE.

Original: These methods are not customized for text generation models.
Revised: These methods are not customized for text generation models.

Original: Xie et al.
Revised: Xie et al.

Original: [48] propose DPGAN to adversarially learn with a generator and a discriminator.
Revised: [48] propose DPGAN to adversarially learn with a generator and a discriminator.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training data for text generation models with DP guarantees.
Revised: In this paper, we propose a novel framework, SeqPATE, to protect the privacy of the training data for text generation models with DP guarantees.

Original: SeqPATE achieves a good privacy-utility trade-off by leveraging both private and public data.
Revised: SeqPATE achieves a good privacy-utility trade-off by leveraging both private and public data.

Original: As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and is therefore adaptive to text generation models.
Revised: As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and is therefore adaptive to text generation models.

Original: We avoid rolling out teachers by providing pseudo-inputs for the teacher’s inference and the student’s training.
Revised: We avoid rolling out teachers by providing pseudo-inputs for the teacher’s inference and the student’s training.

Original: We further reduce the output space by candidate filtering and limit privacy losses via efficient knowledge distillation.
Revised: We further reduce the output space by candidate filtering and limit privacy losses via efficient knowledge distillation.

Original: SeqPATE achieves a better performance with the sample-level protection and further provides much stronger protection on users’ secret phrases.
Revised: SeqPATE achieves a better performance with the sample-level protection and further provides much stronger protection on users’ secret phrases.

Original: The limitations, ethical considerations, and social impacts of this paper are in App.
Revised: The limitations, ethical considerations, and social impacts of this paper are in App.

Original: A and L.
Revised: A and L.

Original:  Acknowledgement
Revised:  Acknowledgement

Original: Research in this paper was supported by Hong Kong Research Grants Council under grand No.
Revised: Research in this paper was supported by Hong Kong Research Grants Council under grand No.

Original: HH is partly supported by the Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI).
Revised: HH is partly supported by the Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI).

Revised: YW is partially supported by NSF Award #2048091.

Original: The authors thank Mr. Wei Dong and Dr. Yiping Song for their help and insights on this paper.
Revised: The authors thank Mr. Wei Dong and Dr. Yiping Song for their help and insights on this paper.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] It can be found in the body paper or the supplemental materials.
Revised: [Yes] It can be found in the body paper or the supplemental materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] It can be found in the body paper or the supplemental materials.
Revised: [Yes] It can be found in the body paper or the supplemental materials.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] It can be found in the supplemental materials.
Revised: [Yes] It can be found in the supplemental materials.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]