Original: S EQ PATE: D IFFERENTIALLY P RIVATE T EXT G ENERATION VIA K NOWLEDGE D ISTILLATION
Revised: S EQ PATE: D IFFERENTIALLY P RIVATE T EXT G ENERATION VIA K NOWLEDGE D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Protecting the privacy of user data is crucial when training neural text generation models, which may leak sensitive user information during generation.
Revised: Protecting the privacy of user data is crucial when training neural text generation models, which may leak sensitive user information during generation.

Original: Differentially private (DP) learning algorithms provide guarantees on identifying the existence of a training sample from model outputs.
Revised: Differentially private (DP) learning algorithms provide guarantees on identifying the existence of a training sample from model outputs.

Original: PATE is a DP learning algorithm that fits large models well, such as GPT.
Revised: PATE is a DP learning algorithm that fits large models well, such as GPT.

Original: In this paper, we propose SeqPATE that adapts
Revised: In this paper, we propose SeqPATE that adapts

Original: PATE to text generation while satisfying DP.
Revised: PATE to text generation while satisfying DP.

Original: There are two key challenges in adapting PATE to text generation: (i) obtaining sequence-level supervision for text generation, and (ii) reducing noise required to protect the privacy given a large output space (i.e. vocabulary size).
Revised: There are two key challenges in adapting PATE to text generation: (i) obtaining sequence-level supervision for text generation, and (ii) reducing noise required to protect the privacy given a large output space (i.e. vocabulary size).

Original: For (i), we generate pseudo input and reduce the sequence generation problem to the next word prediction.
Revised: For (i), we generate pseudo input and reduce the sequence generation problem to the next word prediction.

Original: For (ii), we reduce the output space with top- k and top- p selection strategy that dynamically filters the candidate words; and we refine the teacher aggregation of PATE to avoid the low agreement rates due to voting over the large output space.
Revised: For (ii), we reduce the output space with top- k and top- p selection strategy that dynamically filters the candidate words; and we refine the teacher aggregation of PATE to avoid the low agreement rates due to voting over the large output space.

Original: To limit the privacy loss, we design an efficient knowledge distillation to reduce the frequency of distilling from the private data.
Revised: To limit the privacy loss, we design an efficient knowledge distillation to reduce the frequency of distilling from the private data.

Original: We apply SeqPATE to a simple text generation task (sentence completion) and achieve 39% and 28% gains in Bleu4 on two datasets.
Revised: We apply SeqPATE to a simple text generation task (sentence completion) and achieve 39% and 28% gains in Bleu4 on two datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent work showed that sensitive user information in training corpora, such as address and name, can be extracted from text generation models (Carlini et al., 2019; 2020).
Revised: Recent work showed that sensitive user information in training corpora, such as address and name, can be extracted from text generation models (Carlini et al., 2019; 2020).

Original: Providing privacy guarantees to the text used to train text generation models has become a critical problem.
Revised: Providing privacy guarantees to the text used to train text generation models has become a critical problem.

Original: Differential privacy (DP) provides provable guarantees against the identification of individuals in the dataset.
Revised: Differential privacy (DP) provides provable guarantees against the identification of individuals in the dataset.

Original: Deep learning models with DP guarantees ensure the existence of a specific training sample cannot be detected.
Revised: Deep learning models with DP guarantees ensure the existence of a specific training sample cannot be detected.

Original: The most popular DP algorithm for deep learning is Noisy-SGD
Revised: The most popular DP algorithm for deep learning is Noisy-SGD

Original: (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016), which adds noise to the gradients.
Revised: (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016), which adds noise to the gradients.

Original: However, the advantage of NoisySGD is weaken as the model becomes larger (Yu et al., 2020), and NoisySGD requires a per-example gradient clip, which causes non-convergence and system overheads (Zhu et al., 2020; Bu et al., 2021).
Revised: However, the advantage of NoisySGD is weaken as the model becomes larger (Yu et al., 2020), and NoisySGD requires a per-example gradient clip, which causes non-convergence and system overheads (Zhu et al., 2020; Bu et al., 2021).

Original: Applying Noisy-SGD to large text generation models (e.g. GPT-2) requires additional tricks such as memory reduction (Li et al., 2021) and advanced fine-tuning strategies (Yu et al., 2021).
Revised: Applying Noisy-SGD to large text generation models (e.g. GPT-2) requires additional tricks such as memory reduction (Li et al., 2021) and advanced fine-tuning strategies (Yu et al., 2021).

Original: PATE (Papernot et al., 2017) is a DP learning model using a teacher-student framework: a student accessing non-sensitive data distills knowledge from the aggregated predictions of multiple teachers trained on sensitive data.
Revised: PATE (Papernot et al., 2017) is a DP learning model using a teacher-student framework: a student accessing non-sensitive data distills knowledge from the aggregated predictions of multiple teachers trained on sensitive data.

Original: Calibrated noise is added to the aggregated predictions to satisfy DP.
Revised: Calibrated noise is added to the aggregated predictions to satisfy DP.

Original: PATE can handle large models since its privacy cost derives from the knowledge distillation instead of the model parameters.
Revised: PATE can handle large models since its privacy cost derives from the knowledge distillation instead of the model parameters.

Original: PATE takes advantage of non-sensitive data which is also available in our scenario.
Revised: PATE takes advantage of non-sensitive data which is also available in our scenario.

Original: In this paper, we propose, SeqPATE, a DP learning algorithm for text generation that exploits both sensitive and non-sensitive data by knowledge distillation.
Revised: In this paper, we propose, SeqPATE, a DP learning algorithm for text generation that exploits both sensitive and non-sensitive data by knowledge distillation.

Original: Specifically, we adapt PATE for text generation.
Revised: Specifically, we adapt PATE for text generation.

Original: The adaptation is challenging due to the sequential generation and the large output space (i.e. vocabulary size) in text generation.
Revised: The adaptation is challenging due to the sequential generation and the large output space (i.e. vocabulary size) in text generation.

Original: Firstly, to obtain the sentence-level supervision for text generation, directly applying PATE needs to roll out all teachers to produce a sentence (i.e. all teachers must vote to generate a word, which is then used as the input for the next word prediction).
Revised: Firstly, to obtain the sentence-level supervision for text generation, directly applying PATE needs to roll out all teachers to produce a sentence (i.e. all teachers must vote to generate a word, which is then used as the input for the next word prediction).

Original: Running inference for a large number of teachers can be costly.
Revised: Running inference for a large number of teachers can be costly.

Original: Secondly, the large output space results in (i) low agreement rate among teachers and (ii) large noise required by DP, both would significantly hurt task performance.
Revised: Secondly, the large output space results in (i) low agreement rate among teachers and (ii) large noise required by DP, both would significantly hurt task performance.

Original: To avoid sequential generation, we generate pseudo data using a pre-trained language model such that teachers only need to provide token-level supervision given the pseudo input.
Revised: To avoid sequential generation, we generate pseudo data using a pre-trained language model such that teachers only need to provide token-level supervision given the pseudo input.

Original: To handle the large output space, we aggregate teachers’ outputs by interpolating their output distributions instead of voting with argmax predictions.
Revised: To handle the large output space, we aggregate teachers’ outputs by interpolating their output distributions instead of voting with argmax predictions.

Original: To reduce the noise scale, we propose strategies to dynamically filter candidate words that only keep words with large probabilities.
Revised: To reduce the noise scale, we propose strategies to dynamically filter candidate words that only keep words with large probabilities.

Original: Further, we design an efficient knowledge distillation strategy that only queries the teacher when the student has poor performance.
Revised: Further, we design an efficient knowledge distillation strategy that only queries the teacher when the student has poor performance.

Original: We evaluate our method on sentence completion, a simple but representative task of text generation.
Revised: We evaluate our method on sentence completion, a simple but representative task of text generation.

Original: Compared to baselines including Noisy-SGD (Kerrigan et al., 2020), SeqPATE improves Bleu4 by 39% and 28% on two datasets with meaningful privacy protection 1 .
Revised: Compared to baselines including Noisy-SGD (Kerrigan et al., 2020), SeqPATE improves Bleu4 by 39% and 28% on two datasets with meaningful privacy protection 1 .

Original: We observe SeqPATE works particularly well under relatively strong privacy protection (e.g., ε ≤ 2 ) compared to Noisy-SGD.
Revised: We observe SeqPATE works particularly well under relatively strong privacy protection (e.g., ε ≤ 2 ) compared to Noisy-SGD.

Original: Our contribution is threefold: (i) Our model significantly surpasses baselines and makes DP algorithm achieve satisfactory performance on text generation; (ii) We propose several practical strategies to enable SeqPATE to handle text generation with a sequential of classifications over larger output spaces; (iii) We are the first to adapt PATE, an effective DP algorithm, to text generation.
Revised: Our contribution is threefold: (i) Our model significantly surpasses baselines and makes DP algorithm achieve satisfactory performance on text generation; (ii) We propose several practical strategies to enable SeqPATE to handle text generation with a sequential of classifications over larger output spaces; (iii) We are the first to adapt PATE, an effective DP algorithm, to text generation.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: Our goal is to achieve the privacy protection defined by DP in text generation: preventing attackers from inferring whether a sample or an n-gram appears in the training set.
Revised: Our goal is to achieve the privacy protection defined by DP in text generation: preventing attackers from inferring whether a sample or an n-gram appears in the training set.

Original: (Formal definition in Sec. 5)
Revised: (Formal definition in Sec. 5)

Original: PATE assumes students are trained on unlabeled non-sensitive data.
Revised: PATE assumes students are trained on unlabeled non-sensitive data.

Original: Similar to PATE, our setting contains two textual datasets: 1.
Revised: Similar to PATE, our setting contains two textual datasets: 1.

Original: a private set D pri from a corpus with sensitive information.
Revised: a private set D pri from a corpus with sensitive information.

Original: a public set D pub without sensitive information (e.g. incomplete or masked text spans) or data contributors (e.g. volunteers) have no objection to publishing their data.
Revised: a public set D pub without sensitive information (e.g. incomplete or masked text spans) or data contributors (e.g. volunteers) have no objection to publishing their data.

Original: Algorithms are required to protect the privacy on the private set and can ignore the privacy protection on the public set.
Revised: Algorithms are required to protect the privacy on the private set and can ignore the privacy protection on the public set.

Original: The final goal is to do well on the private set during inference.
Revised: The final goal is to do well on the private set during inference.

Original: Text generation is to generate a sequence of words S
Revised: Text generation is to generate a sequence of words S

Original: = { w 1 , w 2 , . . .
Revised: = { w 1 , w 2 , . . .

Original: , w l } .
Revised: , w l } .

Original: Our application is a simple text generation task, sentence completion, which aims to generate the remaining part of a sentence w 1: l given its prefix p 1: l ′ , where p 1: l ′ and w 1: l are the abbreviation of { p 1 , p 2 , . .
Revised: Our application is a simple text generation task, sentence completion, which aims to generate the remaining part of a sentence w 1: l given its prefix p 1: l ′ , where p 1: l ′ and w 1: l are the abbreviation of { p 1 , p 2 , . .

Original: . , p l ′ } and { w 1 , w 2 , . . .
Revised: . , p l ′ } and { w 1 , w 2 , . . .

Original: , w l } respectively.
Revised: , w l } respectively.

Original: p 1: l ′ and w 1: l compose a complete sentence.
Revised: p 1: l ′ and w 1: l compose a complete sentence.

Original: We collect some input prefixes p 1: l ′ to compose the public set D pub and the original samples (complete sentences) compose the private set D pri .
Revised: We collect some input prefixes p 1: l ′ to compose the public set D pub and the original samples (complete sentences) compose the private set D pri .

Original: Such a setting fits for some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses.
Revised: Such a setting fits for some real-world text generation applications: in dialog systems, the training samples from online services consist of questions and responses.

Original: The questions from customer service staffs or service robots can be public and the response from users carrying individual information should be private.
Revised: The questions from customer service staffs or service robots can be public and the response from users carrying individual information should be private.

Original: In writing assistants where a sample has an input prompt and an essay body, the prompt can be public but the body written by data contributors is private.
Revised: In writing assistants where a sample has an input prompt and an essay body, the prompt can be public but the body written by data contributors is private.

Original:  B ACKGROUND ON DP AND PATE
Revised:  B ACKGROUND ON DP AND PATE

Original: Differential privacy (DP) (Dwork et al., 2006; 2014) is a quantifiable definition of privacy that provides provable guarantees on identifications of individuals in the dataset.
Revised: Differential privacy (DP) (Dwork et al., 2006; 2014) is a quantifiable definition of privacy that provides provable guarantees on identifications of individuals in the dataset.

Original: ML algorithms with DP guarantee ensure that each individual training sample has a degree of plausible deniability , i.e., the trained model is just as likely as to have been trained on an alternative dataset without that sample.
Revised: ML algorithms with DP guarantee ensure that each individual training sample has a degree of plausible deniability , i.e., the trained model is just as likely as to have been trained on an alternative dataset without that sample.

Original: PATE, designed for classification tasks, takes advantage of an unlabeled public dataset D pub and also trains on a labeled private D pri in a semi-supervised scenario.
Revised: PATE, designed for classification tasks, takes advantage of an unlabeled public dataset D pub and also trains on a labeled private D pri in a semi-supervised scenario.

Original: PATE is model-agnostic and treats models as black-boxes.
Revised: PATE is model-agnostic and treats models as black-boxes.

Original: PATE (Papernot et al., 2017) achieves DP via a teacher-student framework with M teacher models and a student model, where the student learns from the private data via knowledge distillation through teachers.
Revised: PATE (Papernot et al., 2017) achieves DP via a teacher-student framework with M teacher models and a student model, where the student learns from the private data via knowledge distillation through teachers.

Original: PATE consists of three parts,
Revised: PATE consists of three parts,

Original: • Teacher models are trained on the private set D pri .
Revised: • Teacher models are trained on the private set D pri .

Original: D pri is shuffled and divided into M disjoint subsets, and m -th subset serves as the training set for m -th teacher model f mϕ .
Revised: D pri is shuffled and divided into M disjoint subsets, and m -th subset serves as the training set for m -th teacher model f mϕ .

Original: • Teacher aggregation merges teachers’ outputs.
Revised: • Teacher aggregation merges teachers’ outputs.

Original: After teachers’ training, each teacher f mϕ conducts the prediction
Revised: After teachers’ training, each teacher f mϕ conducts the prediction

Original: classification
Revised: classification

Original: D aggregates M
Revised: D aggregates M

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: GPT
Revised: GPT

Original: [Figure]
Revised: [Figure]

Original: +Noise
Revised: +Noise

Original: Teacher
Revised: Teacher

Original: Teacher Teacher
Revised: Teacher Teacher

Original: Student
Revised: Student

Original: Public data
Revised: Public data

Original: NLL loss
Revised: NLL loss

Original: Data Data
Revised: Data Data

Original: Data
Revised: Data

Original: Cats sit on the Cats sit on the
Revised: Cats sit on the Cats sit on the

Original: Cats sit on the
Revised: Cats sit on the

Original: Top-K / Top-P Selection |Vocabulary|
Revised: Top-K / Top-P Selection |Vocabulary|

Original: Sit 0.42 Sat 0.

Revised: Sit 0.42 Sat 0.

Original: Lay 0.10 Are 0.
Revised: Lay 0.10 Are 0.

Original: Alaia 1e-6 Ruth 1eTop-K
Revised: Alaia 1e-6 Ruth 1eTop-K

Original: Aggregate
Revised: Aggregate

Original: Teacher
Revised: Teacher

Original: [Figure]
Revised: [Figure]

Original: sit on the mats
Revised: sit on the mats

Original: Selection For Efficient Knowledge Distillation (KD)
Revised: Selection For Efficient Knowledge Distillation (KD)

Original: Teacher
 Teacher
Revised: Teacher
 Teacher

Original: [Figure]
Revised: [Figure]

Original: Cats sit
Revised: Cats sit

Original: Cats sit on the mats
Revised: Cats sit on the mats

Original: [Figure]
Revised: [Figure]

Original: Student
Revised: Student

Original: [Figure]
Revised: [Figure]

Original: teachers’ by voting with their top-1 predictions.
Revised: teachers’ by voting with their top-1 predictions.

Original: PATE adds noise to aggregated teacher outputs so that attackers cannot access the exact outputs to infer original information in samples.
Revised: PATE adds noise to aggregated teacher outputs so that attackers cannot access the exact outputs to infer original information in samples.

Original: • A student model f θ is trained on the public set D pub and distills knowledge from private set (via the teachers) without accessing the private samples.
Revised: • A student model f θ is trained on the public set D pub and distills knowledge from private set (via the teachers) without accessing the private samples.

Original: In this way, the sensitive information in the private set is preserved even if the student’s architecture and parameters are public or reverse-engineered by an adversary.
Revised: In this way, the sensitive information in the private set is preserved even if the student’s architecture and parameters are public or reverse-engineered by an adversary.

Original: We publish only the student model for prediction.
Revised: We publish only the student model for prediction.

Original:  A PPROACH
Revised:  A PPROACH

Original: Fig.
Revised: Fig.

Original: 1 shows an overview of the SeqPATE.
Revised: 1 shows an overview of the SeqPATE.

Original: Given the public prefix (e.g., “Cats sit”), we first obtain pseudo inputs by generating the remaining part of the whole sentence (e.g., “Cats sit on the mats”) using a pre-trained language model (Sec. 4.1).
Revised: Given the public prefix (e.g., “Cats sit”), we first obtain pseudo inputs by generating the remaining part of the whole sentence (e.g., “Cats sit on the mats”) using a pre-trained language model (Sec. 4.1).

Original: The teachers’ output distributions are then averaged to provide word-level supervision on the whole sentence (Sec. 4.2).
Revised: The teachers’ output distributions are then averaged to provide word-level supervision on the whole sentence (Sec. 4.2).

Original: To reduce the noise over large output space and limit the privacy loss, we propose top- k / top- p selection to dynamically filter the unimportant candidates and propose efficient knowledge distillation to reduce the number of teacher queries (Sec. 4.3).
Revised: To reduce the noise over large output space and limit the privacy loss, we propose top- k / top- p selection to dynamically filter the unimportant candidates and propose efficient knowledge distillation to reduce the number of teacher queries (Sec. 4.3).

Original: Our training algorithm is shown in Appendix A.
Revised: Our training algorithm is shown in Appendix A.

Original: 4. S TUDENT M ODEL
Revised: 4. S TUDENT M ODEL

Original: Conventional text generation models generate words in a sentence step-by-step and left-to-right.
Revised: Conventional text generation models generate words in a sentence step-by-step and left-to-right.

Original: Naively applying PATE to the above paradigm requires all the teachers to be roll out to obtain supervision at every step, where all the teachers aggregate to generate a word at each step and the word acts as the next input for all teachers.
Revised: Naively applying PATE to the above paradigm requires all the teachers to be roll out to obtain supervision at every step, where all the teachers aggregate to generate a word at each step and the word acts as the next input for all teachers.

Original: This means we should conduct all teachers’ inferences simultaneously and interdependently, which is costly in both computation (involving hundreds of teachers) and privacy costs.
Revised: This means we should conduct all teachers’ inferences simultaneously and interdependently, which is costly in both computation (involving hundreds of teachers) and privacy costs.

Original: Hence, in SeqPATE, we employ GPT to generate pseudo sentences and conduct teacher inference and student training on the pseudo sentences.
Revised: Hence, in SeqPATE, we employ GPT to generate pseudo sentences and conduct teacher inference and student training on the pseudo sentences.

Original: Firstly, given a prefix in D pub , we employ the public released pre-trained GPT to conduct the inference to complete a pseudo sentence.
Revised: Firstly, given a prefix in D pub , we employ the public released pre-trained GPT to conduct the inference to complete a pseudo sentence.

Original: All the generated pseudo sentences compose the student’s training set, denoted as ˜ D pub .
Revised: All the generated pseudo sentences compose the student’s training set, denoted as ˜ D pub .

Original: Secondly, we conduct teacher inference and student training on the sample S
Revised: Secondly, we conduct teacher inference and student training on the sample S

Original: = { w 1 , w 2 , . .
Revised: = { w 1 , w 2 , . .

Original: , w | S | } from ˜ D pub .
Revised: , w | S | } from ˜ D pub .

Original: At the i -th step, for the training of conventional text generations, the student learns to generate a word ˆ w i given its previous words w 1: i − 1 .
Revised: At the i -th step, for the training of conventional text generations, the student learns to generate a word ˆ w i given its previous words w 1: i − 1 .

Original: At the i -th step, our teachers conduct the inference to generate ˆ w i given the input w 1: i − 1 from the sample S .
Revised: At the i -th step, our teachers conduct the inference to generate ˆ w i given the input w 1: i − 1 from the sample S .

Original: Then, each teacher provides its output for aggregation.
Revised: Then, each teacher provides its output for aggregation.

Original: All teachers share the same input with the student so that the outputs of teachers and student are aligned and comparable.
Revised: All teachers share the same input with the student so that the outputs of teachers and student are aligned and comparable.

Original: 4. T EACHER A GGREGATION
Revised: 4. T EACHER A GGREGATION

Original: The original PATE aggregates teacher results by taking the majority vote of the predicted labels from all the teachers.
Revised: The original PATE aggregates teacher results by taking the majority vote of the predicted labels from all the teachers.

Original: However, this aggregation strategy does not fit for the large output space in text generation.
Revised: However, this aggregation strategy does not fit for the large output space in text generation.

Original: This is because the number of votes for each candidate may be very low and it would result in low agreement.
Revised: This is because the number of votes for each candidate may be very low and it would result in low agreement.

Original: For example, multiple candidates may tie for the top-1 prediction.
Revised: For example, multiple candidates may tie for the top-1 prediction.

Original: Therefore, inspired by (Hinton et al., 2015; Chen et al., 2020), we aggregate M teachers’ results by averaging their output distributions, where each teacher model is trained on one of the disjoint subsets divided from the private set D pri .
Revised: Therefore, inspired by (Hinton et al., 2015; Chen et al., 2020), we aggregate M teachers’ results by averaging their output distributions, where each teacher model is trained on one of the disjoint subsets divided from the private set D pri .

Original: At the i -th step on teacher inference, the m -th teacher model f mϕ predicts a word ˆ w i with a probability p mϕ ( ˆ w i ) .
Revised: At the i -th step on teacher inference, the m -th teacher model f mϕ predicts a word ˆ w i with a probability p mϕ ( ˆ w i ) .

Original: Then, our model aggregates all teachers’ probabilities by averaging p ϕ ( ˆ w i )
Revised: Then, our model aggregates all teachers’ probabilities by averaging p ϕ ( ˆ w i )

Original: =
Revised: =

Original: 1 M (cid:80) Mm =1 p
Revised: 1 M (cid:80) Mm =1 p

Original: mϕ ( ˆ w i ) .
Revised: mϕ ( ˆ w i ) .

Original: Like most DP methods, our model adds noises so that the privacy impact can be analyzed and bounded.
Revised: Like most DP methods, our model adds noises so that the privacy impact can be analyzed and bounded.

Original: Different from PATE that adds noise to vote numbers, we add noise to the aggregation of teachers’ output distributions.
Revised: Different from PATE that adds noise to vote numbers, we add noise to the aggregation of teachers’ output distributions.

Original: Following (Papernot et al., 2018), we apply the Gaussian mechanism which adds i.i.d.
Revised: Following (Papernot et al., 2018), we apply the Gaussian mechanism which adds i.i.d.

Original: N (0 , σ 2 ) to each coordinate, where σ controls the strength of privacy protection (more details in Sec. 5).
Revised: N (0 , σ 2 ) to each coordinate, where σ controls the strength of privacy protection (more details in Sec. 5).

Original: 4. S UPERVISION ON THE S TUDENT M ODEL
Revised: 4. S UPERVISION ON THE S TUDENT M ODEL

Original: Reducing output space via top- k or top- p selection.
Revised: Reducing output space via top- k or top- p selection.

Original: Recall that the teacher output is a distribution over all words in the vocabulary, and the noise must be added to each coordinate to satisfy DP.
Revised: Recall that the teacher output is a distribution over all words in the vocabulary, and the noise must be added to each coordinate to satisfy DP.

Original: To reduce the output dimension (hence the amount of noise), we only assign the probability mass to a subset of candidates W i .
Revised: To reduce the output dimension (hence the amount of noise), we only assign the probability mass to a subset of candidates W i .

Original: For all teachers and the student, we assign zero probability mass to any candidates not in W i and re-normalize distribution as shown in Eq.
Revised: For all teachers and the student, we assign zero probability mass to any candidates not in W i and re-normalize distribution as shown in Eq.

Original: 1, where g ( · ) is the normalizing function.
Revised: 1, where g ( · ) is the normalizing function.

Original: This operation means the words not in W i will never occur in student’s training and teachers’ supervision, so those words do not need noise to provide protection.
Revised: This operation means the words not in W i will never occur in student’s training and teachers’ supervision, so those words do not need noise to provide protection.

Original: [Equation]
Revised: [Equation]

Original: We determine the subset W i by two selection strategies: top- k and top- p .
Revised: We determine the subset W i by two selection strategies: top- k and top- p .

Original: In the top- k selection, at i -th step, we select the top- k candidates according to the probabilities generated by the student model.
Revised: In the top- k selection, at i -th step, we select the top- k candidates according to the probabilities generated by the student model.

Original: Further, inspired by nucleus sampling (Holtzman et al., 2019), we propose a top- p selection that chooses the minimum k where the cumulative probability of the top- k words is larger than a constant close to 1 (e.g. 0.95) such that the candidate set covers a large proportion of the distribution.
Revised: Further, inspired by nucleus sampling (Holtzman et al., 2019), we propose a top- p selection that chooses the minimum k where the cumulative probability of the top- k words is larger than a constant close to 1 (e.g. 0.95) such that the candidate set covers a large proportion of the distribution.

Original: We note that the candidate selection is based on the student’s output on public or already released input, thus not affecting the privacy guarantee.
Revised: We note that the candidate selection is based on the student’s output on public or already released input, thus not affecting the privacy guarantee.

Original: This choice improves the privacy-utility tradeoff by adaptively allocating the available privacy budgets to release the information more relevant to the task.
Revised: This choice improves the privacy-utility tradeoff by adaptively allocating the available privacy budgets to release the information more relevant to the task.

Original: Efficient knowledge distillation.
Revised: Efficient knowledge distillation.

Original: Reducing the number of querying teachers results in better privacy protection.
Revised: Reducing the number of querying teachers results in better privacy protection.

Original: Therefore, we acquire teacher supervision only when the student has poor performance.
Revised: Therefore, we acquire teacher supervision only when the student has poor performance.

Original: Concretely, we measure student’s performance on each token by comparing its output distribution with the ground truth token (from ˜ D pub ).
Revised: Concretely, we measure student’s performance on each token by comparing its output distribution with the ground truth token (from ˜ D pub ).

Original: Let r w
Revised: Let r w

Original: i be the rank of the ground truth token w i in the student’s predicted distribution, (i.e. the number of tokens with higher or equal probability).
Revised: i be the rank of the ground truth token w i in the student’s predicted distribution, (i.e. the number of tokens with higher or equal probability).

Original: The student model acquires the teachers’ supervision only if r w i is larger than a certain threshold r c .
Revised: The student model acquires the teachers’ supervision only if r w i is larger than a certain threshold r c .

Original: We note that the selection of tokens relies only on the student and is independent of the teachers, thus the selection does not cause additional privacy loss.
Revised: We note that the selection of tokens relies only on the student and is independent of the teachers, thus the selection does not cause additional privacy loss.

Original: Training objectives.
Revised: Training objectives.

Original: Different from the semi-supervised setting in PATE, our student model is supervised by both teachers’ aggregation and labels from the pseudo public set ˜ D pub , since text generation corpora are naturally labeled, unlike in classification tasks.
Revised: Different from the semi-supervised setting in PATE, our student model is supervised by both teachers’ aggregation and labels from the pseudo public set ˜ D pub , since text generation corpora are naturally labeled, unlike in classification tasks.

Original: Even if the pseudo set ˜ D pub provides only pseudo samples, the labels from those samples are also useful given that the number of querying teachers is limited due to privacy concerns
Revised: Even if the pseudo set ˜ D pub provides only pseudo samples, the labels from those samples are also useful given that the number of querying teachers is limited due to privacy concerns

Original: • L KL i shows teachers’ supervisions.
Revised: • L KL i shows teachers’ supervisions.

Original: The student learns from teachers by minimizing KL divergence between the student output distribution p θ and the merged teacher output distributions p ϕ .
Revised: The student learns from teachers by minimizing KL divergence between the student output distribution p θ and the merged teacher output distributions p ϕ .

Original: • L NLL i shows the supervisions from the pseudo public set ˜ D pub generated by GPT.
Revised: • L NLL i shows the supervisions from the pseudo public set ˜ D pub generated by GPT.

Original: As conventional
Revised: As conventional

Original: language
Revised: language

Original: models, our student model is trained by negative log-likelihood
Revised: models, our student model is trained by negative log-likelihood

Original: (NLL)
Revised: (NLL)

Original: loss on ˜ D pub .
Revised: loss on ˜ D pub .

Original: The
Revised: The

Original: words w
Revised: words w

Original: i in the training sample is naturally the label for the student f θ ( w 1: i −
Revised: i in the training sample is naturally the label for the student f θ ( w 1: i −

Original: at the i -th step
Revised: at the i -th step

Original: [Equation]
Revised: [Equation]

Original: where L i is loss at i -th step, | S | denotes the length of sentence S , and λ balances the two terms.
Revised: where L i is loss at i -th step, | S | denotes the length of sentence S , and λ balances the two terms.

Original: Meanwhile, the noise scale σ is discussed in Sec.
Revised: Meanwhile, the noise scale σ is discussed in Sec.

Original: g ( · ) is the re-normalization function used in
Revised: g ( · ) is the re-normalization function used in

Original: Eq.
Revised: Eq.

Original: 1 that re-normalizes the aggregated probabilities on the selected top- k or top- p words with noise 2 .
Revised: 1 that re-normalizes the aggregated probabilities on the selected top- k or top- p words with noise 2 .

Original: According to the efficient knowledge distillation strategy, at each step, the loss L i is the combination of L NLL i and L KL i if the student needs teachers’ supervision; otherwise, L i =
Revised: According to the efficient knowledge distillation strategy, at each step, the loss L i is the combination of L NLL i and L KL i if the student needs teachers’ supervision; otherwise, L i =

Original: L NLL i .
Revised: L NLL i .

Original:  P RIVACY A NALYSIS 5. P RELIMINARY OF D IFFERENTIAL P RIVACY
Revised:  P RIVACY A NALYSIS 5. P RELIMINARY OF D IFFERENTIAL P RIVACY

Original: Let D , D ′ denote two neighboring datasets which differ at only one individual.
Revised: Let D , D ′ denote two neighboring datasets which differ at only one individual.

Original: Definition 1 (Differential privacy) .
Revised: Definition 1 (Differential privacy) .

Original: For ε > 0 and δ ≥ 0 , a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if for any neighboring datasets D ∼ D ′ and any S ⊆ Y ,
Revised: For ε > 0 and δ ≥ 0 , a randomized algorithm M : X n → Y is ( ε, δ ) -differentially private if for any neighboring datasets D ∼ D ′ and any S ⊆ Y ,

Original: [Equation]
Revised: [Equation]

Original: The definition ensures that it is information-theoretically impossible for an adversary to infer whether the input dataset is D or D ′ even with arbitrary side information.
Revised: The definition ensures that it is information-theoretically impossible for an adversary to infer whether the input dataset is D or D ′ even with arbitrary side information.

Original: The definition is also future proof thanks to its closure to post-processing.
Revised: The definition is also future proof thanks to its closure to post-processing.

Original: Lemma 2 (Post-processing) .
Revised: Lemma 2 (Post-processing) .

Original: If M obeys ( ε, δ ) -DP, then for any function f , f ◦ M is also ( ε, δ ) -DP.
Revised: If M obeys ( ε, δ ) -DP, then for any function f , f ◦ M is also ( ε, δ ) -DP.

Original: One notable property of DP is that it automatically protects the privacy of groups of multiple units.
Revised: One notable property of DP is that it automatically protects the privacy of groups of multiple units.

Original: Lemma 3 (Group privacy) .
Revised: Lemma 3 (Group privacy) .

Original: An ( ε, δ ) -DP mechanism M on individuals is ( kε, e kε − 1 e ε − 1 δ ) -DP on groups of size k for all integers k ≥ 1 .
Revised: An ( ε, δ ) -DP mechanism M on individuals is ( kε, e kε − 1 e ε − 1 δ ) -DP on groups of size k for all integers k ≥ 1 .

Original: Lemma 4 (Analytical Gaussian mechanism (Balle & Wang, 2018)) .
Revised: Lemma 4 (Analytical Gaussian mechanism (Balle & Wang, 2018)) .

Original: For a numeric query f : X n → R d over a dataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies ( ε, δ ( ε ))
Revised: For a numeric query f : X n → R d over a dataset D , the randomized algorithm that outputs f ( D ) + Z where Z ∼ N (0 , σ 2 I d ) satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) =
Revised: -DP for all ε ≥ 0 and δ ( ε ) =

Original: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .
Revised: Φ( ∆2 σ − εσ ∆ ) − e ε Φ( − ∆2 σ − εσ ∆ ) .

Original: where ∆ := ∆ ( f )= max D∼D ′ ∥
Revised: where ∆ := ∆ ( f )= max D∼D ′ ∥

Original: f ( D ) − f ( D ′ )
Revised: f ( D ) − f ( D ′ )

Original: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .
Revised: ∥ 2 is the global L2 sensitivity of f and Φ is the CDF function of N (0 , 1) .

Original: The above is the ( ε, δ ) -DP of a single Gaussian mechanism, and the following lemma shows that we can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.
Revised: The above is the ( ε, δ ) -DP of a single Gaussian mechanism, and the following lemma shows that we can use the same result for an adaptive composition of a sequence of Gaussian mechanisms.

Original: Lemma 5 (Composition of Gaussian mechanisms (Dong et al., 2019)) .
Revised: Lemma 5 (Composition of Gaussian mechanisms (Dong et al., 2019)) .

Original: The adaptive composition of a sequence of Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .
Revised: The adaptive composition of a sequence of Gaussian mechanisms with a noise level σ 1 , σ 2 , . . .

Original: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .
Revised: and global L2 sensitivity ∆ 1 , ∆ 2 , . . .

Original: satisfies ( ε, δ ( ε ))
Revised: satisfies ( ε, δ ( ε ))

Original: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noise multiplier
Revised: -DP for all ε ≥ 0 and δ ( ε ) ≤ δ M ( ε ) where M is a Gaussian mechanism with noise multiplier

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:0)(cid:80)
Revised: (cid:0)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: Specifically, the adaptive composition of a k identical Gaussian mechanism with noise multiplier σ satisfies the same privacy guarantee of that of a single Gaussian mechanism with a noise multiplier σ/ √ k .
Revised: Specifically, the adaptive composition of a k identical Gaussian mechanism with noise multiplier σ satisfies the same privacy guarantee of that of a single Gaussian mechanism with a noise multiplier σ/ √ k .

Original: By fixing k and ε , we can calibrate the noise by choosing an appropriate σ in our algorithm.
Revised: By fixing k and ε , we can calibrate the noise by choosing an appropriate σ in our algorithm.

Original: 5. D IFFERENTIAL P RIVACY AT THE S AMPLE L EVEL
Revised: 5. D IFFERENTIAL P RIVACY AT THE S AMPLE L EVEL

Original: Recall that we partition the private dataset to M disjoint subsets, and train a teacher model on each subset.
Revised: Recall that we partition the private dataset to M disjoint subsets, and train a teacher model on each subset.

Original: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -th teacher model for a specific word, where |V| is the vocabulary size.
Revised: Let vector x i ∈ R |V| denote the probability distribution predicted by the i -th teacher model for a specific word, where |V| is the vocabulary size.

Original: The function f ( D ) :=
Revised: The function f ( D ) :=

Original: (cid:80) Mi
Revised: (cid:80) Mi

Original: =1 x i is the sum of the probability distributions predicted over all teachers.
Revised: =1 x i is the sum of the probability distributions predicted over all teachers.

Original: Since the dataset is disjoint and also has no repeating sample, changing one sample will only affect one teacher model.
Revised: Since the dataset is disjoint and also has no repeating sample, changing one sample will only affect one teacher model.

Original: For neighboring datasets
Revised: For neighboring datasets

Original: D , D ′ , let j denote the index of different teacher models, i.e., x j ∈ D and x ′ j ∈ D ′ are different.
Revised: D , D ′ , let j denote the index of different teacher models, i.e., x j ∈ D and x ′ j ∈ D ′ are different.

Original: Then, the sensitivity ∆ in Lemma 4 & 5 is (Detailed deduction in Appendix B),
Revised: Then, the sensitivity ∆ in Lemma 4 & 5 is (Detailed deduction in Appendix B),

Original: [Equation]
Revised: [Equation]

Original: Then, adding noise calibrated in Lemma 5 to each coordinate is sufficient to preserve ( ε , δ ( ε ) )
Revised: Then, adding noise calibrated in Lemma 5 to each coordinate is sufficient to preserve ( ε , δ ( ε ) )

Original: -DP for f ( D ) .
Revised: -DP for f ( D ) .

Original: Finally, when we extract top- k coordinates of f ( D ) (Sec. 4.3), the privacy guarantee also holds according to the post-processing property.
Revised: Finally, when we extract top- k coordinates of f ( D ) (Sec. 4.3), the privacy guarantee also holds according to the post-processing property.

Original: Intuitively, the information about whether a sample occurs in the private training set is protected and the protection satisfies ( ε , δ ( ε ) )
Revised: Intuitively, the information about whether a sample occurs in the private training set is protected and the protection satisfies ( ε , δ ( ε ) )

Original: -DP.
Revised: -DP.

Original: 5. D IFFERENTIAL P RIVACY OF U SER ’ S S ECRET P HRASES
Revised: 5. D IFFERENTIAL P RIVACY OF U SER ’ S S ECRET P HRASES

Original: The above analysis focuses on DP at the sample level, but the privacy guarantee also applies to individual users (who contributed many samples), which prevents each user’s secret phrases (or text spans), such as SSN number, phone number, and addresses from being memorized (or generated as is ) by the model.
Revised: The above analysis focuses on DP at the sample level, but the privacy guarantee also applies to individual users (who contributed many samples), which prevents each user’s secret phrases (or text spans), such as SSN number, phone number, and addresses from being memorized (or generated as is ) by the model.

Original: Consider a secret phrase t that occurs s t times ( s t ≥ 1 ) in the private set.
Revised: Consider a secret phrase t that occurs s t times ( s t ≥ 1 ) in the private set.

Original: We obtain
Revised: We obtain

Original: DP guarantee for each t separately 3 in the following two scenarios.
Revised: DP guarantee for each t separately 3 in the following two scenarios.

Original: In the first scenario where the teachers’ data is not partitioned by users, the protection on t satisfies ( s t ε, e stε − 1 e ε − 1 δ ) -DP
Revised: In the first scenario where the teachers’ data is not partitioned by users, the protection on t satisfies ( s t ε, e stε − 1 e ε − 1 δ ) -DP

Original: according to the group privacy lemma.
Revised: according to the group privacy lemma.

Original: SeqPATE allows us to partition the teachers’ training data by users.
Revised: SeqPATE allows us to partition the teachers’ training data by users.

Original: In this scenario, adding or removing a user (and all her samples) only affects one teacher, thus the same privacy guarantee we have derived for sample-level DP applies to the user level, too.
Revised: In this scenario, adding or removing a user (and all her samples) only affects one teacher, thus the same privacy guarantee we have derived for sample-level DP applies to the user level, too.

Original: As a result, our approach enjoys stronger guarantee than naively applying the group privacy lemma.
Revised: As a result, our approach enjoys stronger guarantee than naively applying the group privacy lemma.

Original: Our approach depends only on the total number of teachers affected, which is bounded by ˜ s t :
Revised: Our approach depends only on the total number of teachers affected, which is bounded by ˜ s t :

Original: = min { s t , # of users who know t } .
Revised: = min { s t , # of users who know t } .

Original: ˜ s t is often 2 or 3 (e.g., husband and wife, parents and child), even if s t is large.
Revised: ˜ s t is often 2 or 3 (e.g., husband and wife, parents and child), even if s t is large.

Original: Clearly, adding or removing a phrase t will result in a sensitivity √ 2˜ s t , and the exact ( ε, δ ( ε, ˜ s t ))
Revised: Clearly, adding or removing a phrase t will result in a sensitivity √ 2˜ s t , and the exact ( ε, δ ( ε, ˜ s t ))

Original: -DP for phrase t can be obtained according to Lemma 4 & 5, where δ ( ε, ˜ s t ) =
Revised: -DP for phrase t can be obtained according to Lemma 4 & 5, where δ ( ε, ˜ s t ) =

Original: Φ( ˜ s t √ 2 σ − εσ √ 2˜ s t ) − e ε Φ( − ˜ s t √ 2 σ − εσ √ 2˜ s t ) .
Revised: Φ( ˜ s t √ 2 σ − εσ √ 2˜ s t ) − e ε Φ( − ˜ s t [Equation] 2 σ − εσ √ 2˜ s t ) .

Revised: The above user-level protection is also one merit of SeqPATE that NoisySGD does not have.

Original: How does DP prevent memorization?
Revised: How does DP prevent memorization?

Original: We remark that the protection against unintended memorization of a model follows from the definition of DP.
Revised: We remark that the protection against unintended memorization of a model follows from the definition of DP.

Original: Consider the attack by (Carlini et al., 2019), which uses a language model to predict a secret phrase t given a prefix.
Revised: Consider the attack by (Carlini et al., 2019), which uses a language model to predict a secret phrase t given a prefix.

Original: By the closure to post-processing, the prediction also satisfies DP.
Revised: By the closure to post-processing, the prediction also satisfies DP.

Original: Take S to be the undesirable event that the model correctly generates the phrase t .
Revised: Take S to be the undesirable event that the model correctly generates the phrase t .

Original: The definition of DP implies that the probability of S to happen when t is part of the training data is at most e ε larger than the probability of an alternative model trained without t in the data.
Revised: The definition of DP implies that the probability of S to happen when t is part of the training data is at most e ε larger than the probability of an alternative model trained without t in the data.

Original: The chances for the latter model to generate a sequence with t is astronomically small, thus DP implies that the probability of S under the former model needs to be just as small.
Revised: The chances for the latter model to generate a sequence with t is astronomically small, thus DP implies that the probability of S under the former model needs to be just as small.

Original:  E XPERIMENTS 6. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 6. E XPERIMENTAL S ETTING

Original: We evaluate our model on two datasets.
Revised: We evaluate our model on two datasets.

Original: AirDialog dataset (Wei et al., 2018) consists of 1M utterances from custom service dialog about flight booking; Europarl v6 dataset 4 consists of 2M English sentences collected from European Parliament.
Revised: AirDialog dataset (Wei et al., 2018) consists of 1M utterances from custom service dialog about flight booking; Europarl v6 dataset 4 consists of 2M English sentences collected from European Parliament.

Original: Pre-trained GPT-2 (Radford et al., 2018) provides initial parameters for comparing methods.
Revised: Pre-trained GPT-2 (Radford et al., 2018) provides initial parameters for comparing methods.

Original: Before student training, we conduct additional fine-tuning on the public data ˜ D pub via NLL loss to provide initialization.
Revised: Before student training, we conduct additional fine-tuning on the public data ˜ D pub via NLL loss to provide initialization.

Original: We evaluate the quality of generated texts with PPL (Perplexity, the lower the better) and B-n (Bleu-n, the higher the better) (Papineni et al., 2002) and evaluate the performance of privacy protection with P-n , which is the fraction of unique n-grams in model generated texts that can be found in the private text (lower score indicates fewer privacy leaks).
Revised: We evaluate the quality of generated texts with PPL (Perplexity, the lower the better) and B-n (Bleu-n, the higher the better) (Papineni et al., 2002) and evaluate the performance of privacy protection with P-n , which is the fraction of unique n-grams in model generated texts that can be found in the private text (lower score indicates fewer privacy leaks).

Original: Our base model is GPT2-small and the default number of teacher models is 2,000.
Revised: Our base model is GPT2-small and the default number of teacher models is 2,000.

Original: The batch size is 32 for all comparing methods and the λ in Eq.
Revised: The batch size is 32 for all comparing methods and the λ in Eq.

Original: 2 is 20.
Revised: 2 is 20.

Original: (See details about datasets, settings, metrics, implementation [Table] and more experimental results in Appendix C to L).
Revised: (See details about datasets, settings, metrics, implementation, and more experimental results in Appendix C to L).

Original: 6. O VERALL P ERFORMANCE
Revised: 6. O VERALL P ERFORMANCE

Original: [Table]
Revised: [Table]

Original: Table 1 shows the performance on the two datasets.
Revised: Table 1 shows the performance on the two datasets.

Original: Pri-GPT is the GPT model trained on the private set D pri without privacy protection, acting as an upper bound on the task performance.
Revised: Pri-GPT is the GPT model trained on the private set D pri without privacy protection, acting as an upper bound on the task performance.

Original: The high P-n scores on Pri-GPT show most generated n-grams may come from D pri , indicating a serious issue with privacy leaks in current text generation models.
Revised: The high P-n scores on Pri-GPT show most generated n-grams may come from D pri , indicating a serious issue with privacy leaks in current text generation models.

Original: Pub-GPT uses the pre-trained GPT model to our setting without fine-tuning.
Revised: Pub-GPT uses the pre-trained GPT model to our setting without fine-tuning.

Original: Pub-GPT+ ˜ D pub is further fine-tuned on the pseudo data and outperforms
Revised: Pub-GPT+ ˜ D pub is further fine-tuned on the pseudo data and outperforms

Original: Pub-GPT, indicating that the pseudo dataset helps in this task (Experiments in Appendix L also verify that.).
Revised: Pub-GPT, indicating that the pseudo dataset helps in this task (Experiments in Appendix L also verify that.).

Original: The two baselines do not involve any information from the private data, so their P-n are the upper bound and their utlities are the worst (note that our goal is to do well on D pri in inference).
Revised: The two baselines do not involve any information from the private data, so their P-n are the upper bound and their utlities are the worst (note that our goal is to do well on D pri in inference).

Original: DP learning algorithms are our main baselines.
Revised: DP learning algorithms are our main baselines.

Original: Pub-GPT+DP-SGD is initialized by Pub-GPT and further trained on D pri with DP-SGD (Noisy-SGD)
Revised: Pub-GPT+DP-SGD is initialized by Pub-GPT and further trained on D pri with DP-SGD (Noisy-SGD)

Original: (Abadi et al., 2016; Kerrigan et al., 2020).
Revised: (Abadi et al., 2016; Kerrigan et al., 2020).

Original: PubGPT+DP-SGD+
Revised: PubGPT+DP-SGD+

Original: ˜ D pub further fine-tunes the Pub-GPT+DP-SGD on ˜ D pub .
Revised: ˜ D pub further fine-tunes the Pub-GPT+DP-SGD on ˜ D pub .

Original: The small gap between the two methods demonstrates DP-SGD cannot utilizethe public set ˜ D pubso well.
Revised: The small gap between the two methods demonstrates DP-SGD cannot utilize ˜ D pub well.

Original: Our method, SeqPATE, significantly outperforms the DP baselines (+39.3% and +28.0% in Bleu4) while ensuring the same strength of privacy protection in terms of P-n and ε .
Revised: Our method, SeqPATE, significantly outperforms the DP baselines (+39.3% and +28.0% in Bleu4) while ensuring the same strength of privacy protection in terms of P-n and ε .

Original: Note that, we conduct the experiments based on the sample level privacy of ε = 2 mentioned in Sec. 5.2, and the experimental results also reflect the protection on users’ secret phrase t mentioned in Sec.
Revised: Note that, we conduct the experiments based on the sample level privacy of ε = 2 mentioned in Sec. 5.2, and the experimental results also reflect the protection on users’ secret phrase t mentioned in Sec.

Original: However, the strength of the protection is weaker than the sample-level protection.
Revised: However, the strength of the protection is weaker than the sample-level protection.

Original: The model satisfies ( ε, δ ( ε, ˜ s t )) -DP
Revised: The model satisfies ( ε, δ ( ε, ˜ s t )) -DP

Original: or ( s t ε, e stε − 1 e ε − 1 δ ) -DP
Revised: or ( s t ε, e stε − 1 e ε − 1 δ ) -DP

Original: on the secret phrase t ( ε
Revised: on the secret phrase t ( ε

Original: = 2 ) as mentioned in Sec.
Revised: = 2 ) as mentioned in Sec.

Original: SeqPATE does better in empirical metrics on privacy compared to DP-SGD even if ε = 2 is the same theoretically; this provides empirical evidence of our discussion in Sec.
Revised: SeqPATE does better in empirical metrics on privacy compared to DP-SGD even if ε = 2 is the same theoretically; this provides empirical evidence of our discussion in Sec.

Original: 5.3 on providing stronger guarantees for phrases that appear more than once.
Revised: 5.3 on providing stronger guarantees for phrases that appear more than once.

Original: The curves in Fig.
Revised: The curves in Fig.

Original: 2 show the private-utility tradeoff of all DP algorithms in AirDialog (subfigure a & b) and Europarl v6 (subfigure c & d) dataset 5 .
Revised: 2 show the private-utility tradeoff of all DP algorithms in AirDialog (subfigure a & b) and Europarl v6 (subfigure c & d) dataset 5 .

Original: DP with ε ∈ [0 . 1 , 5] is considered to provide a meaningful protection, and we observe that SeqPATE outperforms DP-SGD in this range.
Revised: DP with ε ∈ [0 . 1 , 5] is considered to provide a meaningful protection, and we observe that SeqPATE outperforms DP-SGD in this range.

Original: SeqPATE does not work better than DP-SGD when ε ≥ 5 .
Revised: SeqPATE does not work better than DP-SGD when ε ≥ 5 .

Original: The reason is DP-SGD approaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).
Revised: The reason is DP-SGD approaches Pri-GPT as ε approaches infinity (i.e. the noise approaches 0).

Original: However, SeqPATE with an infinite ε is still weaker than Pri-GPT because each teacher in SeqPATE is trained on a small partition of the full data.
Revised: However, SeqPATE with an infinite ε is still weaker than Pri-GPT because each teacher in SeqPATE is trained on a small partition of the full data.

Original: The grey lines show the performance of Pub-GPT+ ˜ D pri and the area above the line indicates that
Revised: The grey lines show the performance of Pub-GPT+ ˜ D pri and the area above the line indicates that

Original: the
Revised: the

Original: DP methods outperform Pub-GPT+ ˜ D pri (DP methods on the private data outperforms the non-DP methods on the non-private data).
Revised: DP methods outperform Pub-GPT+ ˜ D pri (DP methods on the private data outperforms the non-DP methods on the non-private data).

Original: The area indicates a range where DP methods (with private data) are useful for this application.
Revised: The area indicates a range where DP methods (with private data) are useful for this application.

Original: Our method always reaches that area when ε ≥ 1 but DP-SGD hardly exceeds the grey line, the very trivial baseline, with ε ≤ 5 .
Revised: Our method always reaches that area when ε ≥ 1 but DP-SGD hardly exceeds the grey line, the very trivial baseline, with ε ≤ 5 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 6. A BLATION S TUDIES AND F URTHER A NALYSES
Revised: 6. A BLATION S TUDIES AND F URTHER A NALYSES

Original: In the ablation studies (Table 2), SeqPATE mainly outperforms its variants shown in rows 2 toverifying the effectiveness of our strategies, where − X represents SeqPAET’s variant without the strategy X .
Revised: In the ablation studies (Table 2), SeqPATE mainly outperforms its variants shown in rows 2 toverifying the effectiveness of our strategies, where − X represents SeqPAET’s variant without the strategy X .

Original: − Merge P is the variant aggregating the teachers by voting instead of merging the probabilities.
Revised: − Merge P is the variant aggregating the teachers by voting instead of merging the probabilities.

Original: Its poor performance shows that voting is not suitable for text generation with a large output space where the voting leads to low agreement rates.
Revised: Its poor performance shows that voting is not suitable for text generation with a large output space where the voting leads to low agreement rates.

Original: − KL means the student learns from the aggregation of teachers’ top-1 predictions via NLL loss instead of KL loss.
Revised: − KL means the student learns from the aggregation of teachers’ top-1 predictions via NLL loss instead of KL loss.

Original: The sharp performance drop on − KL indicates the importance of KL loss.
Revised: The sharp performance drop on − KL indicates the importance of KL loss.

Original: The performance of − NLL slightly decreases, indicating that NLL loss makes a little contribution to SeqPATE.
Revised: The performance of − NLL slightly decreases, indicating that NLL loss makes a little contribution to SeqPATE.

Original: The reason is that we have pre-trained on the student’s training set via NLL before student training.
Revised: The reason is that we have pre-trained on the student’s training set via NLL before student training.

Original: The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl v6, which shows the “clever” student (e.g. models on AirDialog with low PPL and high Bleu) benefits more from this strategy since it can sharply save the privacy cost and spend the cost where the student strongly needs.
Revised: The promotion caused by efficient knowledge distillation (Effi KD) on AirDialog is larger than that on Europarl v6, which shows the “clever” student (e.g. models on AirDialog with low PPL and high Bleu) benefits more from this strategy since it can sharply save the privacy cost and spend the cost where the student strongly needs.

Original: − Gaussian means we use Laplace mechanism as mentioned in PATE instead of Gaussian mechanism, which shows that the Gaussian mechanism works better.
Revised: − Gaussian means we use Laplace mechanism as mentioned in PATE instead of Gaussian mechanism, which shows that the Gaussian mechanism works better.

Original: − All indicates discarding all above strategies, which is similar to but not equivalent to the original PATE (the difference is that
Revised: − All indicates discarding all above strategies, which is similar to but not equivalent to the original PATE (the difference is that

Original: PATE needs to roll out all teachers (Sec. 4.1), which disables PATE from working on text generation).
Revised: PATE needs to roll out all teachers (Sec. 4.1), which disables PATE from working on text generation).

Original: The poor performance of − All verifies that the original PATE is not adaptive to text generation.
Revised: The poor performance of − All verifies that the original PATE is not adaptive to text generation.

Original: Table 3 shows the performance of the top- p and top- k selection strategies (Sec. 4.3).
Revised: Table 3 shows the performance of the top- p and top- k selection strategies (Sec. 4.3).

Original: As shown in row 1, our full model employs the top- p selection (the threshold µ = 0 .
Revised: As shown in row 1, our full model employs the top- p selection (the threshold µ = 0 .

Original: 95 ) surpasses most variants with manually chosen k (rows 2 to 6).
Revised: 95 ) surpasses most variants with manually chosen k (rows 2 to 6).

Original: A selection with a too small k ( k = 1 & k = 10 ) implies discarding too much useful information from the supervision ( k = 1 is different to − KL in Table 2, which uses Top-1 of teachers’ results).
Revised: A selection with a too small k ( k = 1 & k = 10 ) implies discarding too much useful information from the supervision ( k = 1 is different to − KL in Table 2, which uses Top-1 of teachers’ results).

Original: A selection with oversize k results in involving more useless noise: the candidates with very small probabilities should be rarely sampled during generation but random noise may increase their probabilities, so models may generate words that are misled by the noise.
Revised: A selection with oversize k results in involving more useless noise: the candidates with very small probabilities should be rarely sampled during generation but random noise may increase their probabilities, so models may generate words that are misled by the noise.

Original: Supervisions on top- k candidates with k = 50 and k = 100 yields relatively good performance.
Revised: Supervisions on top- k candidates with k = 50 and k = 100 yields relatively good performance.

Original: Tables 4 & 5 in the Appendix shows more teachers lead to better results since the noise assigned to each teacher drops linearly as the teacher’s numbers go up (as Eq. 2).
Revised: Tables 4 & 5 in the Appendix shows more teachers lead to better results since the noise assigned to each teacher drops linearly as the teacher’s numbers go up (as Eq. 2).

Original: We choose ε = 2 for all results in Table 2 to 5.
Revised: We choose ε = 2 for all results in Table 2 to 5.

Original: P-n for all the results in a dataset to be roughly on the same level, except for some methods (e.g. row 2 in Table 3) that perform very badly in all metrics.
Revised: P-n for all the results in a dataset to be roughly on the same level, except for some methods (e.g. row 2 in Table 3) that perform very badly in all metrics.

Original: It shows SeqPATE has the stable performance of protection if ε is fixed (no matter which variant and strategy do we use).
Revised: It shows SeqPATE has the stable performance of protection if ε is fixed (no matter which variant and strategy do we use).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Privacy protection is becoming crucial in deep learning models.
Revised: Privacy protection is becoming crucial in deep learning models.

Original: Research in this area can be categorized into three directions (Liu et al., 2021).
Revised: Research in this area can be categorized into three directions (Liu et al., 2021).

Original: Encryption approaches are to add encryption on training data (Brickell et al., 2007; Bost et al., 2015) or machine learning (ML) models (Aono et al., 2017).
Revised: Encryption approaches are to add encryption on training data (Brickell et al., 2007; Bost et al., 2015) or machine learning (ML) models (Aono et al., 2017).

Original: Aggregation approaches generally come along with distributed training where many parties join the training of an ML model with the purpose of maintaining the privacy of the dataset from each party (Shokri & Shmatikov, 2015; Koneˇcn`y et al., 2016).
Revised: Aggregation approaches generally come along with distributed training where many parties join the training of an ML model with the purpose of maintaining the privacy of the dataset from each party (Shokri & Shmatikov, 2015; Koneˇcn`y et al., 2016).

Original: Obfuscation approaches reduce the precision of the data or model by importing noise into the original input (Zhang et al., 2018), model parameters (Rubinstein et al., 2012), or model gradients (Abadi et al., 2016; McMahan et al., 2018).
Revised: Obfuscation approaches reduce the precision of the data or model by importing noise into the original input (Zhang et al., 2018), model parameters (Rubinstein et al., 2012), or model gradients (Abadi et al., 2016; McMahan et al., 2018).

Original: As a sub-topic of the obfuscation approach, differential privacy (DP) (Dwork et al., 2006; 2014) formally defines and quantifies privacy.
Revised: As a sub-topic of the obfuscation approach, differential privacy (DP) (Dwork et al., 2006; 2014) formally defines and quantifies privacy.

Original: ML models with DP guarantee (Wang et al., 2015; Park et al., 2016;
Revised: ML models with DP guarantee (Wang et al., 2015; Park et al., 2016;

Original: Ziller et al., 2021) prevents the existence of individual training examples from being detected (Carlini et al., 2019).
Revised: Ziller et al., 2021) prevents the existence of individual training examples from being detected (Carlini et al., 2019).

Original: Some researchers protect the privacy of empirical risk minimization (ERM) classifier (Chaudhuri et al., 2011) and SVM (Rubinstein et al., 2012) with DP.
Revised: Some researchers protect the privacy of empirical risk minimization (ERM) classifier (Chaudhuri et al., 2011) and SVM (Rubinstein et al., 2012) with DP.

Original: Following (Song et al., 2013), DP-SGD (Abadi et al., 2016) achieves DP on deep learning models by applying the noise to the clipped gradients.
Revised: Following (Song et al., 2013), DP-SGD (Abadi et al., 2016) achieves DP on deep learning models by applying the noise to the clipped gradients.

Original: (Bu et al., 2020).
Revised: (Bu et al., 2020).

Original: Pichapati et al.
Revised: Pichapati et al.

Original: (2019) adaptively clip the gradient on DP-SGD.
Revised: (2019) adaptively clip the gradient on DP-SGD.

Original: PATE (Papernot et al., 2017) transfers the knowledge from teacher models trained on private data with injected noise to a student model.
Revised: PATE (Papernot et al., 2017) transfers the knowledge from teacher models trained on private data with injected noise to a student model.

Original: KNN-PATE (Zhu et al., 2020) refines PATE by accessing only the k-nearest neighbors from the private set instead of the whole private set.
Revised: KNN-PATE (Zhu et al., 2020) refines PATE by accessing only the k-nearest neighbors from the private set instead of the whole private set.

Original: The above approaches are not customized for text generation models.
Revised: The above approaches are not customized for text generation models.

Original: One direction of privacy protection in text generation is to protect author-level (user-level) information.
Revised: One direction of privacy protection in text generation is to protect author-level (user-level) information.

Original: The methods prevent attackers from inferring the author attributes (e.g. gender, age)
Revised: The methods prevent attackers from inferring the author attributes (e.g. gender, age)

Original: (Li et al., 2018) and the relation between sensitive information and authors (Mireshghallah et al., 2021).
Revised: (Li et al., 2018) and the relation between sensitive information and authors (Mireshghallah et al., 2021).

Original: Shokri et al.
Revised: Shokri et al.

Original: (2017) and Song & Shmatikov (2019) infer the membership (whether a given user’s data is used to train the model) given a black-box model.
Revised: (2017) and Song & Shmatikov (2019) infer the membership (whether a given user’s data is used to train the model) given a black-box model.

Original: Another direction is to prevent attackers from extracting sensitive information in the training set by analyzing the generated outputs (Nakamura et al., 2020;
Revised: Another direction is to prevent attackers from extracting sensitive information in the training set by analyzing the generated outputs (Nakamura et al., 2020;

Original: Kerrigan et al., 2020), which is an urgent need since Carlini et al.
Revised: Kerrigan et al., 2020), which is an urgent need since Carlini et al.

Original: (2020) achieve the extraction.
Revised: (2020) achieve the extraction.

Original: Our application focuses on this direction.
Revised: Our application focuses on this direction.

Original: In this direction, some researchers use regularization, including dropout (Srivastava et al., 2014) and quantization (Hubara et al., 2017), to restrict the model capacity to avoid models memorizing too much original text from the privacy training set (Carlini et al., 2019).
Revised: In this direction, some researchers use regularization, including dropout (Srivastava et al., 2014) and quantization (Hubara et al., 2017), to restrict the model capacity to avoid models memorizing too much original text from the privacy training set (Carlini et al., 2019).

Original: Anonymization methods (Maeda et al., 2016; Suzuki et al., 2017) detects sensitive text spans in training sets and replace the spans with non-sensitive texts.
Revised: Anonymization methods (Maeda et al., 2016; Suzuki et al., 2017) detects sensitive text spans in training sets and replace the spans with non-sensitive texts.

Original: The above methods are non-DP algorithms, which make sense intuitively but do not provide quantifiable and provable guarantees for privacy protection.
Revised: The above methods are non-DP algorithms, which make sense intuitively but do not provide quantifiable and provable guarantees for privacy protection.

Original: Some researchers apply DP to text generation.
Revised: Some researchers apply DP to text generation.

Original: For user-level privacy, ER-AE (Bo et al., 2021) augments the semantic information in the generated text to hide authors’ writing styles for authorship anonymization.
Revised: For user-level privacy, ER-AE (Bo et al., 2021) augments the semantic information in the generated text to hide authors’ writing styles for authorship anonymization.

Original: McMahan et al.
Revised: McMahan et al.

Original: (2018) propose a recurrent language model with a DP guarantee against the identification of the users.
Revised: (2018) propose a recurrent language model with a DP guarantee against the identification of the users.

Original: Kerrigan et al.
Revised: Kerrigan et al.

Original: (2020) adapt DP-SGD to text generation, which prevents the (not the user-level) sensitive training data from being extracted from text generation models.
Revised: (2020) adapt DP-SGD to text generation, which prevents the (not the user-level) sensitive training data from being extracted from text generation models.

Original: Some of our concurrent works also follow this direction.
Revised: Some of our concurrent works also follow this direction.

Original: Li et al.
Revised: Li et al.

Original: (2021) apply DP-SGD to large pre-trained language models and reduce the memory usage of those models.
Revised: (2021) apply DP-SGD to large pre-trained language models and reduce the memory usage of those models.

Original: Yu et al.
Revised: Yu et al.

Original: (2021) propose a framework to carry some advanced fine-tuning strategies on pre-trained language models with DP-SGD.
Revised: (2021) propose a framework to carry some advanced fine-tuning strategies on pre-trained language models with DP-SGD.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose, SeqPATE, a framework to protect the privacy of the training data for text generation model with DP guarantees.
Revised: In this paper, we propose, SeqPATE, a framework to protect the privacy of the training data for text generation model with DP guarantees.

Original: SeqPATE achieves a good privacy-utility tradeoff by leveraging both private and public data.
Revised: SeqPATE achieves a good privacy-utility tradeoff by leveraging both private and public data.

Original: As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and thus is adaptive for text generation models.
Revised: As an extension of PATE, SeqPATE can handle the sequential generation paradigm with large output space at each step and thus is adaptive for text generation models.

Original: We avoid rolling out the teachers by providing pseudo inputs for teacher inference and student training and aggregate teachers by merging probabilities instead of voting.
Revised: We avoid rolling out the teachers by providing pseudo inputs for teacher inference and student training and aggregate teachers by merging probabilities instead of voting.

Original: We further reduce the output space by top- k or top- p selection and limit the privacy loss via an efficient knowledge distillation.
Revised: We further reduce the output space by top- k or top- p selection and limit the privacy loss via an efficient knowledge distillation.

Original: SeqPATE improves the baselines a lot and makes the private data useful to text generation with a meaningful strength of privacy protection.
Revised: SeqPATE improves the baselines a lot and makes the private data useful to text generation with a meaningful strength of privacy protection.

Original: In the future, we will apply SeqPATE to some real-world applications, e.g. dialog systems and writing assistants.
Revised: In the future, we will apply SeqPATE to some real-world applications, e.g. dialog systems and writing assistants.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We’ve submitted our source code as the supplementary material.
Revised: We’ve submitted our source code as the supplementary material.

Original: We introduce the two datasets we used in Appendix C, where we introduce the details about the dataset and also append the link or citation of the dataset.
Revised: We introduce the two datasets we used in Appendix C, where we introduce the details about the dataset and also append the link or citation of the dataset.

Original: For all our experiments, we adopt autodp (Wang et al., 2019) — an open-source library that implements the analytical Gaussian mechanism for privacy accounting and calibration.
Revised: For all our experiments, we adopt autodp (Wang et al., 2019) — an open-source library that implements the analytical Gaussian mechanism for privacy accounting and calibration.

Original: The experimental setting and hyper-parameters can be found in Appendix D. Since we propose a new metric in our experiments, we introduce the new metric with detailed equations in Appendix E.
Revised: The experimental setting and hyper-parameters can be found in Appendix D. Since we propose a new metric in our experiments, we introduce the new metric with detailed equations in Appendix E.

Original:  E THICAL C ONSIDERATIONS
Revised:  E THICAL C ONSIDERATIONS

Original: This paper aims to protect privacy in training the text generation models, so our paper is designed for tackling some ethical issues about privacy concerns in some existing text generation models (e.g.
Revised: This paper aims to protect privacy in training the text generation models, so our paper is designed for tackling some ethical issues about privacy concerns in some existing text generation models (e.g.

Original: GPT-2).
Revised: GPT-2).

Original: In terms of motivation and the algorithm, our paper would not cause ethical issues.
Revised: In terms of motivation and the algorithm, our paper would not cause ethical issues.

Original: However, we should also consider some special situations where someone intentionally applies our model to illegal applications.
Revised: However, we should also consider some special situations where someone intentionally applies our model to illegal applications.

Original: Someone may employ text generation models to create fake news or misinformation and protect himself or herself from being detected.
Revised: Someone may employ text generation models to create fake news or misinformation and protect himself or herself from being detected.

Original: For this purpose, they may use our model for their illegal application.
Revised: For this purpose, they may use our model for their illegal application.

Original: In the future, we will add some constraints to our model so that our model cannot generate text for illegal applications (e.g. fake news).
Revised: In the future, we will add some constraints to our model so that our model cannot generate text for illegal applications (e.g. fake news).

Original: In addition, we know large ε (e.g. ε = 500 ) cannot provide a meaningful protection.
Revised: In addition, we know large ε (e.g. ε = 500 ) cannot provide a meaningful protection.

Original: We should carefully use our model and cannot assume the model is perfect no matter what parameters ( ε and δ ) do we use.
Revised: We should carefully use our model and cannot assume the model is perfect no matter what parameters ( ε and δ ) do we use.

Original: One possible unethical application is to collect the data from users who believe our model can fully protect their privacy regardless of the strength of privacy protection (in terms of the value of ε ).
Revised: One possible unethical application is to collect the data from users who believe our model can fully protect their privacy regardless of the strength of privacy protection (in terms of the value of ε ).

Original: Hence, we kindly remind the researchers, who will use this model, to pay more attention to the strength of privacy protection.
Revised: Hence, we kindly remind the researchers, who will use this model, to pay more attention to the strength of privacy protection.

Original: Further, we should prevent some researchers to collect the data from users who do not have a correct understanding of our algorithm, and the data collectors should make it clear that the potential risk in releasing data to our model.
Revised: Further, we should prevent some researchers to collect the data from users who do not have a correct understanding of our algorithm, and the data collectors should make it clear that the potential risk in releasing data to our model.

Original: A A LGORITHM FOR THE T RAINING OF S EQ PATE
Revised: A A LGORITHM FOR THE T RAINING OF S EQ PATE

Original: The pseudo codes of SeqPATE’s training is shown in Algorithm 1.
Revised: The pseudo codes of SeqPATE’s training is shown in Algorithm 1.

Original: Algorithm 1 Training
Revised: Algorithm 1 Training

Original: Require: D pri , D pub : datasets, GPT : a pre-trained GPT model.
Revised: Require: D pri , D pub : datasets, GPT : a pre-trained GPT model.

Original: 1: { f mϕ } Mm =1 : M teacher models, f θ : a student model, f Θ : a student model for self pre-training, 2: { ϕ m } Mm =1 ← GPT , Θ ← GPT # Initialize teachers and the student for self pre-training.
Revised: 1: { f mϕ } Mm =1 : M teacher models, f θ : a student model, f Θ : a student model for self pre-training, 2: { ϕ m } Mm =1 ← GPT , Θ ← GPT # Initialize teachers and the student for self pre-training.

Original: 3: GPT generates a pseudo dataset ˜ D pub based on D pub .
Revised: 3: GPT generates a pseudo dataset ˜ D pub based on D pub .

Original: 4: {D pri m } M m =1 ← D pri # Divide private dataset into m subsets.
Revised: 4: {D pri m } M m =1 ← D pri # Divide private dataset into m subsets.

Original: 5: for all m in M do 6:
Revised: 5: for all m in M do 6:

Original: Train teacher f mϕ on D prim 7: end for 8: Teachers { ϕ m } Mm =1 conduct inference on ˜ D pub to get p
Revised: Train teacher f mϕ on D prim 7: end for 8: Teachers { ϕ m } Mm =1 conduct inference on ˜ D pub to get p

Original: mϕ ( w i | w 1: i − 1 ) required in Eq.
Revised: mϕ ( w i | w 1: i − 1 ) required in Eq.

Original: 2 for all samples.
Revised: 2 for all samples.

Original: 9: 10: Train f Θ on ˜ D
Revised: 9: 10: Train f Θ on ˜ D

Original: pub
Revised: pub

Original: # self pre-training for the student.
Revised: # self pre-training for the student.

Original: 11: θ ← Θ # Initialize the student model.
Revised: 11: θ ← Θ # Initialize the student model.

Original: 12: 13: while not converge do 14: for all batch of samples { S } batchsize in ˜ D pub do 15:
Revised: 12: 13: while not converge do 14: for all batch of samples { S } batchsize in ˜ D pub do 15:

Original: Student f θ conducts feed-forward on { S } batchsize .
Revised: Student f θ conducts feed-forward on { S } batchsize .

Original: 16: for all sample S in the batch { S } batchsize do 17: for all token w i in sample S do 18:
Revised: 16: for all sample S in the batch { S } batchsize do 17: for all token w i in sample S do 18:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: w 1: i − 1 )
Revised: w 1: i − 1 )

Original: # Aggregate teachers’ outputs 19:
Revised: # Aggregate teachers’ outputs 19:

Original: Select only top- k or top- p predicted tokens as student’s output.
Revised: Select only top- k or top- p predicted tokens as student’s output.

Original: Obtain L KL i and L NLL i as Eq. 2.
Revised: Obtain L KL i and L NLL i as Eq. 2.

Original: # Noise is added into L KL i to protect the privacy.
Revised: # Noise is added into L KL i to protect the privacy.

Original: Get L by combining L KL i and L NLL i
Revised: Get L by combining L KL i and L NLL i

Original: # Efficient knowledge distillation.
Revised: # Efficient knowledge distillation.

Original: 22: end for 23: end for 24:
Revised: 22: end for 23: end for 24:

Original: Update ϕ respect to L .
Revised: Update ϕ respect to L .

Original: 25: end for 26: end while
Revised: 25: end for 26: end while

Original: B D ETAILED D EDUCTION OF THE S ENSITIVITY IN S ENTENCE L EVEL DP
Revised: B D ETAILED D EDUCTION OF THE S ENSITIVITY IN S ENTENCE L EVEL DP

Original: We obtain Eq.
Revised: We obtain Eq.

Original: 3 from Sec.
Revised: 3 from Sec.

Original: [Equation]
Revised: [Equation]

Original: We know ( x jv − x ′ jv ) 2 is smaller than | x jv − x ′ jv | since | x jv − x ′ jv | ∈ (0 , 1) for each v .
Revised: We know ( x jv − x ′ jv ) 2 is smaller than | x jv − x ′ jv | since | x jv − x ′ jv | ∈ (0 , 1) for each v .

Original: Hence, we have, (cid:18)
Revised: Hence, we have, (cid:18)

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | (cid:88) v
Revised: | (cid:88) v

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | (cid:19)
Revised: | (cid:19)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ (cid:18) |V| (cid:88) v =
Revised: ≤ (cid:18) |V| (cid:88) v =

Original: | x jv + x ′ jv
Revised: | x jv + x ′ jv

Original: | (cid:19) 1 /
Revised: | (cid:19) 1 /

Original: We know | a + b | = a + b when a, b ∈ (0 , 1) , so we have, (cid:18) |V| (cid:88) v =
Revised: We know | a + b | = a + b when a, b ∈ (0 , 1) , so we have, (cid:18) |V| (cid:88) v =

Original: | x jv + x ′ jv
Revised: | x jv + x ′ jv

Original: | (cid:19) 1 /= (cid:18) |V| (cid:88) v = x jv + |V| (cid:88) v
Revised: | (cid:19) 1 /= (cid:18) |V| (cid:88) v = x jv + |V| (cid:88) v

Original: [Equation]
Revised: [Equation]

Original: In summary, the upper bound of the sensitivity is,
Revised: In summary, the upper bound of the sensitivity is,

Original: [Equation]
Revised: [Equation]

Original: C D ETAILS ABOUT THE D ATASETS
Revised: C D ETAILS ABOUT THE D ATASETS

Original: The AirDialog dataset (Wei et al., 2018) consists of 402,038 dialogues and each dialogue consists of more than two utterances.
Revised: The AirDialog dataset (Wei et al., 2018) consists of 402,038 dialogues and each dialogue consists of more than two utterances.

Original: We treat each utterance as a sentence in our sentence completion task.
Revised: We treat each utterance as a sentence in our sentence completion task.

Original: We obtain the Europarl v6 dataset from a machine translation benchmark 

Revised: We obtain the Europarl v6 dataset from a machine translation benchmark

Original: and we only use the monolingual English dataset with 2,015,440 raw sentences.
Revised: and we only use the monolingual English dataset with 2,015,440 raw sentences.

Original: For the above datasets, we filter the short sentence with less than eight tokens.
Revised: For the above datasets, we filter the short sentence with less than eight tokens.

Original: Then, the first four tokens act as the prefix, and the rest of the tokens acts as the output (ground-truth).
Revised: Then, the first four tokens act as the prefix, and the rest of the tokens acts as the output (ground-truth).

Original: We split each datasets into a private set D pri and a public set D pub .
Revised: We split each datasets into a private set D pri and a public set D pub .

Original: For the AirDialog dataset, the private set contains 0.95M/5K/50K samples for training/validation/testing and the public set contains 40K/5K for training/validation.
Revised: For the AirDialog dataset, the private set contains 0.95M/5K/50K samples for training/validation/testing and the public set contains 40K/5K for training/validation.

Original: For the Europarl v6 dataset, the private set contains 1.72M/10K/50K samples for training/validation/testing and the public set contains 40K/5K for training/validation.
Revised: For the Europarl v6 dataset, the private set contains 1.72M/10K/50K samples for training/validation/testing and the public set contains 40K/5K for training/validation.

Original: The vocabulary size for the two datasets is set to 50K and we replace the tokens out of the vocabulary with a special token.
Revised: The vocabulary size for the two datasets is set to 50K and we replace the tokens out of the vocabulary with a special token.

Original: D D ETAILS ABOUT THE E XPERIMENTAL S ETTING
Revised: D D ETAILS ABOUT THE E XPERIMENTAL S ETTING

Original: All the comparing algorithms use the same base model, the GPT-small, which has 12 stacked layers as mentioned in the original paper (Radford et al., 2018).
Revised: All the comparing algorithms use the same base model, the GPT-small, which has 12 stacked layers as mentioned in the original paper (Radford et al., 2018).

Original: The pre-trained GPT model comes from the official website 7 .
Revised: The pre-trained GPT model comes from the official website 7 .

Original: All the methods are initialized by the pre-trained GPT model.
Revised: All the methods are initialized by the pre-trained GPT model.

Original: We used the
Revised: We used the

Original: Adam optimization scheme (Kingma & Ba, 2015) and we adjust the initial learning rate for all the comparing methods with a range of 0.01 ∼ 0.00001, which means we conducted a systematic search of the hyper-parameter space among all methods.
Revised: Adam optimization scheme (Kingma & Ba, 2015) and we adjust the initial learning rate for all the comparing methods with a range of 0.01 ∼ 0.00001, which means we conducted a systematic search of the hyper-parameter space among all methods.

Original: We set the batch size as 32 for all the comparing baselines.
Revised: We set the batch size as 32 for all the comparing baselines.

Original: We truncate the sentences with the maximal sentence length of 40.
Revised: We truncate the sentences with the maximal sentence length of 40.

Original: In the top- p strategy, the threshold µ is 0.95.
Revised: In the top- p strategy, the threshold µ is 0.95.

Original: The factor λ balancing the KL loss and NLL loss mentioned in Eq.
Revised: The factor λ balancing the KL loss and NLL loss mentioned in Eq.

Original: 2 is 20.
Revised: 2 is 20.

Original: E D ETAILS ABOUT THE E VALUATION M ETRICS
Revised: E D ETAILS ABOUT THE E VALUATION M ETRICS

Original: To measure the quality of generation, PPL is a widely used metrics in text generation, especially language modeling, which measures the perplexity of generating the next token.
Revised: To measure the quality of generation, PPL is a widely used metrics in text generation, especially language modeling, which measures the perplexity of generating the next token.

Original: B-n (Bleu-n) (Papineni et al., 2002) measures the degree of the n-gram matching between the generated results and the ground-truth text.
Revised: B-n (Bleu-n) (Papineni et al., 2002) measures the degree of the n-gram matching between the generated results and the ground-truth text.

Original: To measure the performance of privacy protection (we can also treat it as the privacy leak of the models), we aim to give an intuitive understanding of the protection on text generation scenario (even if the factor ε in DP indicates the strength of privacy protection according to the DP theorem).
Revised: To measure the performance of privacy protection (we can also treat it as the privacy leak of the models), we aim to give an intuitive understanding of the protection on text generation scenario (even if the factor ε in DP indicates the strength of privacy protection according to the DP theorem).

Original: Hence, we propose a new metric, P-N, indicating how many generated n-grams can be found in the private set D pri .
Revised: Hence, we propose a new metric, P-N, indicating how many generated n-grams can be found in the private set D pri .

Original: We define N Y as the set of unique n-grams in a generated sentence Y and define N D pri as the set of unique n-grams in the private data D pri .
Revised: We define N Y as the set of unique n-grams in a generated sentence Y and define N D pri as the set of unique n-grams in the private data D pri .

Original: As shown in Eq.
Revised: As shown in Eq.

Original: 4, P-N is the ratio of the n-grams that occurs in the two sets over n-grams occurs only in N Y weighted by idf , where the idf is the inverse frequency of this n-gram indicating the sensitivity of this n-gram (Rare n-grams are likely to carry sensitive information, such as SSN number and address).
Revised: 4, P-N is the ratio of the n-grams that occurs in the two sets over n-grams occurs only in N Y weighted by idf , where the idf is the inverse frequency of this n-gram indicating the sensitivity of this n-gram (Rare n-grams are likely to carry sensitive information, such as SSN number and address).

Original: P-N ranges from 0 to 1 and lower P-N indicates the more strong privacy protection (less privacy leak from the models).
Revised: P-N ranges from 0 to 1 and lower P-N indicates the more strong privacy protection (less privacy leak from the models).

Original: For the choice of value N, we note that P-N with a very small value of N can hardly reflect the privacy loss since the uni-grams and bi-grams are very easy to be generated even if the model did not meet it from the private dataset D pri .
Revised: For the choice of value N, we note that P-N with a very small value of N can hardly reflect the privacy loss since the uni-grams and bi-grams are very easy to be generated even if the model did not meet it from the private dataset D pri .

Original: For example, the N=1 does not make sense because it only reflects the vocabulary overlapping between two datasets.
Revised: For example, the N=1 does not make sense because it only reflects the vocabulary overlapping between two datasets.

Original: Thus, we choose 3 and 4 as the value of N.
Revised: Thus, we choose 3 and 4 as the value of N.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: F A NALYSES ABOUT THE N UMBER OF T EACHER M ODELS
Revised: F A NALYSES ABOUT THE N UMBER OF T EACHER M ODELS

Original: Table 4 and Table 5 show the analyses about SeqPATE’s performance with different teacher number on our two datasets.
Revised: Table 4 and Table 5 show the analyses about SeqPATE’s performance with different teacher number on our two datasets.

Original: G T HE COMPUTATIONAL COST OF S EQ PATE
Revised: G T HE COMPUTATIONAL COST OF S EQ PATE

Original: It seems that our model requires huge computational resources and a costly infrastructure to run.
Revised: It seems that our model requires huge computational resources and a costly infrastructure to run.

Original: But in fact, our model is able to train and inference on a single GPU machine.
Revised: But in fact, our model is able to train and inference on a single GPU machine.

Original: In this section, we will introduce some simple strategies we used in our implementation and also introduce the total computational cost of our model.
Revised: In this section, we will introduce some simple strategies we used in our implementation and also introduce the total computational cost of our model.

Original: Memory usage and hard-disk space usage.
Revised: Memory usage and hard-disk space usage.

Original: Since our method uses a large number of teachers, the naive implementation of loading all teachers into memory for aggregation is impractical.
Revised: Since our method uses a large number of teachers, the naive implementation of loading all teachers into memory for aggregation is impractical.

Original: However, note that our algorithm only needs to access each teacher’s top-
Revised: However, note that our algorithm only needs to access each teacher’s top-

Original: k prediction.
Revised: k prediction.

Original: Therefore, we train teacher models sequentially.
Revised: Therefore, we train teacher models sequentially.

Original: Once a teacher model is trained, we obtain its top- k prediction ( k = in our experiments) on the public training data and save the results (i.e. k probabilities).
Revised: Once a teacher model is trained, we obtain its top- k prediction ( k = in our experiments) on the public training data and save the results (i.e. k probabilities).

Original: Then, we discard the teacher model.
Revised: Then, we discard the teacher model.

Original: Finally, SeqPATE only needs teacher supervision on a small number of samples.
Revised: Finally, SeqPATE only needs teacher supervision on a small number of samples.

Original: In our experiments, training on 1 ∼ 2k teacher labeled samples is sufficient.
Revised: In our experiments, training on 1 ∼ 2k teacher labeled samples is sufficient.

Original: Overall, saving teachers’ inference results uses 8 ∼ 16G.
Revised: Overall, saving teachers’ inference results uses 8 ∼ 16G.

Original: The memory usage is similar to that of a GPT2 model because we do not load all teacher models into memory and instead run inference sequentially and merge teachers’ predictions offline.
Revised: The memory usage is similar to that of a GPT2 model because we do not load all teacher models into memory and instead run inference sequentially and merge teachers’ predictions offline.

Original: Training time.
Revised: Training time.

Original: While we have a large number of teachers, each teacher is trained on only a small fraction of the entire dataset.
Revised: While we have a large number of teachers, each teacher is trained on only a small fraction of the entire dataset.

Original: Thus the time it takes to train all teachers is roughly equal to the time of training a single GPT2 model on the full dataset (of 1 ∼ 2M samples in our experiments).
Revised: Thus the time it takes to train all teachers is roughly equal to the time of training a single GPT2 model on the full dataset (of 1 ∼ 2M samples in our experiments).

Original: In summary, with simple tricks, the teacher training and aggregation steps are not much more expensive than training a GPT2 model.
Revised: In summary, with simple tricks, the teacher training and aggregation steps are not much more expensive than training a GPT2 model.

Original: Compared to standard NLG model training, our algorithm does not require special hardware or distributed learning.
Revised: Compared to standard NLG model training, our algorithm does not require special hardware or distributed learning.

Original: H A DDITIONAL E XPERIMENTS ON L ARGER ε AND L ARGER δ
Revised: H A DDITIONAL E XPERIMENTS ON L ARGER ε AND L ARGER δ

Original: We adopted stronger privacy protection at ε =2 and δ =1e-9 in Table 1, which results in a quite large utility gap between SeqPATE and the non-private training method (Pri-GPT) on the AirDialog dataset.
Revised: We adopted stronger privacy protection at ε =2 and δ =1e-9 in Table 1, which results in a quite large utility gap between SeqPATE and the non-private training method (Pri-GPT) on the AirDialog dataset.

Original: Actually, our model performance can be increased a lot if we conduct weaker privacy protection (i.e. using a larger ε and a larger δ ).
Revised: Actually, our model performance can be increased a lot if we conduct weaker privacy protection (i.e. using a larger ε and a larger δ ).

Original: In Table 6, we show the experimental results with ε =3 and δ =1e-6 on
Revised: In Table 6, we show the experimental results with ε =3 and δ =1e-6 on

Original: AirDialog dataset.
Revised: AirDialog dataset.

Original: Methods
Revised: Methods

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (Ours
Revised: (Ours

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I H ANDLE L ARGE ε : U SING F EWER T EACHERS
Revised: I H ANDLE L ARGE ε : U SING F EWER T EACHERS

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, when we use large ε , our SeqPATE cannot obtain better performance than
Revised: 2, when we use large ε , our SeqPATE cannot obtain better performance than

Original: DP-SGD.
Revised: DP-SGD.

Original: As discussed in Sec. 6.2, the reason is that the amount of training samples for each teacher model is quite small in our model (especially, when the teacher number is large).
Revised: As discussed in Sec. 6.2, the reason is that the amount of training samples for each teacher model is quite small in our model (especially, when the teacher number is large).

Original: We note that this can be improved by using fewer teachers.
Revised: We note that this can be improved by using fewer teachers.

Original: As shown in Table 7, we conduct experiments on AirDialog dataset with fewer (i.e. 100) teachers and larger ε (i.e. 50 and 500), where SeqPATE outperforms
Revised: As shown in Table 7, we conduct experiments on AirDialog dataset with fewer (i.e. 100) teachers and larger ε (i.e. 50 and 500), where SeqPATE outperforms

Original: DP-SGD.
Revised: DP-SGD.

Original: For DP-SGD, we choose Pub-GPT+ DP-SGD instead of Pub-GPT+DP-SGD+ ˜ D pub since
Revised: For DP-SGD, we choose Pub-GPT+ DP-SGD instead of Pub-GPT+DP-SGD+ ˜ D pub since

Original: Pub-GPT+ DP-SGD obtains higher performance when ε is large.
Revised: Pub-GPT+ DP-SGD obtains higher performance when ε is large.

Original: [Table]
Revised: [Table]

Original: J E VALUATING P RIVACY L EAK IN A NOTHER M ETRIC (E XPOSURE )
Revised: J E VALUATING P RIVACY L EAK IN A NOTHER M ETRIC (E XPOSURE )

Original: We have evaluated the performance of privacy protection in P-3 and P-4.
Revised: We have evaluated the performance of privacy protection in P-3 and P-4.

Original: We also conduct experiments on another metric, exposure , to evaluate the privacy leak.
Revised: We also conduct experiments on another metric, exposure , to evaluate the privacy leak.

Original: The exposure is proposed by (Carlini et al., 2019).
Revised: The exposure is proposed by (Carlini et al., 2019).

Original: To calculate the score, we use three kinds of patterns “is < word > | are < word > | < digital > ”.
Revised: To calculate the score, we use three kinds of patterns “is < word > | are < word > | < digital > ”.

Original: The repetitions r are 100 for all patterns, and the lengths of sampled words or digits are three.
Revised: The repetitions r are 100 for all patterns, and the lengths of sampled words or digits are three.

Original: The candidate words are randomly sampled from the whole vocabulary and the digits are randomly sampled from { 0 , 1 , ..., 9 } .
Revised: The candidate words are randomly sampled from the whole vocabulary and the digits are randomly sampled from { 0 , 1 , ..., 9 } .

Original: We calculate the average exposure score over the three kinds of patterns.
Revised: We calculate the average exposure score over the three kinds of patterns.

Original: We follow the implementation of the exposure score from 8 .
Revised: We follow the implementation of the exposure score from 8 .

Original: [Table]
Revised: [Table]

Original: Table 8 shows the experimental results, where the results are consistent with the results on P-3 and
Revised: Table 8 shows the experimental results, where the results are consistent with the results on P-3 and

Original: P-4 mentioned in Table 1.
Revised: P-4 mentioned in Table 1.

Original: For example, in both metrics, the privacy leak in AirDialog is more serious than that in Europarl v6 and the gap between Pri-GPT and SeqPATE on AirDialog is larger than that on Europarl v6.
Revised: For example, in both metrics, the privacy leak in AirDialog is more serious than that in Europarl v6 and the gap between Pri-GPT and SeqPATE on AirDialog is larger than that on Europarl v6.

Original: K A DDITIONAL E XPERIMENTS W HEN THE D ATASET IS S MALLER
Revised: K A DDITIONAL E XPERIMENTS W HEN THE D ATASET IS S MALLER

Original: To verify our performance on smaller datasets, we produce a small dataset by randomly sampling 50k samples from the AirDialog dataset.
Revised: To verify our performance on smaller datasets, we produce a small dataset by randomly sampling 50k samples from the AirDialog dataset.

Original: The experimental results are shown in Table 9.
Revised: The experimental results are shown in Table 9.

Original: In such a small dataset, our method still outperforms the DP-SGD with ε = 2 and δ = 1e-9.
Revised: In such a small dataset, our method still outperforms the DP-SGD with ε = 2 and δ = 1e-9.

Original: [Table]
Revised: [Table]

Original: L T HE C ONTRIBUTION OF THE P SEUDO P UBLIC D ATASET ˜ D pub
Revised: L T HE C ONTRIBUTION OF THE P SEUDO P UBLIC D ATASET ˜ D pub

Original: We conducted experiments to verify the contribution of pseudo dataset ˜ D pub to our task on AirDialog dataset.
Revised: We conducted experiments to verify the contribution of pseudo dataset ˜ D pub to our task on AirDialog dataset.

Original: The results in Table 10 shows that using ˜ D pub can prompt the performance a lot
Revised: The results in Table 10 shows that using ˜ D pub can prompt the performance a lot

Original: [Table]
Revised: [Table]

Original: where the promotion can be found in both SeqPATE and DP-SGD methods.
Revised: where the promotion can be found in both SeqPATE and DP-SGD methods.

Original: [Table]
Revised: [Table]

Original: GPT
Revised: GPT

Original: [Table]
Revised: [Table]

Original: DP
Revised: DP

Original: [Table]
Revised: [Table]

Original: SGD+
Revised: SGD+

Original: [Table]
Revised: [Table]

Original: M
Revised: M

Original: T HE I LLUSTRATION
Revised: T HE I LLUSTRATION

Original: OF A R UNNING E XAMPLE .
Revised: OF A R UNNING E XAMPLE .

Original: Here, we will use an example to show our training processing.
Revised: Here, we will use an example to show our training processing.

Original: In this example, the prefix from the public dataset D pub is “I want to book”.
Revised: In this example, the prefix from the public dataset D pub is “I want to book”.

Original: We feed the prefix to pre-trained GPT2 to generate a pseudo sentence “I want to book a flight from Tokyo to Hawaii.”.
Revised: We feed the prefix to pre-trained GPT2 to generate a pseudo sentence “I want to book a flight from Tokyo to Hawaii.”.

Original: The pseudo sentence will serve as an example in the pseudo public dataset ˜ D
Revised: The pseudo sentence will serve as an example in the pseudo public dataset ˜ D

Original: pub .
Revised: pub .

Original: We feed the pseudo sentence to the teacher models to conduct the teacher inference and also feed it to the student model to conduct the feed-forward of the student training.
Revised: We feed the pseudo sentence to the teacher models to conduct the teacher inference and also feed it to the student model to conduct the feed-forward of the student training.

Original: Teacher models output the probability distributions on all words (10 words, in total) of the sentence.
Revised: Teacher models output the probability distributions on all words (10 words, in total) of the sentence.

Original: Then, we aggregate all teachers’ probability distributions and add the calibrated noise on the aggregated distributions.
Revised: Then, we aggregate all teachers’ probability distributions and add the calibrated noise on the aggregated distributions.

Original: The student model also generates the corresponding probability distributions on those words and we conduct the efficient knowledge distillation and top- k or top- p selection over the student’s probability distributions.
Revised: The student model also generates the corresponding probability distributions on those words and we conduct the efficient knowledge distillation and top- k or top- p selection over the student’s probability distributions.

Original: For example, if the student model can do well on the words (“I”, “want”, “book”, “flight”, and “Tokyo”), the student will only query the teachers’ output distributions on the rest of words (“to”, “from”, “to”, and “Hawaii”) through the KL loss mentioned in 2.
Revised: For example, if the student model can do well on the words (“I”, “want”, “book”, “flight”, and “Tokyo”), the student will only query the teachers’ output distributions on the rest of words (“to”, “from”, “to”, and “Hawaii”) through the KL loss mentioned in 2.

Original: Besides, the student is always supervised by the NLL loss on the whole pseudo sentence (“I want to book a flight from Tokyo to Hawaii.”).
Revised: Besides, the student is always supervised by the NLL loss on the whole pseudo sentence (“I want to book a flight from Tokyo to Hawaii.”).

Original: Finally, the student model conducts back-propagation according to the above losses.
Revised: Finally, the student model conducts back-propagation according to the above losses.