Original: A UGMENTATION WITH P ROJECTION : T OWARDS AN E FFECTIVE AND E FFICIENT D ATA A UGMENTATION P ARADIGM FOR D ISTILLATION
Revised: A UGMENTATION WITH P ROJECTION : T OWARDS AN E FFECTIVE AND E FFICIENT D ATA A UGMENTATION P ARADIGM FOR D ISTILLATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Knowledge distillation is one of the primary methods of transferring knowledge from large to small models.
Revised: Knowledge distillation is one of the primary methods of transferring knowledge from large to small models.

Original: However, it requires massive task-specific data, which may not be plausible in many real-world applications.
Revised: However, it requires massive task-specific data, which may not be plausible in many real-world applications.

Original: Data augmentation methods such as representation interpolation, token replacement, or augmentation with models are applied to tackle this problem.
Revised: Data augmentation methods such as representation interpolation, token replacement, or augmentation with models are applied to tackle this problem.

Original: However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models).
Revised: However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models).

Original: To this end, we propose AugPro ( Aug mentation with Pro jection), an effective and efficient data augmentation method for distillation.
Revised: To this end, we propose AugPro ( Aug mentation with Pro jection), an effective and efficient data augmentation method for distillation.

Original: Our method builds on top of representation interpolation augmentation methods to maintain the diversity of expressions and converts the augmented data to tokens to avoid shifting decision boundaries.
Revised: Our method builds on top of representation interpolation augmentation methods to maintain the diversity of expressions and converts the augmented data to tokens to avoid shifting decision boundaries.

Original: It uses simple operations that come with little computational overhead.
Revised: It uses simple operations that come with little computational overhead.

Original: The results on multiple
Revised: The results on multiple

Original: GLUE tasks show that our methods can improve distillation performance by a large margin at a low time cost.
Revised: GLUE tasks show that our methods can improve distillation performance by a large margin at a low time cost.

Revised: Codes are available at https://github.com/ google-research/google-research/tree/master/augpro .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale language models (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Zhang et al., 2022c) have achieved great success on various natural language processing (NLP) tasks, such as information extraction (Lu et al., 2021) and question answering (Kassner & Sch ¨ utze, 2020).
Revised: Large-scale language models (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Zhang et al., 2022c) have achieved great success on various natural language processing (NLP) tasks, such as information extraction (Lu et al., 2021) and question answering (Kassner & Sch ¨ utze, 2020).

Original: However, large-scale models have high computational overhead, which limits their deployment in edge devices and fast response scenarios (Sun et al., 2020b).
Revised: However, large-scale models have high computational overhead, which limits their deployment in edge devices and fast response scenarios (Sun et al., 2020b).

Original: One widely used solution is to perform knowledge distillation (Hinton et al., 2015) from large-scale models to small-scale models.
Revised: One widely used solution is to perform knowledge distillation (Hinton et al., 2015) from large-scale models to small-scale models.

Original: This method, however, usually requires a large amount of data to guarantee the transfer quality, which may not be easily obtained in real-world applications.
Revised: This method, however, usually requires a large amount of data to guarantee the transfer quality, which may not be easily obtained in real-world applications.

Original: To this end, data augmentation methods are applied (Liang et al., 2020; Wang & Yang, 2020; Zhang et al., 2022b) to improve the distillation performance.
Revised: To this end, data augmentation methods are applied (Liang et al., 2020; Wang & Yang, 2020; Zhang et al., 2022b) to improve the distillation performance.

Original: There are three major types of data augmentation methods: (1) Representation interpolation.
Revised: There are three major types of data augmentation methods: (1) Representation interpolation.

Original: For example, Liang et al.
Revised: For example, Liang et al.

Original: (2020), Chen et al.
Revised: (2020), Chen et al.

Original: (2020a)
Revised: (2020a)

Original: and Sun et al.
Revised: and Sun et al.

Original: (2020a) apply linear interpolation (Zhang et al., 2017) to word embeddings, hidden states between transformer layers, and encoder outputs, respectively, to augment the original dataset with virtual data points.
Revised: (2020a) apply linear interpolation (Zhang et al., 2017) to word embeddings, hidden states between transformer layers, and encoder outputs, respectively, to augment the original dataset with virtual data points.

Original: Data points are virtual because they are not real language inputs.
Revised: Data points are virtual because they are not real language inputs.

Original: Instead, they are representations (e.g., embeddings).
Revised: Instead, they are representations (e.g., embeddings).

Original: Token replacement.
Revised: Token replacement.

Original: Kobayashi (2018) replaces tokens with their synonyms.
Revised: Kobayashi (2018) replaces tokens with their synonyms.

Original: Easy Data augmentation (Wei & Zou, 2019) combines synonym replacement, random insertion, random swap, and random deletion.
Revised: Easy Data augmentation (Wei & Zou, 2019) combines synonym replacement, random insertion, random swap, and random deletion.

Original: (3) Augmentation with models.
Revised: (3) Augmentation with models.

Original: Yoo et al.
Revised: Yoo et al.

Original: (2021) and Zhou et al.
Revised: (2021) and Zhou et al.

Original: (2021) use GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020) respectively as the language model to generate new text data of similar types.
Revised: (2021) use GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020) respectively as the language model to generate new text data of similar types.

Original: (1) supports many operations such as linear interpolation (Zhang et al., 2017) and small perturbation (Madry et al., 2017).
Revised: (1) supports many operations such as linear interpolation (Zhang et al., 2017) and small perturbation (Madry et al., 2017).

Original: It makes the methods very expressive in generating a diverse range of data.
Revised: It makes the methods very expressive in generating a diverse range of data.

Original: However, the newly generated representations (e.g., embeddings) may sit outside of the real data distribution.
Revised: However, the newly generated representations (e.g., embeddings) may sit outside of the real data distribution.

Original: For instance, word embeddings are converted from a vocabulary in the text domain.
Revised: For instance, word embeddings are converted from a vocabulary in the text domain.

Original: Performing augmentation at this level may result in representations that do not have their counterparts in the vocabulary.
Revised: Performing augmentation at this level may result in representations that do not have their counterparts in the vocabulary.

Original: As a result, the augmented data may mislead the model to generate a shifted decision boundary that can largely affect the qualities (Section 3).
Revised: As a result, the augmented data may mislead the model to generate a shifted decision boundary that can largely affect the qualities (Section 3).

Original: (2) can generate in-domain data easily.
Revised: (2) can generate in-domain data easily.

Original: By using synonym replacement (Wang & Yang, 2015), new data can be obtained at a low cost.
Revised: By using synonym replacement (Wang & Yang, 2015), new data can be obtained at a low cost.

Original: Despite this good property, this stream of methods lacks the ability to generate diversified data.
Revised: Despite this good property, this stream of methods lacks the ability to generate diversified data.

Original: Subsequently, they contribute little to sampling low-resource data areas and limit the performance gains in practice.
Revised: Subsequently, they contribute little to sampling low-resource data areas and limit the performance gains in practice.

Original: (3) generates both diversified and in-domain data using large language models such as GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020).
Revised: (3) generates both diversified and in-domain data using large language models such as GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020).

Original: Due to their large computational overheads, on the other hand, the final distillation quality will be highly limited to the amount of generated data, which is usually not affordable to the scale of even tens of thousands of sentences in practice.
Revised: Due to their large computational overheads, on the other hand, the final distillation quality will be highly limited to the amount of generated data, which is usually not affordable to the scale of even tens of thousands of sentences in practice.

Original: Figure 1 summarizes the advantages of each augmentation method.
Revised: Figure 1 summarizes the advantages of each augmentation method.

Original: [Figure]
Revised: [Figure]

Original: Considering all the approaches above, we propose AugPro, an effective and efficient data augmentation method for the distillation scenario, which absorbs the advantages above without being limited by their drawbacks.
Revised: Considering all the approaches above, we propose AugPro, an effective and efficient data augmentation method for the distillation scenario, which absorbs the advantages above without being limited by their drawbacks.

Original: Specifically,
Revised: Specifically,

Original: AugPro: (1) (effectiveness) is as expressive as representation interpolation; (2) (effectiveness) does not mislead decision boundaries; (3) (efficiency) has low computational overhead.
Revised: AugPro: (1) (effectiveness) is as expressive as representation interpolation; (2) (effectiveness) does not mislead decision boundaries; (3) (efficiency) has low computational overhead.

Original: In distillation settings, we can always use the teacher to label the hallucinated data in the knowledge distillation scenario.
Revised: In distillation settings, we can always use the teacher to label the hallucinated data in the knowledge distillation scenario.

Original: This suggests that we can encourage AugPro to produce as diverse data as possible that are not limit to instances with only the same or flipped labels.
Revised: This suggests that we can encourage AugPro to produce as diverse data as possible that are not limit to instances with only the same or flipped labels.

Original: Concretely, our method builds on top of representation interpolation augmentation methods (property (1)), which does not constrain the generated data to be within small regions of their “parents”.
Revised: Concretely, our method builds on top of representation interpolation augmentation methods (property (1)), which does not constrain the generated data to be within small regions of their “parents”.

Original: The key of AugPro is to convert the augmented data to the format of tokens through projections (property (2)) with low-cost operations (property (3)).
Revised: The key of AugPro is to convert the augmented data to the format of tokens through projections (property (2)) with low-cost operations (property (3)).

Original: We conduct our experiments on GLUE (Wang et al., 2018) datasets.
Revised: We conduct our experiments on GLUE (Wang et al., 2018) datasets.

Original: Results show that our method could boost the distillation performance significantly with low computational overhead.
Revised: Results show that our method could boost the distillation performance significantly with low computational overhead.

Original: To sum up, our contributions are:
Revised: To sum up, our contributions are:

Original: • We proposeAugPro, an effective and efficient data augmentation method for knowledge distillation.
Revised: • We propose an effective and efficient data augmentation method for knowledge distillation.

Original: • We empirically evaluate the effectiveness and efficiency of AugPro and theoretically examine that AugPro satisfies three properties under certain circumstances.
Revised: • We empirically evaluate the effectiveness and efficiency of AugPro and theoretically examine that AugPro satisfies three properties under certain circumstances.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Knowledge Distillation Knowledge distillation was first proposed by (Hinton et al., 2015).
Revised: Knowledge Distillation Knowledge distillation was first proposed by (Hinton et al., 2015).

Original: It aims to distill knowledge from one model to another by minimizing the distance between the outputs of two models on the same input.
Revised: It aims to distill knowledge from one model to another by minimizing the distance between the outputs of two models on the same input.

Original: With the rise of transformers (Vaswani et al., 2017) and BERT (Devlin et al., 2018), more and more attention has been paid to the distillation of pre-training language models.
Revised: With the rise of transformers (Vaswani et al., 2017) and BERT (Devlin et al., 2018), more and more attention has been paid to the distillation of pre-training language models.

Original: Tang et al.
Revised: Tang et al.

Original: (2019) distill fine-tuned BERT to a single-layer BiLSTM network and makes the BiLSTM network as good as ELMo (Peters et al., 2018).
Revised: (2019) distill fine-tuned BERT to a single-layer BiLSTM network and makes the BiLSTM network as good as ELMo (Peters et al., 2018).

Original: Sun et al.
Revised: Sun et al.

Original: (2019) not only distill from outputs but also distill from teacher models’ hidden layers.
Revised: (2019) not only distill from outputs but also distill from teacher models’ hidden layers.

Original: These methods distill language models in the fine-tuning stage, whereas Sanh et al.
Revised: These methods distill language models in the fine-tuning stage, whereas Sanh et al.

Original: and Sun et al.
Revised: and Sun et al.

Original: (2020b) focus on distilling language models in the pre-training stage directly to make student models task-agnostic.
Revised: (2020b) focus on distilling language models in the pre-training stage directly to make student models task-agnostic.

Original: TinyBERT (Jiao et al., 2019) distill BERT from both the pre-training and fine-tuning stages.
Revised: TinyBERT (Jiao et al., 2019) distill BERT from both the pre-training and fine-tuning stages.

Original: In our work, we focus on a widely used setting.
Revised: We focus on a widely used setting.

Original: We distill knowledge in the fine-tuning stage by minimizing the distance between two models’ outputs.
Revised: We distill knowledge in the fine-tuning stage by minimizing the distance between two models’ outputs.

Original: Data Augmentation Representation interpolation methods are popular in the computer vision research community.
Revised: Data Augmentation Representation interpolation methods are popular in the computer vision research community.

Original: MixUp (Zhang et al., 2017) uses linear interpolation to get augmented images and labels.
Revised: MixUp (Zhang et al., 2017) uses linear interpolation to get augmented images and labels.

Original: Given an image x 1 , x 2 and their labels y 1 , y 2 , MixUp uses a linear interpolation to generate a new data point x ′ and its label y ′ :
Revised: Given an image x 1 , x 2 and their labels y 1 , y 2 , MixUp uses a linear interpolation to generate a new data point x ′ and its label y ′ :

Original: [Equation]
Revised: [Equation]

Original: FGSM (Goodfellow et al., 2014) and PGA (Madry et al., 2017) use gradients to generate adversarial examples.
Revised: FGSM (Goodfellow et al., 2014) and PGA (Madry et al., 2017) use gradients to generate adversarial examples.

Original: Given an image x , FGSM will generate new data x ′ :
Revised: Given an image x , FGSM will generate new data x ′ :

Original: [Equation]
Revised: [Equation]

Original: where L is the loss of a specific task and ϵ is a small value.
Revised: where L is the loss of a specific task and ϵ is a small value.

Original: x ′ and x have the same label.
Revised: x ′ and x have the same label.

Original: CutMix (Yun et al., 2019) cuts images and then concatenates them together to get a new one.
Revised: CutMix (Yun et al., 2019) cuts images and then concatenates them together to get a new one.

Original: Though these methods were originally designed for images, they can be adapted to NLP tasks.
Revised: Though these methods were originally designed for images, they can be adapted to NLP tasks.

Original: Liang et al.
Revised: Liang et al.

Original: (2020) use MixUp on word embeddings to augment data for knowledge distillation.
Revised: (2020) use MixUp on word embeddings to augment data for knowledge distillation.

Original: Chen et al.
Revised: Chen et al.

Original: (2020b) use
Revised: (2020b) use

Original: MixUp on hidden states between transformer layers.
Revised: MixUp on hidden states between transformer layers.

Original: Jindal et al.
Revised: Jindal et al.

Original: (2020) also use MixUp on hidden states but consider the effect of mean and variance.
Revised: (2020) also use MixUp on hidden states but consider the effect of mean and variance.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022b) apply PGA to student models’ embeddings and leave teacher models’ embeddings unchanged, finding that PGA can benefit knowledge distillation.
Revised: (2022b) apply PGA to student models’ embeddings and leave teacher models’ embeddings unchanged, finding that PGA can benefit knowledge distillation.

Original: Token replacement methods mainly focus on language inputs.
Revised: Token replacement methods mainly focus on language inputs.

Original: Synonym replacement methods (Kobayashi, 2018) replace tokens with their synonyms.
Revised: Synonym replacement methods (Kobayashi, 2018) replace tokens with their synonyms.

Original: Easy data augmentation (EDA) (Wei & Zou, 2019) incorporates synonym replacement, random insertion, random swap, and random deletion.
Revised: Easy data augmentation (EDA) (Wei & Zou, 2019) incorporates synonym replacement, random insertion, random swap, and random deletion.

Original: TreeMix (Zhang et al., 2022a) uses a constituency parser to decide which token should be replaced.
Revised: TreeMix (Zhang et al., 2022a) uses a constituency parser to decide which token should be replaced.

Original: Augmentation with models is another approach to generating new data.
Revised: Augmentation with models is another approach to generating new data.

Original: FlipDA (Zhou et al., 2021) uses T5 to generate data that has flipped labels.
Revised: FlipDA (Zhou et al., 2021) uses T5 to generate data that has flipped labels.

Original: GPT3Mix (Yoo et al., 2021) designs prompts and uses GPT3 to generate new data.
Revised: GPT3Mix (Yoo et al., 2021) designs prompts and uses GPT3 to generate new data.

Original: Back translation (Yu et al., 2018) uses neural networks to translate inputs to another language and then translate them back.
Revised: Back translation (Yu et al., 2018) uses neural networks to translate inputs to another language and then translate them back.

Original: Our method (AugPro) uses representation interpolation as the backbone and utilizes projection to convert representations to tokens with low-cost operations.
Revised: Our method (AugPro) uses representation interpolation as the backbone and utilizes projection to convert representations to tokens with low-cost operations.

Original:  M OTIVATING E XAMPLES
Revised:  M OTIVATING E XAMPLES

Original: Though representation interpolation methods have the good property of generating diverse data, we find that these vision techniques cannot be directly applied to NLP tasks.
Revised: Though representation interpolation methods have the good property of generating diverse data, we find that these vision techniques cannot be directly applied to NLP tasks.

Original: This is because representation interpolation augments data in a continuous manner, under which case the new data may never exist in the discrete input space, causing the decision boundary shifts.
Revised: This is because representation interpolation augments data in a continuous manner, under which case the new data may never exist in the discrete input space, causing the decision boundary shifts.

Original: Take an example of a simple two-dimensional problem with linear separability (Figure 2).
Revised: Take an example of a simple two-dimensional problem with linear separability (Figure 2).

Original: Let X
Revised: Let X

Original: = { x 1 , x 2 , x 3 , x 4 } be the universe of all data to learn, and Y = { y 1 , y 2 , y 3 , y 4 } be the corresponding labels.
Revised: = { x 1 , x 2 , x 3 , x 4 } be the universe of all data to learn, and Y = { y 1 , y 2 , y 3 , y 4 } be the corresponding labels.

Original: Suppose we know all of
Revised: Suppose we know all of

Original: X , Y and run the linear support vector machine (SVM) with a hard margin, i.e., min β,b ∥ β ∥ 2 such that y i ( β ⊤ x
Revised: X , Y and run the linear support vector machine (SVM) with a hard margin, i.e., min β,b ∥ β ∥ 2 such that y i ( β ⊤ x

Original: i + b ) ≥ 1 for all i , we get the solution β ∗ , b ∗ Since it is hard to get all data in the real-world setting, we suppose that we only have { x 1 , x 3 } as the training dataset.
Revised: i + b ) ≥ 1 for all i , we get the solution β ∗ , b ∗ Since it is hard to get all data in the real-world setting, we suppose that we only have { x 1 , x 3 } as the training dataset.

Original: If we simply use MixUp, we get x MixUp , as the augmented data, whose label y MixUp = sign(( β ∗ )
Revised: If we simply use MixUp, we get x MixUp , as the augmented data, whose label y MixUp = sign(( β ∗ )

Original: ⊤ x MixUp + b ∗ ) =
Revised: ⊤ x MixUp + b ∗ ) =

Original: y 

Revised: y

Original: = y 4 .
Revised: = y 4 .

Original: Now running the linear SVM with { x 2 , x 4 , x MixUp } with labels { y 2 , y 4 , y MixUp } , we get β MixUp and b MixUp .
Revised: Now running the linear SVM with { x 2 , x 4 , x MixUp } with labels { y 2 , y 4 , y MixUp } , we get β MixUp and b MixUp .

Original: Nevertheless, sign( β ⊤ MixUp x 3 + b MixUp ) ̸
Revised: Nevertheless, sign( β ⊤ MixUp x 3 + b MixUp ) ̸

Original: = y 3 .
Revised: = y 3 .

Original: As a comparison, if we project x
Revised: As a comparison, if we project x

Original: MixUp to its nearest neighbor and get x MixUp − P = x 2 whose label y MixUp − P = y 2 , running SVM with { x 1 , x 2 , x MixUp − P } and { y 1 , y 2 , y MixUp − P } can get βMixUp − P and b
Revised: MixUp to its nearest neighbor and get x MixUp − P = x 2 whose label y MixUp − P = y 2 , running SVM with { x 1 , x 2 , x MixUp − P } and { y 1 , y 2 , y MixUp − P } can get β

Original: MixUp − P  , which can classify all data correctly.
Revised: MixUp − P and b MixUp − P , which can classify all data correctly.

Original: The concret number of each parameter can be found in Appendix A.
Revised: Appendix B shows the concret number of each parameter.

Original: To this end, augmented data should be the real data in the input space, i.e., the format of tokens, to leverage this problem.
Revised: To this end, augmented data should be the real data in the input space, i.e., the format of tokens, to leverage this problem.

Original: This observation leads to our method AugPro which uses projection to convert augmented representations to symbolic tokens.
Revised: This observation leads to our method AugPro which uses projection to convert augmented representations to symbolic tokens.

Original: Compared with the virtual data point generated by representation interpolation, projection can explore more real data and leads to a lower error (Section 4.2 and Appendix G).
Revised: Compared with the virtual data point generated by representation interpolation, projection can explore more real data and leads to a lower error (Section 4.2 and Appendix H).

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we first formulate the definition of knowledge distillation in NLP (Hinton et al., 2015) and then introduce our method AugPro.
Revised: In this section, we first formulate the definition of knowledge distillation in NLP (Hinton et al., 2015) and then introduce our method AugPro.

Original: Knowledge distillation is a method to distill knowledge from large-scale models to small-scale models.
Revised: [Figure]

Original: Formally speaking, considering an NLP classification task, we have a corpus D = { ( x i , y i ) } Ni =1 that contains N input-output pairs, where x i is an input sentence with tokens x i = [ w i 1 , · · · , w i ni ] , w k ∈ xx2 x
Revised: xx

Original: [Figure]
Revised: [Figure]

Original: x
Revised: x

Original: [Figure]
Revised: [Figure]

Original: − +

Original: [Figure]

Original: [-xx 
 xxx-] MixUp − +
Revised: [-xx 
 xxx-] MixUp − +

Original: [Figure]
Revised: [Figure]

Original: MixUp SVM xx 2 xx− +
Revised: xx 2 xx− +

Original: Projection
Revised: Projection

Original: [Figure]
Revised: [Figure]

Revised: that the correction of projection

Revised: ) brings smaller decision boundary shifts than

Revised: 4. KNOWLEDGE DISTILLATION

Revised: Knowledge distillation is a method to distill knowledge from large-scale models to small-scale models.

Original:  V, V is the vocabulary, n i is the number of tokens in x i .
Revised: Formally speaking, considering an NLP classification task, we have a corpus D = { ( x i , y i ) } Ni =1 that contains N input-output pairs, where x i is an input sentence with tokens x i = [ w i 1 , · · · , w i ni ] , w k ∈ V, V is the vocabulary, n i is the number of tokens in x i .

Original: y i is the output label for x i .
Revised: y i is the output label for x i .

Original: We use plain texts rather than bold texts for x because language inputs are a sequence of tokens, which is different from images.
Revised: We use plain texts rather than bold texts for x because language inputs are a sequence of tokens, which is different from images.

Original: Then we distill knowledge from a large-scale model f ( · , θ T ) with parameter θ T (i.e., a teacher model) to a small-scale model g ( · , θ S ) with parameter θ S (i.e., a student model).
Revised: Then we distill knowledge from a large-scale model f ( · , θ T ) with parameter θ T (i.e., a teacher model) to a small-scale model g ( · , θ S ) with parameter θ S (i.e., a student model).

Original: In practice, θ T has much more parameters than θ S .
Revised: In practice, θ T has much more parameters than θ S .

Original: The distillation process can be divided into two stages:
Revised: The distillation process can be divided into two stages:

Original: • Teacher training.
Revised: • Teacher training.

Original: Optimize θ T on the dataset D .
Revised: Optimize θ T on the dataset D .

Original: In classification problems, we use cross-entropy loss to do empirical risk minimization on θ T 
Revised: In classification problems, we use cross-entropy loss to do empirical risk minimization on θ T :

Original: [Equation]
Revised: [Equation]

Original: • Student training.
Revised: • Student training.

Original: Optimize θ S on the dataset D with both ground-truth labels and outputs from teachers.
Revised: Optimize θ S on the dataset D with both ground-truth labels and outputs from teachers.

Original: In classification problems, θ
Revised: In classification problems

Original: [Equation]
Revised: [Equation]

Original: where d ( · , · ) is a distance function.
Revised: where d ( · , · ) is a distance function.

Original: In practice, d ( · , · ) could be cross-entropy or mean square error.
Revised: In practice, d ( · , · ) could be cross-entropy or mean square error.

Original: Empirical results from former studies (Hinton et al., 2015; Sun et al., 2020b; Sanh et al., 2019; Sun et al., 2019) show that knowledge distillation will train a better θ ′ S because the student model not only learns from ground-truth labels but also learns the generality from the teacher model.
Revised: Empirical results from former studies (Hinton et al., 2015; Sun et al., 2020b; Sanh et al., 2019; Sun et al., 2019) show that knowledge distillation will train a better θ ′ S because the student model not only learns from ground-truth labels but also learns the generality from the teacher model.

Original: Note for the student training, we can combine knowledge distillation and data augmentation together:
Revised: Note for the student training, we can combine knowledge distillation and data augmentation together:

Original: [Equation]
Revised: [Equation]

Original: where L Aug denotes the knowledge distillation loss on augmented data which leads to different variants of methods.
Revised: where L Aug denotes the knowledge distillation loss on augmented data which leads to different variants of methods.

Original: As one important way to help the student learn more effectively, how to generate new data with augmentation loss is the key and major discussion topic in the remaining sections.
Revised: As one important way to help the student learn more effectively, how to generate new data with augmentation loss is the key and major discussion topic in the remaining sections.

Original: 4. A UG P RO : A UGMENTATION WITH P ROJECTION
Revised: 4. A UG P RO : A UGMENTATION WITH P ROJECTION

Original: In this section, we will introduce four variants of L Aug : two backbones (MixUp and FGSM) and two
Revised: In this section, we will introduce four variants of L Aug : two backbones (MixUp and FGSM) and two

Original: AugPro variants building on top of them.
Revised: AugPro variants building on top of them.

Revised: Figure 4 shows the concept of our proposed method.

Original: AugPro builds on top of representation interpolation augmentation methods.
Revised: AugPro builds on top of representation interpolation augmentation methods.

Original: The pipeline (Algorithm 1) can be divided into three steps: (1) We first get augmented representations (i.e. h ( · ) ).
Revised: The pipeline (Algorithm 1) can be divided into three steps: (1) We first get augmented representations (i.e. h ( · ) ).

Original: Then we use projection to convert representations to tokens (i.e. p ( · ) )).
Revised: Then we use projection to convert representations to tokens (i.e. p ( · ) )).

Original: (3) At last, we compute L Aug to update student models.
Revised: (3) At last, we compute L Aug to update student models.

Original: The key to step (2) is projection.
Revised: The key to step (2) is projection.

Original: Concretely, language models map tokens to representations, and projection aims to find an inverse mapping to map representations back to tokens.
Revised: Concretely, language models map tokens to representations, and projection aims to find an inverse mapping to map representations back to tokens.

Original: This way, AugPro will avoid shifting decision boundaries (Section 3).
Revised: This way, AugPro will avoid shifting decision boundaries (Section 3).

Original: However, the inverse mapping is hard to find in practice.
Revised: However, the inverse mapping is hard to find in practice.

Original: First, popular language model architectures such as transformers (Vaswani et al., 2017) are usually complex and are hard to get the inverse mapping.
Revised: First, popular language model architectures such as transformers (Vaswani et al., 2017) are usually complex and are hard to get the inverse mapping.

Original: Second, the input space of language is discrete, making the mapping irreversible.
Revised: Second, the input space of language is discrete, making the mapping irreversible.

Original: Thus, we could only use the approximation technique to get the approximated inverse mapping.
Revised: Thus, we could only use the approximation technique to get the approximated inverse mapping.

Original: To this end, we focus on the inverse mapping on the embedding level.
Revised: To this end, we focus on the inverse mapping on the embedding level.

Original: First, the embedding mapping’s structure is much simpler than the transformer layer and is straightforward for us to find the inverse mapping.
Revised: First, the embedding mapping’s structure is much simpler than the transformer layer and is straightforward for us to find the inverse mapping.

Original: Second, we can use the nearest-neighbors as the approximation method, which is a cheap approximation.
Revised: Second, we can use the nearest-neighbors as the approximation method, which is a cheap approximation.

Original: Based on the analysis above, we use the nearest-neighbors to find our projection, i.e., the function p ( · ) in the Algorithm 1.
Revised: Based on the analysis above, we use the nearest-neighbors to find our projection, i.e., the function p ( · ) in the Algorithm 1.

Original: AugPro does not rely on specific representation interpolation methods.
Revised: AugPro does not rely on specific representation interpolation methods.

Original: In this paper, we apply AugPro to MixUp and FGSM, i.e. the h ( · ) in Algorithm 1 is MixUp or FGSM as in Equation (1,2).
Revised: In this paper, we apply AugPro to MixUp and FGSM, i.e. the h ( · ) in Algorithm 1 is MixUp or FGSM as in Equation (1,2).

Original: We will illustrate two variants to perform projection (step (2)) and compute
Revised: We will illustrate two variants to perform projection (step (2)) and compute

Original: L Aug loss (step (3)) in the following texts.
Revised: L Aug loss (step (3)) in the following texts.

Original: We slightly abuse the notion of f and g to illustrate AugPro better.
Revised: We slightly abuse the notion of f and g to illustrate AugPro better.

Original: We divide f and g into two parts: the first part is an embedding function that maps tokens to embedding vectors ( f e and g e , e denotes embeddings ), the rest is the second part ( f l and g l , l denotes layers ).
Revised: We divide f and g into two parts: the first part is an embedding function that maps tokens to embedding vectors ( f e and g e , e denotes embeddings ), the rest is the second part ( f l and g l , l denotes layers ).

Original: Under this definition, f = f l ◦ f e and g = g l ◦ g e .
Revised: Under this definition, f = f l ◦ f e and g = g l ◦ g e .

Original: AugPro-Mix We get AugPro-Mix by applying the AugPro paradigm to MixUp.
Revised: AugPro-Mix We get AugPro-Mix by applying the AugPro paradigm to MixUp.

Original: First, we apply MixUp on word embeddings and labels.
Revised: First, we apply MixUp on word embeddings and labels.

Original: This gives us embeddings from the teacher e f MixUp , embeddings from the student e g MixUp , and the label y MixUp .
Revised: This gives us embeddings from the teacher e f MixUp , embeddings from the student e g MixUp , and the label y MixUp .

Original: If we use MixUp data to construct losses, L Aug becomes 
Revised: L Aug becomes L MixUp = 1 M M (cid:88) j

Original: [Equation]
Revised: [Equation]

Revised: [ CrossEntropy (

Revised: [Equation]

Original:  where M denotes the number of augmented data.
Revised: [Equation] + [Equation] T ))] when we use MixUp data to construct losses, where M denotes the number of augmented data.

Original: For AugPro, we use the nearest-neighbors to get AugPro-Mix tokens x AugPro - Mix :
Revised: For AugPro, we use the nearest-neighbors to get AugPro-Mix tokens x AugPro - Mix :

Original: [Equation]
Revised: [Equation]

Original: where w 
Revised: where w AugPro

Original: [Equation]
Revised: [Equation]

Revised: Mix

Revised: [Equation]

Original: Sim means the similarity function, which can be the cosine similarity.
Revised: Sim means the similarity function, which can be the cosine similarity.

Original: e ( i ) means the i th embedding vector in e .
Revised: e ( i ) means the i th embedding vector in e .

Original: Concrete examples can be found in Appendix C. Then the loss function L Aug becomes:
Revised: Concrete examples can be found in Appendix D. Then the loss function L Aug becomes:

Original: [Equation]
Revised: [Equation]

Original: We do not use y MixUp because the projection operation (the nearest-neighbors in AugPro-Mix) does not necessarily preserve the label.
Revised: We do not use y MixUp because the projection operation (the nearest-neighbors in AugPro-Mix) does not necessarily preserve the label.

Original: AugPro-FGSM
Revised: AugPro-FGSM

Original: Though adversarial examples (AE) are originally aimed to improve the robustness of models and may harm the performance on clean inputs (Raghunathan et al., 2019), Zhang et al.
Revised: Though adversarial examples (AE) are originally aimed to improve the robustness of models and may harm the performance on clean inputs (Raghunathan et al., 2019), Zhang et al.

Original: (2022b) shows that AE can benefit knowledge distillation.
Revised: (2022b) shows that AE can benefit knowledge distillation.

Original: We get AugPro-FGSM by applying
Revised: We get AugPro-FGSM by applying

Original: AugPro to FGSM.
Revised: AugPro to FGSM.

Original: We first apply FGSM to the student model and get augmented data e g FGSM .
Revised: We first apply FGSM to the student model and get augmented data e g FGSM .

Original: The augmented data can be used to construct L Aug directly:
Revised: The augmented data can be used to construct L Aug directly:

Original: [Equation]
Revised: [Equation]

Original: For AugPro, we use the nearest-neighbors to get x AugPro - FGSM : [Equation] where w AugPro - FGSM ,i = max w ∈ V Sim ( e g FGSM ( i [Equation]
Revised: Following Equation 5, we could get x AugPro - FGSM by changing the footnotes accordingly.

Original: f

Original: [Equation] w [Equation] We usually set ϵ in Equation (2) large in AugPro-FGSM since we do not want x AugPro - FGSM to be the same as the original input, whereas ϵ in Equation (2) is a small value in FGSM.
Revised: We usually set ϵ in Equation (2) large in AugPro-FGSM since we do not want x AugPro - FGSM to be the same as the original input, whereas ϵ in Equation (2) is a small value in FGSM.

Original: We use the cosine similarity to implement the Sim function.
Revised: We use the cosine similarity to implement the Sim function.

Original: Concrete examples can be found in Appendix C.

Original: The loss function L Aug becomes:
Revised: The loss function L Aug becomes:

Original: [Equation]
Revised: [Equation]

Original: Label Diversity We take two sentences from SST-2 dataset (Socher et al., 2013) as an example to further explain that projection does not necessarily preserve labels but generates diverse labels.
Revised: Label Diversity We take two sentences from SST-2 dataset (Socher et al., 2013) as an example to further explain that projection does not necessarily preserve labels but generates diverse labels.

Original: The first sentence is watch on video at home with the sentiment Neutral .
Revised: The first sentence is watch on video at home with the sentiment Neutral .

Original: The second sentence is as good with the sentiment Positive .
Revised: The second sentence is as good with the sentiment Positive .

Original: Then we can get the AugPro-Mix sentence watch good video at home .
Revised: Then we can get the AugPro-Mix sentence watch good video at home .

Original: Obviously the label of AugPro-Mix sentence should be Positive rather than the linear interpolation of Positive and Neutral .
Revised: Obviously the label of AugPro-Mix sentence should be Positive rather than the linear interpolation of Positive and Neutral .

Original: This is the desired property in distillation as we will use the teacher to label these newly generated data points.
Revised: This is the desired property in distillation as we will use the teacher to label these newly generated data points.

Original: Computational Overhead .
Revised: Computational Overhead .

Original: If we assume the complexity of computing cosine similarity between two vectors is O ( d ) where d is the dimension of vectors, then the complexity of the projection (the nearest-neighbors in our implementation) is O ( NV d ) , where N is the sentence length and V is the vocabulary size.
Revised: If we assume the complexity of computing cosine similarity between two vectors is O ( d ) where d is the dimension of vectors, then the complexity of the projection (the nearest-neighbors in our implementation) is O ( NV d ) , where N is the sentence length and V is the vocabulary size.

Original: N is usually within hundreds.
Revised: N is usually within hundreds.

Original: V is usually around 30 , 000 in popular pre-train language models using sub-word tokens such as BERT (Devlin et al., 2018) and T5 (Raffel et al., 2020).
Revised: V is usually around 30 , 000 in popular pre-train language models using sub-word tokens such as BERT (Devlin et al., 2018) and T5 (Raffel et al., 2020).

Original: As a result, O ( NV d ) brings little costs.
Revised: As a result, O ( NV d ) brings little costs.

Original: On the other hand, the projection operation could be parallelized since the NV similarity calculations do not affect each other.
Revised: On the other hand, the projection operation could be parallelized since the NV similarity calculations do not affect each other.

Original: In modern parallel computing architectures, such as GPUs and TPUs, projection can be calculated in a much faster manner.
Revised: In modern parallel computing architectures, such as GPUs and TPUs, projection can be calculated in a much faster manner.

Original: Compared to the major large-scale language models’ complexities, this computation will take a small portion of resources.
Revised: Compared to the major large-scale language models’ complexities, this computation will take a small portion of resources.

Original: The detailed running time comparison can be found in Section 5.2.
Revised: The detailed running time comparison can be found in Section 5.2.

Original: Three Properties of AugPro.
Revised: Three Properties of AugPro.

Original: Since AugPro supports operations used in representation interpolation methods, AugPro is expressive (property (1)).
Revised: Since AugPro supports operations used in representation interpolation methods, AugPro is expressive (property (1)).

Original: AugPro also converts representations to tokens to avoid shifting decision boundaries, leading to a smaller error rate (property (2)).
Revised: AugPro also converts representations to tokens to avoid shifting decision boundaries, leading to a smaller error rate (property (2)).

Original: It can be shown that AugPro-Mix has a 14 N lower error rate than MixUp, and AugPro-FGSM has a 12 N lower error rate than FGSM with certain assumptions (Appendix G).
Revised: It can be shown that AugPro-Mix has a 14 N lower error rate than MixUp, and AugPro-FGSM has a 12 N lower error rate than FGSM with certain assumptions (Appendix H).

Original: Moreover, AugPro has a low computational overhead to guarantee efficiency (property (3)), as described in the previous paragraph.
Revised: Moreover, AugPro has a low computational overhead to guarantee efficiency (property (3)), as described in the previous paragraph.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments aim to answer two questions: (1) How effective is AugPro when applied to the knowledge distillation scenario?
Revised: Our experiments aim to answer two questions: (1) How effective is AugPro when applied to the knowledge distillation scenario?

Original: (2) Is AugPro efficient?
Revised: (2) Is AugPro efficient?

Original: Datasets and Settings Following previous knowledge distillation works (Liang et al., 2020; Zhang et al., 2022b), we use GLUE (Wang et al., 2018) datasets as the benchmark.
Revised: Datasets and Settings Following previous knowledge distillation works (Liang et al., 2020; Zhang et al., 2022b), we use GLUE (Wang et al., 2018) datasets as the benchmark.

Original: We use EncT5 (Liu et al., 2021) as our teacher and student models for the following reaons: (1) T5 has a much better performance than BERT and is close to SOTA in many tasks.
Revised: We use EncT5 (Liu et al., 2021) as our teacher and student models for the following reaons: (1) T5 has a much better performance than BERT and is close to SOTA in many tasks.

Original: EncT5 is a simplified version of T that uses the whole encoders of T5 but only one decoder layer.
Revised: EncT5 is a simplified version of T that uses the whole encoders of T5 but only one decoder layer.

Original: EncT5 performs similarly to T5 on classification tasks such as GLUE tasks with fewer parameters.
Revised: EncT5 performs similarly to T5 on classification tasks such as GLUE tasks with fewer parameters.

Original: For example, EncT5 (small) only contains 37M parameters but can perform similarly to T5 (small), which contains 77M parameters.
Revised: For example, EncT5 (small) only contains 37M parameters but can perform similarly to T5 (small), which contains 77M parameters.

Original: Using EncT5 will make the results more convincing and show that our method is still useful even with powerful models.
Revised: Using EncT5 will make the results more convincing and show that our method is still useful even with powerful models.

Original: (2) Previous methods (Liang et al., 2020;
Revised: (2) Previous methods (Liang et al., 2020;

Original: Zhang et al., 2022b) distill knowledge from a 12-layer BERT to a 6-layer BERT or a 3-layer BERT.
Revised: Zhang et al., 2022b) distill knowledge from a 12-layer BERT to a 6-layer BERT or a 3-layer BERT.

Original: However, the gap between the teacher and student models is marginal.
Revised: However, the gap between the teacher and student models is marginal.

Original: Therefore, the improvement space is limited, and the existence of variance will weaken the credibility of the results.
Revised: Therefore, the improvement space is limited, and the existence of variance will weaken the credibility of the results.

Original: To this end, we distill knowledge from EncT (Large, 24-layer, 354M, teacher) to EncT5 (small, 8-layer, 37M, student), as the two models have a significant performance gap.
Revised: To this end, we distill knowledge from EncT (Large, 24-layer, 354M, teacher) to EncT5 (small, 8-layer, 37M, student), as the two models have a significant performance gap.

Original: Baselines and Training We train several baselines for comparison: (1) Fine-Tuning (FT) :
Revised: Baselines and Training We train several baselines for comparison: (1) Fine-Tuning (FT) :

Original: We directly fine-tune EncT5 on the dataset.
Revised: We directly fine-tune EncT5 on the dataset.

Original: (2) Knowledge Distillation (KD) :
Revised: (2) Knowledge Distillation (KD) :

Original: We first fine-tune a teacher model (EncT5 Large), then distill knowledge from the teacher model to the student model (EncT5 Small).
Revised: We first fine-tune a teacher model (EncT5 Large), then distill knowledge from the teacher model to the student model (EncT5 Small).

Original: (3) Knowledge Distillation + Back Translation (KD+BT) :
Revised: (3) Knowledge Distillation + Back Translation (KD+BT) :

Original: Back translation (Yu et al., 2018) first translates input to another language and then translates it back.
Revised: Back translation (Yu et al., 2018) first translates input to another language and then translates it back.

Original: We choose back translation as a representative method for the data augmentation type “augmentation with models”.
Revised: We choose back translation as a representative method for the data augmentation type “augmentation with models”.

Original: (4) Knowledge Distillation + K-Nearest-Neighbors (KD+KNN) KNN (Wang & Yang, 2015) first selects tokens from inputs, then replaces them with the K nearest neighbors in the embedding space.
Revised: (4) Knowledge Distillation + K-Nearest-Neighbors (KD+KNN) KNN (Wang & Yang, 2015) first selects tokens from inputs, then replaces them with the K nearest neighbors in the embedding space.

Original: KNN can be regarded as one token replacement method.
Revised: KNN can be regarded as one token replacement method.

Original: (5) KD+MixUp (6) KD+FGSM (7) KD+
Revised: (5) KD+MixUp (6) KD+FGSM (7) KD+

Original: TMix (Chen et al., 2020b) MixUp on the hidden state between transformer layers.
Revised: TMix (Chen et al., 2020b) MixUp on the hidden state between transformer layers.

Original: The last three methods are of the “representation interpolation” type.
Revised: The last three methods are of the “representation interpolation” type.

Original: We train student models with 0.6M steps and 512 batch size.
Revised: We train student models with 0.6M steps and 512 batch size.

Original: Due to the high computation cost, we only augment data to twice as large as the original dataset size for back translation.
Revised: Due to the high computation cost, we only augment data to twice as large as the original dataset size for back translation.

Original: For all other methods, we augment data to twice as large as the original batch size for each batch, i.e., we augment 0 .
Revised: For all other methods, we augment data to twice as large as the original batch size for each batch, i.e., we augment 0 .

Original: 6M  · 512  = 307 .
Revised: 6M steps · 512 batch size = 307 .

Original: 2M data in total.
Revised: 2M data in total.

Original: More training details are in Appendix B.
Revised: More training details are in Appendix C.

Original: 5. E FFECTIVENESS OF A UG P RO
Revised: 5. E FFECTIVENESS OF A UG P RO

Original: Table 1 shows the results of knowledge distillation.
Revised: Table 1 shows the results of knowledge distillation.

Original: Due to the high cost, we only report back translation results on the RTE dataset.
Revised: Due to the high cost, we only report back translation results on the RTE dataset.

Original: We first use the training data to train a teacher model and then distill knowledge from the teacher model to the student model on the training data.
Revised: We first use the training data to train a teacher model and then distill knowledge from the teacher model to the student model on the training data.

Original: We can conclude that: (1) All data augmentation methods will benefit the distillation.
Revised: We can conclude that: (1) All data augmentation methods will benefit the distillation.

Original: (2) AugPro can significantly improve the distillation performance compared with corresponding baselines.
Revised: (2) AugPro can significantly improve the distillation performance compared with corresponding baselines.

Original: Specifically, AugPro is extremely useful for low-resource datasets such as CoLA and RTE.
Revised: Specifically, AugPro is extremely useful for low-resource datasets such as CoLA and RTE.

Original: AugPro-Mix achieves scores 5.97% and 9.02% higher than MixUp on CoLA and RTE, respectively.
Revised: AugPro-Mix achieves scores 5.97% and 9.02% higher than MixUp on CoLA and RTE, respectively.

Original: AugPro-FGSM achieves scores 10.52% and 8.31% higher than FGSM on CoLA and RTE, respectively.
Revised: AugPro-FGSM achieves scores 10.52% and 8.31% higher than FGSM on CoLA and RTE, respectively.

Original: For large datasets such as MNLI, AugPro-Mix and AugPro-FGSM can also improve the performance.
Revised: For large datasets such as MNLI, AugPro-Mix and AugPro-FGSM can also improve the performance.

Original: (3) Moreover, combining AugPro-FGSM and AugPro-Mix achieves the best performance in all listed methods.
Revised: (3) Moreover, combining AugPro-FGSM and AugPro-Mix achieves the best performance in all listed methods.

Original: Compared with vanilla knowledge distillation, combining AugPro-Mix and AugPro-FGSM improves the performance from 2% to 14%.
Revised: Compared with vanilla knowledge distillation, combining AugPro-Mix and AugPro-FGSM improves the performance from 2% to 14%.

Original: Table 2 uses a different setting from Table 1.
Revised: Table 2 uses a different setting from Table 1.

Original: We only keep 10% training data labeled and assume others are unlabeled.
Revised: We only keep 10% training data labeled and assume others are unlabeled.

Original: Then we use labeled training data to train a teacher model and unlabeled training data to do knowledge distillation—this is a more realistic setting since it is often easier to get unlabeled data than to get labeled data.
Revised: Then we use labeled training data to train a teacher model and unlabeled training data to do knowledge distillation—this is a more realistic setting since it is often easier to get unlabeled data than to get labeled data.

Original: The conclusions above still hold.
Revised: The conclusions above still hold.

Original: Specifically, AugPro can improve the accuracy from 1% to 2% on average on three datasets.
Revised: Specifically, AugPro can improve the accuracy from 1% to 2% on average on three datasets.

Original: Compared with the vanilla distillation, AugPro can improve around 2% accuracy at most on three datasets.
Revised: Compared with the vanilla distillation, AugPro can improve around 2% accuracy at most on three datasets.

Original: 5. E FFICIENCY OF A UG P RO
Revised: 5. E FFICIENCY OF A UG P RO

Original: The efficiency of AugPro lies in two aspects.
Revised: The efficiency of AugPro lies in two aspects.

Original: First, its complexity is low.
Revised: First, its complexity is low.

Original: Second, it can be computed in parallel.
Revised: Second, it can be computed in parallel.

Original: To fully demonstrate these two advantages, we report the real-time cost of AugPro and baselines in Table 3.
Revised: To fully demonstrate these two advantages, we report the real-time cost of AugPro and baselines in Table 3.

Original: KD+data augmentation is rough twice the time of vanilla KD since these methods use twice the data as vanilla KD.
Revised: KD+data augmentation is rough twice the time of vanilla KD since these methods use twice the data as vanilla KD.

Original: We can also observe that augmentation with models (KD+BT) takes much more time than other kinds of baselines, which shows that this method is not efficient enough.
Revised: We can also observe that augmentation with models (KD+BT) takes much more time than other kinds of baselines, which shows that this method is not efficient enough.

Original: At last, AugPro brings little computational overhead as the time cost is the same as
Revised: At last, AugPro brings little computational overhead as the time cost is the same as

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: trained on labeled training data and then used for knowledge distillation on unlabeled training data.
Revised: on labeled training data and then used for knowledge distillation on unlabeled training data.

Original: the baselines.
Revised: the baselines.

Original: Results also show that KNN is much slower than other methods, which is explained in
Revised: Results also show that KNN is much slower than other methods, which is explained in

Original: Appendix F.
Revised: Appendix G.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: In ablation studies, we follow settings used in Table 1 unless otherwise stated.
Revised: In ablation studies, we follow settings used in Table 1 unless otherwise stated.

Original: Perturbation scale for ϵ in AugPro-FGSM
Revised: Perturbation scale for ϵ in AugPro-FGSM

Original: The key hyperparameter in AugPro-FGSM is ϵ in
Revised: The key hyperparameter in AugPro-FGSM is ϵ in

Original: Equation (2).
Revised: Equation (2).

Original: Small ϵ will make x AugPro-FGSM the same as the original input.
Revised: Small ϵ will make x AugPro-FGSM the same as the original input.

Original: Large ϵ tends to make x AugPro-FGSM hard to understand, meaningless, and out of the domain.
Revised: Large ϵ tends to make x AugPro-FGSM hard to understand, meaningless, and out of the domain.

Original: Therefore, a proper ϵ is essential.
Revised: Therefore, a proper ϵ is essential.

Original: Our experiments find that ϵ = 35 is the best fit for T5 embeddings.
Revised: Our experiments find that ϵ = 35 is the best fit for T5 embeddings.

Original: Table 4 shows
Revised: Table 4 shows

Original: KD+AugPro-FGSM performance with different ϵ .
Revised: KD+AugPro-FGSM performance with different ϵ .

Original: Signs of gradients in AugPro-FGSM are not important The effectiveness of AugPro-FGSM comes from gradients’ signs and the projection in AugPro.
Revised: Signs of gradients in AugPro-FGSM are not important The effectiveness of AugPro-FGSM comes from gradients’ signs and the projection in AugPro.

Original: To prove that AugPro-FGSM mainly benefits from AugPro, we implement two AugPro-FGSM variants: AugPro-FGSMD (Descent Projection) that uses the opposite signs to AugPro-FGSM , and AugPro-FGSMR (Random Projection) that uses random signs.
Revised: To prove that AugPro-FGSM mainly benefits from AugPro, we implement two AugPro-FGSM variants: AugPro-FGSMD (Descent Projection) that uses the opposite signs to AugPro-FGSM , and AugPro-FGSMR (Random Projection) that uses random signs.

Original: Table 6 shows the results of AugPro-FGSM and its two variants.
Revised: Table 6 shows the results of AugPro-FGSM and its two variants.

Original: We can observe
Revised: We can observe

Original: AugPro-FGSM has a similar score to its variants in all settings.
Revised: AugPro-FGSM has a similar score to its variants in all settings.

Original: Thus AugPro-FGSM mainly benefits from AugPro.
Revised: Thus AugPro-FGSM mainly benefits from AugPro.

Original: We also conduct experiments that follow the setting of Table 2, and results can be found in Appendix D.
Revised: We also conduct experiments that follow the setting of Table 2, and results can be found in Appendix E.

Original: AugPro generates diverse labels We show that AugPro generates diverse labels at the end of Section 4.
Revised: AugPro generates diverse labels We show that AugPro generates diverse labels at the end of Section 4.

Original: Here we empirically show that assuming AugPro preserving labels may harm performance.
Revised: Here we empirically show that assuming AugPro preserving labels may harm performance.

Original: If
Revised: If

Original: AugPro preserves labels, AugPro-Mix and AugPro-FGSM data should have the same labels as
Revised: AugPro preserves labels, AugPro-Mix and AugPro-FGSM data should have the same labels as

Original: MixUp and original data, respectively.
Revised: MixUp and original data, respectively.

Original: We use these augmented data together with labels to fine-tune student models directly.
Revised: We use these augmented data together with labels to fine-tune student models directly.

Original: Results in Table 5 suggest that such augmented data and labels may harm performance.
Revised: Results in Table 5 suggest that such augmented data and labels may harm performance.

Original: Therefore, AugPro generates diverse labels and does not necessarily preserve labels.
Revised: Therefore, AugPro generates diverse labels and does not necessarily preserve labels.

Original: AugPro consistently benefits KD with different data sizes Figure 3 shows the performance of
Revised: AugPro consistently benefits KD with different data sizes Figure 3 shows the performance of

Original: AugPro with different data sizes.
Revised: AugPro with different data sizes.

Original: It can be observed that AugPro is better than all baselines in all data sizes.
Revised: It can be observed that AugPro is better than all baselines in all data sizes.

Original: Moreover, AugPro is extremely useful when the data size is small.
Revised: Moreover, AugPro is extremely useful when the data size is small.

Original: For example, AugPro can
Revised: For example, AugPro can

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: accuracy of 4% (SST-2) and 6% (MNLI-M) when the data size is 10%.
Revised: of 4% (SST-2) and 6% (MNLI-M) when the data size is 10%.

Original: We also report results on the MNLI-MM dataset in Appendix E.
Revised: We also report results on the MNLI-MM dataset in Appendix F.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We propose AugPro, an effective and efficient data augmentation paradigm for knowledge distillation.
Revised: We propose AugPro, an effective and efficient data augmentation paradigm for knowledge distillation.

Original: We use projections to tackle the problem of shifting decision boundaries caused by traditional representation interpolation methods in knowledge distillation.
Revised: We use projections to tackle the problem of shifting decision boundaries caused by traditional representation interpolation methods in knowledge distillation.

Original: Moreover, AugPro has low computation costs and is fast in modern computing architectures.
Revised: Moreover, AugPro has low computation costs and is fast in modern computing architectures.

Original: Results on GLUE tasks prove the effectiveness and efficiency of AugPro.
Revised: Results on GLUE tasks prove the effectiveness and efficiency of AugPro.

Original: In the future, we will further explore the impact of AugPro on labels to make it helpful in other scenarios.
Revised: In the future, we will further explore the impact of AugPro on labels to make it helpful in other scenarios.

Original: A C ONCRETE NUMBERS OF THE EXAMPLE IN S ECTION
Revised: B C ONCRETE NUMBERS OF THE EXAMPLE IN S ECTION

Original: X = { x 1 = (2 . 5 , 2) , x 2 = (2 , − 2) , x 3 = ( − 2 . 5 , − 2) , x 4 = ( − 2 , 2) }
Revised: X = { x 1 = (2 . 5 , 2) , x 2 = (2 , − 2) , x 3 = ( − 2 . 5 , − 2) , x 4 = ( − 2 , 2) }

Original: Y = { y 1 = +1 , y 2 = +1 , y 3 = − 1 , y 4 = − 1 }
Revised: Y = { y 1 = +1 , y 2 = +1 , y 3 = − 1 , y 4 = − 1 }

Original: • β ∗ = 
Revised: • β ∗ = (

Original: (4 / 9 , − 1 / 18) , b ∗ = 0 .
Revised: 9 , − 1 / 18) , b ∗ = 0 .

Original: • x MixUp = 1325 x 3 + 1225 x 1 , β MixUp =
Revised: • x MixUp = 1325 x 3 + 1225 x 1 , β MixUp =

Original: (250 / 533 , 200 / 533) , b MixUp = − 12 /• β MixUp − P
Revised: (250 / 533 , 200 / 533) , b MixUp = − 12 /• β MixUp − P

Original: =
Revised: =

Original: (4 / 9 , 0) , b MixUp − P = 1 /
Revised: (4 / 9 , 0) , b MixUp − P = 1 /

Original: B I MPLEMENTATION D ETAILS B. H YPERPARAMETERS
Revised: C I MPLEMENTATION D ETAILS C. H YPERPARAMETERS

Original: We use JAX and T5X to implement EncT5 and AugPro, and use T5 1.1 checkpoints to initialize models.
Revised: We use JAX and T5X to implement EncT5 and AugPro, and use T5 1.1 checkpoints to initialize models.

Original: The batch size is 512, and the maximum sentence length is 128.
Revised: The batch size is 512, and the maximum sentence length is 128.

Original: Training steps are 0.6M for all experiments.
Revised: Training steps are 0.6M for all experiments.

Original: We use 8 TPU v3 slices to do all experiments.
Revised: We use 8 TPU v3 slices to do all experiments.

Original: To fine-tune the teacher model, we use a dropout rate of 0.1 and a learning rate of 1e-3 for all GLUE tasks.
Revised: To fine-tune the teacher model, we use a dropout rate of 0.1 and a learning rate of 1e-3 for all GLUE tasks.

Original: For knowledge distillation, we set the dropout rate to be 0.1 for both the teacher and student models.
Revised: For knowledge distillation, we set the dropout rate to be 0.1 for both the teacher and student models.

Original: We find that adding dropout to teachers will make the distillation better.
Revised: We find that adding dropout to teachers will make the distillation better.

Original: We run all experiments with 1e-3 and 1e-5 learning rates and report the best results.
Revised: We run all experiments with 1e-3 and 1e-5 learning rates and report the best results.

Original: As a result, learning rate is set to 1e-5 for all experiments on  STSB dataset, EncT5 8 -FT and EncT5 8 -KD experiments on CoLA, MRPC and RTE datasets.
Revised: As a result, learning rate is set to 1e-5 for all experiments on the STSB dataset, EncT5 8 -FT and EncT5 8 -KD experiments on CoLA, MRPC and RTE datasets.

Original: All other experiments use the learning rate 1e-3.
Revised: All other experiments use the learning rate 1e-3.

Original: The λ (Equation (1)) for AugPro-Mix is 0.5.
Revised: The λ (Equation (1)) for AugPro-Mix is 0.5.

Original: Previous works (Zhang et al., 2017; Liang et al., 2020) use a beta distribution to sample λ for MixUp.
Revised: Previous works (Zhang et al., 2017; Liang et al., 2020) use a beta distribution to sample λ for MixUp.

Original: We try λ ∼ Beta (0 . 4 , 0 . 4) and λ = 0 .
Revised: We try λ ∼ Beta (0 . 4 , 0 . 4) and λ = 0 .

Original: 5 for MixUp and find they have similar performance in most tasks.
Revised: 5 for MixUp and find they have similar performance in most tasks.

Original: In some tasks such as CoLA, λ = 0 .
Revised: In some tasks such as CoLA, λ = 0 .

Original: 5 is better.
Revised: 5 is better.

Original: Therefore, we use λ = 0 .
Revised: Therefore, we use λ = 0 .

Original: 5 for MixUp.
Revised: 5 for MixUp.

Original: Following previous works (Zhou et al., 2021), we set k = 15 for the KNN baseline, and randomly select 0 .
Revised: Following previous works (Zhou et al., 2021), we set k = 15 for the KNN baseline and randomly select 0 .

Original: 1 portion of tokens to replace.
Revised: 1 portion of tokens to replace.

Original: We use outputs of the 4th layer of the student model and the 12th layer of the teacher model, i.e. the middle layer of both models, to conduct TMix experiments.
Revised: We use outputs of the 4th layer of the student model and the 12th layer of the teacher model, i.e., the middle layer of both models, to conduct TMix experiments.

Original: B. I MPLEMENTATION DETAIL OF E QUATION5 AND
Revised: C. I MPLEMENTATION DETAIL OF E QUATION

Original: Algorithm 1 shows the AugPro pipeline.
Revised: Algorithm 1 shows the AugPro pipeline.

Original: Here, we show a detailed implementation of Equationsand 7 in Algorithm 2.
Revised: Here, we show a detailed implementation of Equation 5 in Algorithm 2.

Original: We belive these two algorithms can help readers reproduce our methods and results.
Revised: We believe these two algorithms can help readers reproduce our methods and results.

Original: Algorithm 
Revised: Algorithm 2: Projection Algorithm

Original: [Table]
Revised: Input: Augmented representations B rep ∈ R B × L × H , where B is the batch size, L is the sentence length, and H is the embedding dimension. Vocabulary V contains Z tokens. embeddings E R Z × H .

Revised: Output: Projected augmented data B

Original:  • Compute the similarity between each token pair Sims =
Revised: ∈ V B × L • Compute the similarity between each token pair Sims =

Original: B rep · E T ∈ R B × L [Table] Z • Find the nearest neighbor B ′ index =
Revised: B rep · E T ∈ R B × L × Z • Find the nearest neighbor B ′ index =

Original: arg min axis =2 Sims ∈ N B × L • Find tokens according to indices to get B ′ • return B ′
Revised: arg min axis =2 Sims ∈ N B × L • Find tokens according to indices to get B ′ • return B ′

Original: C E XAMPLES OF A UG P RO -M IX AND A UG P RO -FGSM
Revised: D E XAMPLES OF A UG P RO -M IX AND A UG P RO -FGSM

Original: We use three MNLI inputs to show the augmented data by AugPro-Mix and AugPro-FGSM.
Revised: We use three MNLI inputs to show the augmented data by AugPro-Mix and AugPro-FGSM.

Original: Following are three MNLI inputs:
Revised: Following are three MNLI inputs:

Original: (i) hypothesis : The judgments need to consider the broader public interest.
Revised: (i) hypothesis : The judgments need to consider the broader public interest.

Original: premise : These judgments need to be made in a consistent manner with consideration of the broader public interest in the program or activity under review.
Revised: premise : These judgments need to be made in a consistent manner with consideration of the broader public interest in the program or activity under review.

Original: (ii) hypothesis : I agree.
Revised: (ii) hypothesis : I agree.

Original: premise :
Revised: premise :

Original: yeah
Revised: yeah

Original: i i agree with that (iii) hypothesis : She’s never been to a hospital.
Revised: i i agree with that (iii) hypothesis : She’s never been to a hospital.

Original: premise : and uh as if that wasn’t bad enough the the ones that were half alive that they rushed to the hospital and she got to work on
Revised: premise : and uh as if that wasn’t bad enough the the ones that were half alive that they rushed to the hospital and she got to work on

Original: and she got to see them die
Revised: and she got to see them die

Original: and uh and they just get all the world’s worst situations very few rewarding situations uh and
Revised: and uh and they just get all the world’s worst situations very few rewarding situations uh and

Original: If we use AugPro-Mix on (i) and (ii), (ii) and (iii) we will get: (i + ii) hypothesis : I judgment.
Revised: If we use AugPro-Mix on (i) and (ii), (ii) and (iii), we will get: i + ii) hypothesis : I judgment.

Original: premise:
Revised: premise

Original: yeah judgmenti needi agree made in (ii + iii) hypothesis :She agree.
Revised: yeah judgmenti needi agree made in ( ii + iii) hypothesis :

Revised: She agree.

Original: never been to a hospital.
Revised: never been to a hospital.

Original: premise :
Revised: premise :

Original: yeah uh as agreeif that wasn’t bad enough the the ones that were half alive that they rushed to the hospital
Revised: yeah uh as agreeif that wasn’t bad enough the the ones that were half alive that they rushed to the hospital

Original: and she got to work on
Revised: and she got to work on

Original: and she got to see them die
Revised: and she got to see them die

Original: and uh and they just get all the world’s worst situations very few rewarding situations uh an
Revised: and uh and they just get all the world’s worst situations very few rewarding situations uh an

Original: If we use AugPro-FGSM on (i), (ii) and (iii) we will get:
Revised: If we use AugPro-FGSM on (i), (ii) and (iii), we will get:

Original: (i) hypothesis : River judgment Mit handy to consider the broader public interest.
Revised: (i) hypothesis : River judgment Mit handy to consider the broader public interest.

Original: premise : These judgment Mit handy to be made in a consistent manner with consideration- the broader public interest in the federal cl˘adire activity under reviewed.
Revised: premise : These judgment Mit handy to be made in a consistent manner with consideration- the broader public interest in the federal cl˘adire activity under reviewed.

Original: (ii) hypothesis : Kyle agree.
Revised: (ii) hypothesis : Kyle agree.

Original: premise :
Revised: premise :

Original: yeah chambres chambres agree with that (iii) hypothesis : She’ Mit never been to a hospital.
Revised: yeah chambres chambres agree with that [Table] [Table] She’ Mit never been to a hospital.

Original: premise : and uh as if that wasn’ financing bad enough the the ones that are half alive that they 
Revised: premise : and uh as if that wasn’ financing bad enough the the ones that are half alive that they rushed to the hospital

Original: rushed to

Original: the hospital and she got to work 
Revised: and she got to work on

Original: on and she got to see them die
Revised: and she got to see them die

Original: and uh

Original: and  they just get all the world’ Mit worst situations very few rewarding situations 
Revised: and uh and they just get all the world’ Mit worst situations very few rewarding situations uh and

Original: uh and

Original: The above-augmented sentences are originally token lists, but not real sentences.
Revised: The above-augmented sentences are originally token lists, but not real sentences.

Original: Luckily, T5 uses SentencePiece to construct its vocabulary, which supports the precise de-tokenization for any token list.
Revised: Luckily, T5 uses SentencePiece to construct its vocabulary, which supports the precise de-tokenization for any token list.

Original: To convert token lists to sentences, we use a simple command "".join(tokens).replace
Revised: To convert token lists to sentences, we use a simple command "".join(tokens).replace

Original: Linguistic Analysis.
Revised: Linguistic Analysis.

Original: Our motivation focuses on the perspective of machine learning, i.e., avoiding shifting decision boundaries by converting representations to tokens.
Revised: Our motivation focuses on the perspective of machine learning, i.e., avoiding shifting decision boundaries by converting representations to tokens.

Original: Here, we would like to add a brief analysis from a linguistic perspective.
Revised: Here, we would like to add a brief analysis from a linguistic perspective.

Original: We can observe that the above-augmented sentences may have grammatical errors, ”meaningless” tokens, and may be less meaningful than original sentences.
Revised: We can observe that the above-augmented sentences may have grammatical errors, ”meaningless” tokens, and may be less meaningful than original sentences.

Original: However, ”meaningless” to humans does not suggest meaningless to models, as AugPro indeed boosts the distillation performance.
Revised: However, ”meaningless” to humans does not suggest meaningless to models, as AugPro indeed boosts the distillation performance.

Original: Besides, augmented sentences are not totally semantically meaningless to humans.
Revised: Besides, augmented sentences are not totally semantically meaningless to humans.

Original: It is hard to see why augmented data is so helpful from the linguistic perspective, suggesting that we should focus on analyzing these data from the machine learning perspective, which is exactly our motivation.
Revised: It is hard to see why augmented data is so helpful from the linguistic perspective, suggesting that we should focus on analyzing these data from the machine learning perspective, which is exactly our motivation.

Original: To further support our motivation, we conduct a simple baseline ”random generation” that randomly chooses tokens from the vocabulary and concatenates[Table] them to form an augmented sentence.
Revised: To further support our motivation, we conduct a simple baseline ”random generation” that randomly chooses tokens from the vocabulary and concatenates them to form an augmented sentence.

Original: Random generation can generate meaningless sentences easily.
Revised: Random generation can generate meaningless sentences easily.

Original: Results are shown in Table 7.
Revised: Results are shown in Table 7.

Original: We can conclude that random generation is a poor augmentation method, suggesting that AugPro is helpful not because of the meaningful or meaningless semantics, but because of avoiding shifting decision boundaries
Revised: We can conclude that random generation is a poor augmentation method, suggesting that AugPro is helpful not because of the meaningful or meaningless semantics, but because of avoiding shifting decision boundaries.

Original: [Table]

Original: D S IGNS OF GRADIENTS IN A UG P RO -FGSM IS NOT IMPORTANT (M ORE RESULTS )
Revised: E S IGNS OF GRADIENTS IN A UG P RO -FGSM IS NOT IMPORTANT (M ORE RESULTS )

Original: Table 8 shows the effect of signs of gradients in AugPro-FGSM with the setting of Table 2.
Revised: Table 8 shows the effect of signs of gradients in AugPro-FGSM with the setting of Table 2.

Original: The conclusion is same as the conslusion concluded from Table 6.
Revised: The conclusion is same as the conslusion concluded from Table 6.

Original: [Table]

Original: This table follows the setting in Table 2.

Original: E A UG P RO CONSISTENTLY BENEFITS KD WITH DIFFERENT DATA SIZES (M ORE RESULTS )
Revised: F A UG P RO CONSISTENTLY BENEFITS KD WITH DIFFERENT DATA SIZES (M ORE RESULTS )

Original: The main texts show that AugPro consistently benefits KD with different data sizes on SST-2 and
Revised: The main texts show that AugPro consistently benefits KD with different data sizes on SST-2 and

Original: MNLI-M datasets.
Revised: MNLI-M datasets.

Original: Figure 4 shows this conclusion still holds on the MNLI-MM dataset.
Revised: Figure 5 shows this conclusion still holds on the MNLI-MM dataset.

Original: The overall trend is similar to the MNLI-M dataset.
Revised: The overall trend is similar to the MNLI-M dataset.

Original: F KNN IS SLOWER THAN OTHER METHODS
Revised: G KNN IS SLOWER THAN OTHER METHODS

Original: There are two reasons that KNN is slower than other methods.
Revised: There are two reasons that KNN is slower than other methods.

Original: First, KNN has a higher computational complexity
Revised: First, KNN has a higher computational complexity

Original: O
Revised: O

Original: ( NV dlogk )
Revised: ( NV dlogk )

Original: than AugPro ( O ( NV d ) ).
Revised: than AugPro ( O ( NV d ) ).

Original: Second, KNN is hard to be implemented by
Revised: Second, KNN is hard to be implemented by

Original: XLA.
Revised: XLA.

Original: A popular and fast implementation of KNN is to use np.argpartition .
Revised: A popular and fast implementation of KNN is to use np.argpartition .

Original: However, XLA  does not support partition operations 2 , making KNN hard to be implemented with JAX on TPUs.
Revised: However, XLA [Table] [Figure] does not support partition operations 2 , making KNN hard to be implemented with JAX on TPUs.

Original: To this end, Chern et al.
Revised: To this end, Chern et al.

Original: (2022) propose another method to run KNN on TPUs at peak
Revised: (2022) propose another method to run KNN on TPUs at peak

Original: FLOP/s.
Revised: FLOP/s.

Original: We use their method to implement KNN, but such implementation may not be the optimal solution for
Revised: We use their method to implement KNN, but such implementation may not be the optimal solution for

Original: KNN.
Revised: KNN.

Original: [Figure]

Original: G A UG P RO CAN ACHIEVE LESS ERROR RATES
Revised: H A UG P RO CAN ACHIEVE LESS ERROR RATES

Original: Suppose X = {− 1 , 1 } 2log n is the universe of all data with labels Y to learn.
Revised: Suppose X = {− 1 , 1 } 2log n is the universe of all data with labels Y to learn.

Original: Define a distribution P such that ( x, y ) ∼ P if x is uniformly independently drawn from X and y is uiformly independently drawn from {− 1 , +1 } .
Revised: Define a distribution P such that ( x, y ) ∼ P if x is uniformly independently drawn from X and y is uiformly independently drawn from {− 1 , +1 } .

Original: X D is the training data compositing of n samples uniformly and independently drawn from
Revised: X D is the training data compositing of n samples uniformly and independently drawn from

Original: X .
Revised: X .

Original: Suppose n is even.
Revised: Suppose n is even.

Original: Let x ( j ) be the j -th coordinate of vector x .
Revised: Let x ( j ) be the j -th coordinate of vector x .

Original: We construct X AugPro - Mix as follows: Index the elements in X D arbitrarily, and x i denotes the i -th element.
Revised: We construct X AugPro - Mix as follows: Index the elements in X D arbitrarily, and x i denotes the i -th element.

Original: For each positive integer i ∈ [ n/ 2] , we construct an augmented data z i such that, for each coordinate j ∈ [2 log n ] , z i ( j ) =
Revised: For each positive integer i ∈ [ n/ 2] , we construct an augmented data z i such that, for each coordinate j ∈ [2 log n ] , z i ( j ) =

Original: x 2 i − 1 ( j ) if x 2 i − 1 ( j ) =
Revised: x 2 i − 1 ( j ) if x 2 i − 1 ( j ) =

Original: x 2 i ( j ) , otherwise z i ( j ) is uniformly chosen from { 1 , − 1 } .
Revised: x 2 i ( j ) , otherwise z i ( j ) is uniformly chosen from { 1 , − 1 } .

Original: Let’s consider optimal algorithms A and A AugPro - Mix .
Revised: Let’s consider optimal algorithms A and A AugPro - Mix .

Original: A can only observe X D (or X MixUp , X FGSM ) and their labels
Revised: A can only observe X D (or X MixUp , X FGSM ) and their labels

Original: Y D (or Y MixUp , Y FGSM ).
Revised: Y D (or Y MixUp , Y FGSM ).

Original: For proof, we assume A can only observe X D and their labels
Revised: For proof, we assume A can only observe X D and their labels

Original: Y D .
Revised: Y D .

Original: The other two can be proved similarly.
Revised: The other two can be proved similarly.

Original: A AugPro - Mix can observe X AugPro - Mix and their corresponding labels
Revised: A AugPro - Mix can observe X AugPro - Mix and their corresponding labels

Original: Y AugPro - Mix .
Revised: Y AugPro - Mix .

Original: Define the generalization error error( A ) :
Revised: Define the generalization error error( A ) :

Original: =
Revised: =

Original: Pr ( x,y ) ∼P , ( X D , Y D )
Revised: Pr ( x,y ) ∼P , ( X D , Y D )

Original: [ A ( x, X D , Y D )
Revised: [ A ( x, X D , Y D )

Original: ̸
Revised: ̸

Original: = y ] , where
Revised: = y ] , where

Original: A ( x, X D , Y D ) ∈
Revised: A ( x, X D , Y D ) ∈

Original: { +1 , − 1 } is the prediction of x outputted by A , and error( A AugPro - Mix ) =
Revised: { +1 , − 1 } is the prediction of x outputted by A , and error( A AugPro - Mix ) =

Original: Pr ( x,y ) ∼P , ( X AugPro - Mix , Y AugPro - Mix )
Revised: Pr ( x,y ) ∼P , ( X AugPro - Mix , Y AugPro - Mix )

Original: [ A ( x, X AugPro - Mix , Y AugPro - Mix ) ̸
Revised: [ A ( x, X AugPro - Mix , Y AugPro - Mix ) ̸

Original: = y ] .
Revised: = y ] .

Original: We have the following claim:
Revised: We have the following claim:

Original: Claim 1.
Revised: Claim 1.

Original: One has
Revised: One has

Original: [Equation]
Revised: [Equation]

Original: →∞
Revised: 1 / (4 n

Original: [Equation]
Revised: [Equation]

Revised: = 1 .

Original: As for constructing
Revised: As for constructing

Original: X AugPro - FGSM , for each x i ∈
Revised: X AugPro - FGSM , for each x i ∈

Original: X D , we add
Revised: X D , we add

Original: Gaussian
Revised: Gaussian

Original: N (0 , 4) to each coordinate of x i , and then project back to X to get w i .
Revised: N (0 , 4) to each coordinate of x i , and then project back to X to get w i .

Original: Similarly, assume A AugPro - FGSM can observe the dataset X AugPro - FGSM and its labels
Revised: Similarly, assume A AugPro - FGSM can observe the dataset X AugPro - FGSM and its labels

Original: Y AugPro - FGSM , and define the generalization error to be error( A AugPro - FGSM ) =
Revised: Y AugPro - FGSM , and define the generalization error to be error( A AugPro - FGSM ) =

Original: Pr ( x,y ) ∼P , ( X AugPro - FGSM , Y AugPro - FGSM )
Revised: Pr ( x,y ) ∼P , ( X AugPro - FGSM , Y AugPro - FGSM )

Original: [ A ( x, X AugPro - FGSM , Y AugPro - FGSM ) ̸
Revised: [ A ( x, X AugPro - FGSM , Y AugPro - FGSM ) ̸

Original: = y ] .
Revised: = y ] .

Original: Claim 2.
Revised: Claim 2.

Original: One has
Revised: One has

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF C LAIM
Revised: H. P ROOF OF C LAIM

Original: We provide the following preliminary background of martingale to prove Claim 1 for completeness.
Revised: We provide the following preliminary background of martingale to prove Claim 1 for completeness.

Original: Definition 1 (Martingale) .
Revised: Definition 1 (Martingale) .

Original: A sequence of random variables Y 1 , Y 2 , · · · is a martingale with respect to another sequence X 1 , X 2 , · · · if for all n , E [ | Y n | ] < ∞ and E [ Y n +1 | X 1 , · · · , X n ]
Revised: A sequence of random variables Y 1 , Y 2 , · · · is a martingale with respect to another sequence X 1 , X 2 , · · · if for all n , E [ | Y n | ] < ∞ and E [ Y n +1 | X 1 , · · · , X n ]

Original: = Y n .
Revised: = Y n .

Original: Definition 2 (Martingale Difference) .
Revised: Definition 2 (Martingale Difference) .

Original: { D k } ∞ k =1 is a martingale difference sequence w.r.t.
Revised: { D k } ∞ k =1 is a martingale difference sequence w.r.t.

Original: { X k } ∞ k =1 if for all n :
Revised: { X k } ∞ k =1 if for all n :

Original: • D n is a measurable function of X 1 , · · · , X n • E [ | D n | ] < ∞ • E
Revised: • D n is a measurable function of X 1 , · · · , X n • E [ | D n | ] < ∞ • E

Original: [ D n +1 |
Revised: [ D n +1 |

Original: X 1 , · · · , X n ] = 0 .
Revised: X 1 , · · · , X n ] = 0 .

Original: If Y k is a martingale, then D k
Revised: If Y k is a martingale, then D k

Original: =
Revised: =

Original: Y k − Y k − 1 is a martingale difference.
Revised: Y k − Y k − 1 is a martingale difference.

Original: For martingale difference, we have the following theorem:
Revised: For martingale difference, we have the following theorem:

Original: Proposition 1 (Freedman’s Inequality, Freedman (1975)) .
Revised: Proposition 1 (Freedman’s Inequality, Freedman (1975)) .

Original: Consider a real-valued martingale difference sequence { X t } which is uniformly bounded, i.e. | X t | ≤ M almost surely for all t .
Revised: Consider a real-valued martingale difference sequence { X t } which is uniformly bounded, i.e. | X t | ≤ M almost surely for all t .

Original: Define the predictable quadratic variation process of the martingale W t := (cid:80) tj =1 E [ X 2 j | F j − 1 ] for all t , where {F t } is the filtration.
Revised: Define the predictable quadratic variation process of the martingale W t := (cid:80) tj =1 E [ X 2 j | F j − 1 ] for all t , where {F t } is the filtration.

Original: Then for all ℓ ≥ 0 and σ 2 > 0 ,
Revised: Then for all ℓ ≥ 0 and σ 2 > 0 ,

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: With these tools, we are ready to prove the claim.
Revised: With these tools, we are ready to prove the claim.

Original: Proof of Claim 1.
Revised: Proof of Claim 1.

Original: In our setting, it is evident that error( A ) =
Revised: In our setting, it is evident that error( A ) =

Original: Pr x ∼P [ x / ∈ X D ] / 2 = 1 / 2 − E
Revised: Pr x ∼P [ x / ∈ X D ] / 2 = 1 / 2 − E

Original: [ |X D | ] / 2 n 2 , where the expectation is taken over the randomness of X D , and | · | denotes the cardinality after removing duplicate elements.
Revised: [ |X D | ] / 2 n 2 , where the expectation is taken over the randomness of X D , and | · | denotes the cardinality after removing duplicate elements.

Original: Similarly, we have error( A AugPro - Mix )
Revised: Similarly, we have error( A AugPro - Mix )

Original: = 1 / 2 − E
Revised: = 1 / 2 − E

Original: [ |X AugPro - Mix | ] / 2 n 2 .
Revised: [ |X AugPro - Mix | ] / 2 n 2 .

Original: It suffices to prove
Revised: It suffices to prove

Original: [Equation]
Revised: [Equation]

Original: First, we show for some constant c 1 > 0 ,
Revised: First, we show for some constant c 1 > 0 ,

Original: [Equation]
Revised: [Equation]

Original: It is equivalent to drawing the elements in X D one by one.
Revised: It is equivalent to drawing the elements in X D one by one.

Original: Let X i D denote the elements after drawing x i .
Revised: Let X i D denote the elements after drawing x i .

Original: Let Y i
Revised: Let Y i

Original: = |X
Revised: = |X

Original: i D | , and D i be the indicator that x i ̸
Revised: i D | , and D i be the indicator that x i ̸

Original: = x j for all 0 < j < i , where D 1 = 1 .
Revised: = x j for all 0 < j < i , where D 1 = 1 .

Original: Then we know |X D
Revised: Then we know |X D

Original: | =
Revised: | =

Original: Y n , D i
Revised: Y n , D i

Original: =
Revised: =

Original: Y i − Y i − 1 , and for any X i − 1 D ,
Revised: Y i − Y i − 1 , and for any X i − 1 D ,

Original: [Equation]
Revised: [Equation]

Original: Let F i be the filtration, and hence E
Revised: Let F i be the filtration, and hence E

Original: [ D i | F i − 1 ] ≥ 1 − 1 n .
Revised: [ D i | F i − 1 ] ≥ 1 − 1 n .

Original: Let ˜ D
Revised: Let ˜ D

Original: i :=
Revised: i :=

Original: D i − E
Revised: D i − E

Original: [ D i | F i − 1 ] be a martingale difference sequence.
Revised: [ D i | F i − 1 ] be a martingale difference sequence.

Original: Hence | ˜ D i | ≤ 1 almost surely, and E [ ˜ D 2 i | F i − 1 ] ≤ 1 .
Revised: Hence | ˜ D i | ≤ 1 almost surely, and E [ ˜ D 2 i | F i − 1 ] ≤ 1 .

Original: Let
Revised: Let

Original: W i = (cid:80) ij =1 E [ ˜ D 2 j | F j − 1 ] .
Revised: W i = (cid:80) ij =1 E [ ˜ D 2 j | F j − 1 ] .

Original: By Freedman’s Inequality, one has
Revised: By Freedman’s Inequality, one has

Original: [Equation]
Revised: [Equation]

Original: − c 21 n log n 2 n + 2 c 1 √ n log n/ 3 ) .

Original: Choosing c 1 large enough proves Equation (9).
Revised: Choosing c 1 large enough proves Equation (8).

Original: Similarly we show for some constant c 2 > 0 ,
Revised: Similarly we show for some constant c 2 > 0 ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: It is equivalent to constructing X AugPro - Mix by iterations.
Revised: It is equivalent to constructing X AugPro - Mix by iterations.

Original: For i -th iteration, we draw x 2 i − 1 and x 2 i i.i.d.
Revised: For i -th iteration, we draw x 2 i − 1 and x 2 i i.i.d.

Original: uniformly and construct z i as described before.
Revised: uniformly and construct z i as described before.

Original: Let X i AugPro - Mix denote the data we get after constructing x 2 i − 1 , x 2 i and z i .
Revised: Let X i AugPro - Mix denote the data we get after constructing x 2 i − 1 , x 2 i and z i .

Original: Let Y ′
Revised: Let Y ′

Original: i = |X
Revised: i = |X

Original: i AugPro - Mix | and D
Revised: i AugPro - Mix | and D

Original: ′ i be the indicator that
Revised: ′ i be the indicator that

Original: Y ′
Revised: Y ′

Original: i − Y ′
Revised: i − Y ′

Original: i − 1 =(
Revised: i − 1 =(

Original: i.e. all of the three data are distinct and first constructed).
Revised: i.e. all of the three data are distinct and first constructed).

Original: We know Y ′ n/ 2 ≥ 3 (cid:80)
Revised: We know Y ′ n/ 2 ≥ 3 (cid:80)

Original: n/ 2 i =1 D
Revised: n/ 2 i =1 D

Original: ′ i .
Revised: ′ i .

Original: For any fixed vector x ∈ {− 1 , 1 } 2log n , we know Pr
Revised: For any fixed vector x ∈ {− 1 , 1 } 2log n , we know Pr

Original: [ x 2 i − 1 = x ] =
Revised: [ x 2 i − 1 = x ] =

Original: Pr[ x 2 i = x ] =
Revised: Pr[ x 2 i = x ] =

Original: Pr[ z i = x ] = 1 /n 2 .
Revised: Pr[ z i = x ] = 1 /n 2 .

Original: Hence for any X i − 1 AugPro - Mix and by union bound, one has
Revised: Hence for any X i − 1 AugPro - Mix and by union bound, one has

Original: [Equation]
Revised: [Equation]

Original: Let F ′ i be the filtration and hence E
Revised: Let F ′ i be the filtration and hence E

Original: [ D ′ i | F ′ i − 1 ] ≥ 1 − 92 n .
Revised: [ D ′ i | F ′ i − 1 ] ≥ 1 − 92 n .

Original: Let ˜ D
Revised: Let ˜ D

Original: ′ i :
Revised: ′ i :

Original: = D ′
Revised: = D ′

Original: i − E
Revised: i − E

Original: [ D ′ i
Revised: [ D ′ i

Original: | F i − 1 ] be a martingale difference sequence.
Revised: | F i − 1 ] be a martingale difference sequence.

Original: Similarly | ˜ D ′ i | ≤ 1 almost surely and E [ ˜ D ′ 

Revised: Similarly | ˜ D ′ i | ≤ 1 almost surely and E [ ˜ D ′

Original: i | F i − 1 ] ≤ 1 .
Revised: i | F i − 1 ] ≤ 1 .

Original: Let
Revised: Let

Original: W ′ i
Revised: W ′ i

Original: =
Revised: =

Original: (cid:80) ij =1 E [ ˜ D ′ 

Revised: (cid:80) ij =1 E [ ˜ D ′

Original: j | F ′ j − 1 ] .
Revised: j | F ′ j − 1 ] .

Original: By Freedman’s Inequality, one has
Revised: By Freedman’s Inequality, one has

Original: [Equation]
Revised: [Equation]

Original: where c ′ 

Revised: where c ′

Original: = c 2 − O (1) .
Revised: = c 2 − O (1) .

Original: Choosing c 2 large enough completes the proof of Equation (10).
Revised: Choosing c 2 large enough completes the proof of Equation (9).

Original: Combining equations (9) and (10) proves the statement.
Revised: Combining equations (8) and (9) proves the statement.

Original: G. P ROOF OF C LAIM
Revised: H. P ROOF OF C LAIM

Original: Proof.
Revised: Proof.

Original: Similarly, we have error( A AugPro - FGSM )
Revised: Similarly, we have error( A AugPro - FGSM )

Original: = 1 / 2 − E
Revised: = 1 / 2 − E

Original: [ |X AugPro - FGSM | ] / 2 n 

Revised: [ |X AugPro - FGSM | ] / 2 n

Original: and it suffices to prove
Revised: and it suffices to prove

Original: [Equation]
Revised: [Equation]

Original: We already have Equation (9).
Revised: We already have Equation (8).

Original: It suffices to prove that for some constant c 3 > 0 ,
Revised: It suffices to prove that for some constant c 3 > 0 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: For each x i ∈ X AugPro - FGSM and each coordinate j ∈
Revised: For each x i ∈ X AugPro - FGSM and each coordinate j ∈

Original: [2 log n ] , we know Pr[ x i ( j )
Revised: [2 log n ] , we know Pr[ x i ( j )

Original: = w i ( j )]
Revised: = w i ( j )]

Original: = Φ(1 / 2) < 0 .
Revised: = Φ(1 / 2) < 0 .

Original: 7 , where Φ is cumulative distribution function (CDF) of one-dimensional standard
Revised: 7 , where Φ is cumulative distribution function (CDF) of one-dimensional standard

Original: Gaussian distribution, i.e. Φ( t ) =
Revised: Gaussian distribution, i.e. Φ( t ) =

Original: Pr x ∼N (0 , 1) [ x ≤ t ] .
Revised: Pr x ∼N (0 , 1) [ x ≤ t ] .

Original: It is equivalent to constructing X AugPro - FGSM iteration by iteration, where in i -th iteration, we draw x i and construct w i as described before.
Revised: It is equivalent to constructing X AugPro - FGSM iteration by iteration, where in i -th iteration, we draw x i and construct w i as described before.

Original: Let X i AugPro - FGSM be the data we get after constructing x i and w i , let Y i
Revised: Let X i AugPro - FGSM be the data we get after constructing x i and w i , let Y i

Original: = |X
Revised: = |X

Original: i AugPro - FGSM | and let D i be the indicator that Y i − Y
Revised: i AugPro - FGSM | and let D i be the indicator that Y i − Y

Original: i − 1 = 2 .
Revised: i − 1 = 2 .

Original: For any fixed vector x ∈ {− 1 , 1 } 2log n , we know Pr[ x i = x ] = 1 /n 2 =
Revised: For any fixed vector x ∈ {− 1 , 1 } 2log n , we know Pr[ x i = x ] = 1 /n 2 =

Original: Pr[ w i = x ] and Pr[ w i = x i ] < 0 .
Revised: Pr[ w i = x ] and Pr[ w i = x i ] < 0 .

Original: 7 2log n < 1 /n 1 .
Revised: 7 2log n < 1 /n 1 .

Original: Let F i be the filtration, and for any X i − 1 AugPro - FGSM , by union bound, one has
Revised: Let F i be the filtration, and for any X i − 1 AugPro - FGSM , by union bound, one has

Original: [Equation]
Revised: [Equation]

Original: Let ˜ D
Revised: Let ˜ D

Original: i :=
Revised: i :=

Original: D i − E
Revised: D i − E

Original: [ D i | F i − 1 ] be a martingale difference sequence.
Revised: [ D i | F i − 1 ] be a martingale difference sequence.

Original: We know | ˜
Revised: We know | ˜

Original: D i | ≤ 1 and E [ ˜ D i | F i − 1 ] ≤ 1 almost surely.
Revised: D i | ≤ 1 and E [ ˜ D i | F i − 1 ] ≤ 1 almost surely.

Original: Let W i
Revised: Let W i

Original: =
Revised: =

Original: (cid:80) ij =1 E
Revised: (cid:80) ij =1 E

Original: [ ˜ D 2 i | F j − 1 ] .
Revised: [ ˜ D 2 i | F j − 1 ] .

Original: By Freedman’s Inequality, one has
Revised: By Freedman’s Inequality, one has

Original: [Equation]
Revised: [Equation]

Original: where c ′ 

Revised: where c ′

Original: = c 3 − O (1) .
Revised: = c 3 − O (1) .

Original: Choosing c 3 large enough completes the proof.
Revised: Choosing c 3 large enough completes the proof.

Original: Combing Equation (9) and (11) completes the proof.
Revised: Combing Equation (8) and (10) completes the proof.